How to use jq as SED for large json file (optionally with a mapping table)
I have a large json file which contains several IPs within json files. I want to replace specific IP addresses within the json file (without having to write the putout into a new file). The equivalent SED command would be sed -i 's/old_ip/new_ip/g'
The structure is as follows (of a single json entry in the json file):
{
"destination":{
"port":53,
"bytes":100,
"ip":"1.1.1.1"
},
"source":{
"port":54894,
"bytes":84,
"ip":"10.1.1.49",
},
"related":{
"ip":"10.5.5.45"
},
"event":{
"code":"0000000013",
"action":"accept",
"type":[
"allowed",
"connection",
"end",
"protocol"
],
"outcome":"success"
},
"@timestamp":"2021-08-29T04:47:10.000+02:00"
}
I have now two problems which I can't figure to solve.
Problem 1: I want to replace every IP by a corresponding IP address. So imagine I want to replace the IP 10.1.1.49 with 10.10.10.10
I know I can use the command jq '(.. | .ip?)' | select(.ip == "10.2.1.49")| .ip |= "10.10.10.10"'
but unfortunately this just outputs the new IP instead of replacing it in the file like SED does (this refers to all solutions I found on the web).
Problem 2: Now suppose I have a mapping table (mapping.json) with the following structure:
{
"10.2.1.49":"10.10.10.10",
"10.15.15.15":"10.10.10.10"
}
Can I provide and combine this mapping table with jq for a find and replace? I have the feeling that this is not possible in jq meaning I would need to figure out my first problem and then writing a bash script. Someone had similar question (see here) by using --sluprfile. But apparently I can't use the solution from this link as a ID or unique value for selection is required which I don't have in my json.
Is there a simple solution or should I just use SED combined with a shellscript? Thanks
Solution 1:
Re Problem 1
You can update the relevant IP addresses using
jq '(.. | .ip? | select(. == "10.1.1.49")) |= "10.10.10.10"'
{
"destination": {
"port": 53,
"bytes": 100,
"ip": "1.1.1.1"
},
"source": {
"port": 54894,
"bytes": 84,
"ip": "10.10.10.10"
},
"related": {
"ip": "10.5.5.45"
},
"event": {
"code": "0000000013",
"action": "accept",
"type": [
"allowed",
"connection",
"end",
"protocol"
],
"outcome": "success"
},
"@timestamp": "2021-08-29T04:47:10.000+02:00"
}
Demo
But you cannot update the file as you can with sed -i
- at least not with just using a flag. Using temporary files or sponge
etc. will, however, work fine.
Re Problem 2
Read in the mappings e.g. using --argfile
and update using a lookup in combination with the alternative operator //
to fall back to the original value, if it doesn't exist in the mapping
jq --argfile mapping mapping.json '(.. | .ip? // empty) |= ($mapping[.]? // .)'
Demo