Support Questions
Find answers, ask questions, and share your expertise

NIFI - Removing spaces in all keys on a JSON file

NIFI - Removing spaces in all keys on a JSON file

Contributor

Hi, i'm having a problem with this, I decompress a gz, it gives me 10k+ json files with 1 element each.

I make a transformation with JOLT and SHIFT only to reorder the fields.

Now I have this problem, when trying to InferAvroSchema, it founds spaces on some keys, the keys can change, so I don't want to force the schema.

How can I replace all the keys with spaces (replace with "_") and keeping the current tree of the JSON? don't want to hardcode anything if possible, because some of the keys are actions, and when a new element is created on the app wich generates the json, a new child key is added.

 

My last try was this:

https://community.cloudera.com/t5/Support-Questions/How-to-do-JOLT-replace-on-all-JSON-keys-in-Nifi/...

But it doesn't modify the child elements, only root ones, and if I tell on wich element is what I want to replace it "shift" it to root and all the other elements dissapear.

 

The JSON is something like:

 

{
"server_received_time": "2019-08-01",
"app": 8,
"event_id": int,
"event_properties":{"event buy":"one",
"event try":"yes"}
"event_state":{"event profile":"absent",
"event_fail":"yes",
"event text":"will try again"}
}

The json has a lot more keys, a space will not appear in the root elements but it can in the child elements, so I only need to replace those keeping the others intact.

The transformations I try make me lost the schema format or name all the child elements (that I want to keep this way because I dont know if new ones will appear.

 

The idea is to remove this spaces on keys so I can InferAvroSchema without loosing elements and continue to process and store them

Any idea?

Thank you!