Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Converting JSON to Avro (Field names have special chars ":") in NiFI.

Solved Go to solution

Converting JSON to Avro (Field names have special chars ":") in NiFI.

New Contributor

Hello,

I'm trying to convert JSON to Avro in Nifi. The problem is that some of the fields have ":" in their names -- which is not allowed in avro. (JSON was translated from XML that had namespaces)

EX:

{

"sourceTimeStamp": "2018-06-21T17:35:48Z",

"msgType": "blah",

"Ref": 71768908,

"fdm:Information": { ... }

}

Would it be easier to just go from XML (with name spaces) to Avro?

Any suggestions? Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Converting JSON to Avro (Field names have special chars ":") in NiFI.

If they only have one colon, you could use the following transform in JoltTransformJSON to change the names to be valid in Avro:

[
  {
    "operation": "shift",
    "spec": {
      "*:*": "&(0,1)_&(0,2)",
      "*": "&"
    }
  }
]

If you are translating from XML using a stylesheet, perhaps you could do a similar thing with the XSLT. Also as of NiFi 1.7.0 there is an XML record reader, so you could also use UpdateRecord to replace the values from the namespaced fields into a field with an Avro-valid name.

3 REPLIES 3

Re: Converting JSON to Avro (Field names have special chars ":") in NiFI.

If they only have one colon, you could use the following transform in JoltTransformJSON to change the names to be valid in Avro:

[
  {
    "operation": "shift",
    "spec": {
      "*:*": "&(0,1)_&(0,2)",
      "*": "&"
    }
  }
]

If you are translating from XML using a stylesheet, perhaps you could do a similar thing with the XSLT. Also as of NiFi 1.7.0 there is an XML record reader, so you could also use UpdateRecord to replace the values from the namespaced fields into a field with an Avro-valid name.

Highlighted

Re: Converting JSON to Avro (Field names have special chars ":") in NiFI.

New Contributor

Matt,

this works great - but is there a way to apply the same shift operation recursively to nested fields with colons as well.

EX:

{

"sourceTimeStamp": "2018-06-21T17:35:48Z",

"msgType": "blah",

"Ref": 71768908,

"fdm:Information": {

"fdm:first":{

"fdm:last": etc

}

}

}

Thanks, for the XML tips for 1.7.

Re: Converting JSON to Avro (Field names have special chars ":") in NiFI.

You can't do it recursively but you can manually do it if you know the max depth of the nested fields, you just need a similar spec to match each nested field, on the level that it matches. So instead of "*": "&", you'd try to match the nested fields underneath, with a sub-spec pretty much identical to the top-level one.

Don't have an account?
Coming from Hortonworks? Activate your account here