About jerry_pylarinos

jerry_pylarinos · ‎06-21-2019

In continuation to the above using Distributed Map Cache to hold a Jolt Spec read from a file, I can confirm that it works. Flow A: GetFile (periodically scheduled by CRON) --> PutDistributedMapCache --> LogAttribute: Jolt spec file contents ( jspec.json ) on disk: [{"operation":"shift","spec":{"variables":{"0":"variables[0].username","1":"variables[0].active","2":"variables[0].temperature","3":"variables[0].age"},"*":"&"}}] Note: Jolt spec in compact form!! GetFile processor: File Filter <-- jspec.json PutDistributedMapCache processor: Cache Entry Identifier <-- joltspec Flow B: GenerateFlowFile --> FetchDistributedMapCache --> JoltTransformJSON GenerateFlowFile processor: Custom Text <--- { "id" : 123456, "ip" : "*", "t" : -12.9, "T" : -23.8, "variables": [ "user1", 0, 12.97, 23 ] } FetchDistributedMapCache processor: Cache Entry Identifier <-- joltspec Put Cache Value in Attribute <-- jspec JoltTransformJSON processor: Jolt Transformation DSL <-- Chain Jolt Specification <-- ${jspec}

jerry_pylarinos · ‎06-20-2019

@Matt Burgess I have noticed that there hasn't been progress on this open issue (NIFI-4957). Nevertheless I tried placing the Jolt spec in a processor group variable (e.g. jspec) and then using this variable in the JoltTransformJSON processor ( Jolt Specification = ${jspec} ) and it transformed the JSON FlowFile successfully. Hence an intermediate solution (until NIFI-4957 is resolved) could be to update the processor group variable at the cost of resetting NiFi. I will also investigate the process: 1.- Get Jolt Spec from file ( periodically with a CRON ) 2.- Put FlowFile ( Jolt Spec ) in Distributed Map Cache with e.g. Cache Entry Identifier = jspec 3.- Fetch Distributed Map Cache (as and when required to transform the JSON FlowFile) and put cache value in attribute jspec 4.- Assign jspec to Jolt Specification of JoltTransformJSON processor

jerry_pylarinos · ‎06-19-2019

I solved this by using a GenerateFlowFile processor to create an attribute url and to trigger the InvokeHTTP processor at the schedule rate of interest. On a failure condition (e.g. InvokeHTTP connection timeout) I use an UpdateAttribute processor to assign to the attribute url the Remote URL using a Regex: ${invokehttp.java.exception.message:substringAfterLast('/'):substringBefore(':')}

jerry_pylarinos · ‎06-03-2019

I am working on a NIFI flow that begins with an InvokeHTTP processor. On the occasion that there is a connection timeout (Failure condition), I wish to assign the InvokeHTTP processor Remote URL value to a new property and have this property passed as data (attribute) through the "Failure" path to another processor. Is it possible to do within the same InvokeHTTP processor? Can anyone help?

jerry_pylarinos · ‎04-09-2019

That's great Matt. Looks like it will become a reality. Thank you!

jerry_pylarinos · ‎04-08-2019

Yes, excellent! Thank you @Matt Burgess!!! On this topic, I tried reading the jolt spec from a file using the GetFile processor and then extracting the text (jolt spec) using ExtractText and assigning it to a new property such that I can pass the spec to the JoltTransformJSON processor but it doesn't work. How can this be done? That is, how can the jolt spec be read from a file instead of configuring the Jolt processor? Thanking you in advance!

jerry_pylarinos · ‎04-07-2019

Hello, Receiving from an embedded micro over Ethernet a JSON file of the form: { "id" : 123456, "ip" : "*", "t" : -12.9, "T" : -23.8, "variables": [ "user1", 0, 12.97, 23 ] } and knowing what each variable represents in the "variables" array as: "username" : "user1", "active" : 0, "temperature" : 12.97, "age" : 23 How can I update the JSON file with the key name of the variables such that the end result is: { "id" : 123456, "ip" : "*", "t" : -12.9, "T" : -23.8, "variables": [ "username" : "user1", "active" : 0, "temperature" : 12.97, "age" : 23 ] } and then be able to access these variables with their name? Thank you.

Online	Offline
Last Visited	‎02-17-2020 04:34 AM

Member Since	‎04-01-2019 01:52 PM
Last Visited	‎02-17-2020 04:34 AM
Posts	10

Cloudera Community

Re: InvokeHTTP write a property on Failure Conditi...

Re: Format JSON file using schema

Re: Format JSON file using schema

Re: InvokeHTTP write a property on Failure Conditi...

InvokeHTTP write a property on Failure Condition

Re: Format JSON file using schema

Re: Format JSON file using schema

Format JSON file using schema