Hello all..   I would like to set a Notify/Wait logic in my flow. A file comes in the PG, and it's being flaten out with JOLT.  I want it to be passed to a second PG only when this file is successfuly stored on HDFS successfuly exited the JOLT conversion Group.   Now, i've set up a new attribute called flag after it's stored and after it's successfully converted with JOLT. However, The notify processor is not working as expected.   This is my notify configuration: I'm not sure what to do with the successful relationship here.. do I terminate it or what? And my wait configuration...   Now.. the after the file is stored as .orc... i need the .json file that's waiting to be passed on to the next process group outside this one. But I'm not sure what I'm doing wrong?   Thanks, Rosa     ... View more

Hello all,   I'm working on a metadata driven solution. The idea is to store metadata in a SQL table. This table would contain something like this: Filename Filename Details name 1 expression language name 2 expression language   Is it possible to use that FilenameDetails attribute value and use that value in another processor? Something like evaluate Attribute? then I would use ${FilenameDetails} as a variable in processor to apply that expression language?   @mburgess  do you mind checking out my issue?    Thanks, Rosa   ... View more

HI All, does anybody know how to check if the content of the file is null or empty? (JSON for example)? Thanks, Nera ... View more

HI All, Is there an easy way to capture object index ? INPUT: { "header": { "namespace": "someNamespace", "version": "1" }, "body": { "dataM": { "sourceM": [ { "name": "State", "type": "string" }, { "name": "City", "type": "string" }, { "name": "var", "type": "string" }, { "name": "unit", "type": "string" }, { "name": "type", "type": "string" } ], "valueM": [ { "name": "timestamp", "type": "date-time" }, { "name": "value", "type": "string" }, { "name": "quality", "type": "integer" }, { "name": "error", "type": "string" } ] }, "data": [ { "source": [ "Austria", "Vienna", "temp", "F", "now" ], "values": [ [ "2018-10-02T15:00:00.05Z", "3.1234", 7, "SomeError" ], [ "2018-10-02T15:00:46.05Z", "3.221", 0, "SomeOtherError" ] ] }, { "source": [ "Germany", "Berlin", "pressure", "PA", "now" ], "values": [ [ "2018-10-02T16:00:00.05Z", "5", 0, "NoError" ] ] } ] } } spec: [{ "operation": "shift", "spec": { "body": { "data": { "*": { "@(4,header)": { "*": "newdata[&2].&" }, "source": { "*": "newdata.[&2].@(4,dataM.sourceM.[&].name)" }, "values": { "*": { "*": "newdata.[&3].@(5,dataM.valueM[&].name)" } } } } } } } , { "operation": "shift", "spec": { "newdata": { "*": "[&0]" } } } ] Desired Output: [ { "namespace" : "someNamespace", "version" : "1", "State" : "Austria", "City" : "Vienna", "var" : "temp", "unit" : "F", "type" : "now", "timestamp" :"2018-10-02T15:00:00.05Z", "value" : "3.1234", "quality" : 7, "error" : "SomeError" }, { "namespace" : "someNamespace", "version" : "1", "State" : "Austria", "City" : "Vienna", "var" : "temp", "unit" : "F", "type" : "now", "timestamp" : "2018-10-02T15:00:46.05Z" , "value" : "3.221" , "quality" : 0, "error" : "" },{ "namespace" : "someNamespace", "version" : "1", "State" : "Germany", "City" : "Berlin", "var" : "pressure", "unit" : "PA", "type" : "now", "timestamp" : "2018-10-02T16:00:00.05Z", "value" : "5", "quality" : 0, "error" : "NoError" } ] Currently, I'm getting arrays for timestamp, value, quality and error for each object. Instead, I need source array repeated per number of values within the data object. ... View more

Hi all, i'm using a simple key lookup service to lookup a schema name and save that as an attribute. Later I want to convert JSON to AVRO using HW shema rehgistry but I'm getting the error: Could not retrieve schema with name "ValueSchemaName" from the configured Schema Registry. JSON Tree reader and AvroRecordSetWriter are using : Schema Access Strategy: Use 'Schema Name' and are pointed to HW Schema Registry. Json Tree Reader: Avro Record Set Writer: Am I configuring the schema access wrongly? Also, where can I get the correct URL for configuring the HWSchemaRegistry controller service? Is it the same link that I see in the Schema Registry UI? Thanks, Rosa ... View more

Hello all, Is there a way to access provenance repository? Not through UI. I don't need a FF specific provenance, but the entire provenance of all PG, processors etc. Thanks, Rosa ... View more

Hi All, My flow is moving a lot of JSON files in the following format: [{         "name": "Don",         "id": 15         "opened": "2019-03-26T05:47:14.670Z",         "a": "2019-03-26T06:31:45.133Z",         "closed": "2019-03-26T06:33:56.960Z",         "periodA": "00:44:31",         "periodN": "00:02:11"     }, {         "name": "Don,         "opened": "2019-03-26T05:57:43.763Z",         "a": "2019-03-26T06:13:48.150Z",         "closed": "2019-03-26T06:31:16.273Z",         "periodA": "00:16:05",         "periodN": "00:17:28"     }] Is there a way to check if timestamp fields are in ISO8601 format, and if not convert it accordingly? What is the correct way of doing this? The utlimate result will be pushed to Kafka in AVRO format. ... View more

I'm trying to convert nested arrays into objects depending on the number of values in the second nested array. I can't seem to get the number of the value fields and use that as a key in my spec. Now this is my input JSON file: {  "meta": {    "regId": "us",    "cId": "SomeProduct",    "weId": 15  },  "data": {    "name": "R",    "details": {      "headers": [        "id",        "cityId",        "cityName"      ],      "values": [        [          1539,          17,          "Moskow"        ],        [          1539,          17,          "Berlin"        ],        [         1539,          17,          "Vienna"        ]      ]    }  } } This my desired JSON Output: [     {"regId": "us",         "cId": "SomeProduct",         "weId": 15,         "name":"R",         "id":1539,         "cityId":17,         "cityName":Moskow     },     {"regId": "us",         "cId": "SomeProduct",         "weId": 15,         "name":"R",         "id":1540,         "cityId":11,         "cityName":Berlin     },     {"regId": "us",         "cId": "SomeProduct",         "weId": 15,         "name":"R",         "id":151,         "cityId":18,         "cityName":Vienna     } ] This is my current spec:     [   {     "operation": "shift",     "spec": {       "meta": {         "*": "&"       },       "data": {         "name": "&",         "details": {           "values": {             "*": {               "*": "@(3,headers[&])"             }           }         }       }     }   } ] I will be having multiple schemas arriving my way, is it better to have an InferAvroSchema and store the schemas into the Schema Library? What processors would that actually use? So I've got multiple files incoming in the flow with multple JSON schemas. And I need to denormalize every one of them to a queriable ORC form. Thanks in advance, N ... View more

Hi all, I am a rookie at both NiFi and Python and I need your help to pass the Flow File attribute value to the script. The script is converting a nested json into csv. When I run the script locally it works. import pandas as pd import json from pandas.io.json import json_normalize src_json = "C:/Users/name/Documents/Filename.json" tgt_csv = "C:/Users/name/Documents/Filename.csv" jfile = open(src_json) jdata = json.load(jfile) How can I pass the FlowFile name to src_json and tgt_csv? Thanks, Rosa ... View more