Member since
05-10-2019
22
Posts
0
Kudos Received
0
Solutions
10-27-2019
06:58 AM
Hi all, I am trying to pass two attributes through the Notify AttributeCacheRegex property, but I can only pass one of them (the first one): layer_timestamp|a.absolute.hdfs.path How do I do this with Regex? Thanks, Rosa
... View more
Labels:
10-02-2019
01:37 AM
Hello all..
I would like to set a Notify/Wait logic in my flow.
A file comes in the PG, and it's being flaten out with JOLT.
I want it to be passed to a second PG only when this file is
successfuly stored on HDFS
successfuly exited the JOLT conversion Group.
Now, i've set up a new attribute called flag after it's stored and after it's successfully converted with JOLT. However, The notify processor is not working as expected.
This is my notify configuration: I'm not sure what to do with the successful relationship here.. do I terminate it or what?
And my wait configuration...
Now.. the after the file is stored as .orc... i need the .json file that's waiting to be passed on to the next process group outside this one. But I'm not sure what I'm doing wrong?
Thanks,
Rosa
... View more
Labels:
08-29-2019
06:57 AM
HI @DennisJaheruddi , thank you for your reply. let me try to explain. I want to read from my Metadata table and store the column values into attributes. FilenamePattern TimestampPattern FilenameFilterRegex indiana-YYYY-MM-DD THH-MM-SSZ-YYYYMMDDHHMMSS.gz YYYY-MM-DD THH-MM-SSZ-YYYYMMDDHHMMSS ${http.param.path:matches('indiana-\d{4}-\d{2}-\d{2}T\d{2}-\d{2}-\d{2}Z-[0-9]{14}.gz')} So, now I have my FF with the following attributes: ${FilenamePattern}=indiana-YYYY-MM-DD THH-MM-SSZ-YYYYMMDDHHMMSS.gz ${TimeStampPattern}=YYYY-MM-DD THH-MM-SSZ-YYYYMMDDHHMMSS ${FilenameFilterRegex}= ${http.param.path:matches('indiana-\d{4}-\d{2}-\d{2}T\d{2}-\d{2}-\d{2}Z-[0-9]{14}.gz')} And, when I go and actually fetch the Flow File from HDFS, I want to Route on that FilenameFilterRegex attribute like this: So, my questions is... can I literally evaluate the FilenameFilterRegex. Because, now, it will only evaluate it as a string, but it needs to be evaluated as regular expression. Thanks, Rosa
... View more
08-22-2019
04:31 AM
@puneetn HI, I'm having the same scenario. I wonder, have you ever come up with a solution? Thanks, Rosa
... View more
08-22-2019
02:48 AM
Hello all, I'm working on a metadata driven solution. The idea is to store metadata in a SQL table. This table would contain something like this: Filename Filename Details name 1 expression language name 2 expression language Is it possible to use that FilenameDetails attribute value and use that value in another processor? Something like evaluate Attribute? then I would use ${FilenameDetails} as a variable in processor to apply that expression language? @mburgess do you mind checking out my issue? Thanks, Rosa
... View more
Labels:
08-07-2019
09:06 AM
@Justen Thanks!
... View more
08-07-2019
09:05 AM
@ Justen i believe this is the answer: Thanks!
... View more
08-07-2019
08:49 AM
@ Justen my JSON file is being converted to null (file size iz 4bytes) after wrong JOLT is applied. So, when I open the JSON it's litterally 'null'. There are no brackets.. just null.
... View more
08-07-2019
08:26 AM
HI All, does anybody know how to check if the content of the file is null or empty? (JSON for example)? Thanks, Nera
... View more
Labels:
07-26-2019
12:52 PM
HI All, Is there an easy way to capture object index ? INPUT: { "header": { "namespace": "someNamespace", "version": "1" }, "body": { "dataM": { "sourceM": [ { "name": "State", "type": "string" }, { "name": "City", "type": "string" }, { "name": "var", "type": "string" }, { "name": "unit", "type": "string" }, { "name": "type", "type": "string" } ], "valueM": [ { "name": "timestamp", "type": "date-time" }, { "name": "value", "type": "string" }, { "name": "quality", "type": "integer" }, { "name": "error", "type": "string" } ] }, "data": [ { "source": [ "Austria", "Vienna", "temp", "F", "now" ], "values": [ [ "2018-10-02T15:00:00.05Z", "3.1234", 7, "SomeError" ], [ "2018-10-02T15:00:46.05Z", "3.221", 0, "SomeOtherError" ] ] }, { "source": [ "Germany", "Berlin", "pressure", "PA", "now" ], "values": [ [ "2018-10-02T16:00:00.05Z", "5", 0, "NoError" ] ] } ] } } spec: [{ "operation": "shift", "spec": { "body": { "data": { "*": { "@(4,header)": { "*": "newdata[&2].&" }, "source": { "*": "newdata.[&2].@(4,dataM.sourceM.[&].name)" }, "values": { "*": { "*": "newdata.[&3].@(5,dataM.valueM[&].name)" } } } } } } } , { "operation": "shift", "spec": { "newdata": { "*": "[&0]" } } } ] Desired Output: [ { "namespace" : "someNamespace", "version" : "1", "State" : "Austria", "City" : "Vienna", "var" : "temp", "unit" : "F", "type" : "now", "timestamp" :"2018-10-02T15:00:00.05Z", "value" : "3.1234", "quality" : 7, "error" : "SomeError" }, { "namespace" : "someNamespace", "version" : "1", "State" : "Austria", "City" : "Vienna", "var" : "temp", "unit" : "F", "type" : "now", "timestamp" : "2018-10-02T15:00:46.05Z" , "value" : "3.221" , "quality" : 0, "error" : "" },{ "namespace" : "someNamespace", "version" : "1", "State" : "Germany", "City" : "Berlin", "var" : "pressure", "unit" : "PA", "type" : "now", "timestamp" : "2018-10-02T16:00:00.05Z", "value" : "5", "quality" : 0, "error" : "NoError" } ] Currently, I'm getting arrays for timestamp, value, quality and error for each object. Instead, I need source array repeated per number of values within the data object.
... View more
Labels:
07-12-2019
11:14 AM
@ Justen use EvaluateJsonPath to extract the record into an attribute, apply the changes and push it back in with update record processor
... View more
07-12-2019
08:58 AM
@ Justen as far as the schemas go, try using SimpleKeyLookupService controller service, create two new properties that will point to AvroSchemaRegistry controller service. That way you will call for ${schema.name} attribute in your processors instead of hardcoding the schemas into the processor directly.
... View more
07-12-2019
08:56 AM
@ Justen try using evalate json path... https://jsonpath.com/
... View more
07-09-2019
11:33 AM
@Matt Burgess there it is. Nothing much to it. This is the same link (minus the api/v1) that I'm using to access the HW Schema UI
... View more
07-09-2019
11:29 AM
you could first read that line from the json file (evaluate json path) and then update attribute with expression language and push it back into the json file with update record processor. let me know if you need further help.
... View more
07-08-2019
02:34 PM
Hi all, i'm using a simple key lookup service to lookup a schema name and save that as an attribute. Later I want to convert JSON to AVRO using HW shema rehgistry but I'm getting the error: Could not retrieve schema with name "ValueSchemaName" from the configured Schema Registry. JSON Tree reader and AvroRecordSetWriter are using : Schema Access Strategy: Use 'Schema Name' and are pointed to HW Schema Registry. Json Tree Reader: Avro Record Set Writer: Am I configuring the schema access wrongly? Also, where can I get the correct URL for configuring the HWSchemaRegistry controller service? Is it the same link that I see in the Schema Registry UI? Thanks, Rosa
... View more
Labels:
06-13-2019
11:21 AM
Hello all, Is there a way to access provenance repository? Not through UI. I don't need a FF specific provenance, but the entire provenance of all PG, processors etc. Thanks, Rosa
... View more
06-04-2019
01:13 PM
@Matt Burgess do you mind taking a look? Is this something that can be done with Validate Record?
... View more
06-03-2019
05:19 PM
Hi All, My flow is moving a lot of JSON files in the following format: [{
"name": "Don",
"id": 15
"opened": "2019-03-26T05:47:14.670Z",
"a": "2019-03-26T06:31:45.133Z",
"closed": "2019-03-26T06:33:56.960Z",
"periodA": "00:44:31",
"periodN": "00:02:11"
}, {
"name": "Don,
"opened": "2019-03-26T05:57:43.763Z",
"a": "2019-03-26T06:13:48.150Z",
"closed": "2019-03-26T06:31:16.273Z",
"periodA": "00:16:05",
"periodN": "00:17:28"
}] Is there a way to check if timestamp fields are in ISO8601 format, and if not convert it accordingly? What is the correct way of doing this? The utlimate result will be pushed to Kafka in AVRO format.
... View more
Labels:
06-03-2019
05:09 PM
@Matt Burgess do you mind helping me out with a similar issue? I have a json file comming in as : [ {
"regId" : "us",
"cId" : "SomeProduct",
"weId" : 15,
"name" : "R",
"id" : 1539,
"cityId" : 17,
"cityName" : "Moskow"
}, {
"regId" : "us",
"cId" : "SomeProduct",
"weId" : 15,
"name" : "R",
"id" : 1540,
"cityId" : 18,
"cityName" : "Berlin"
}, {
"regId" : "us",
"cId" : "SomeProduct",
"weId" : 15,
"name" : "R",
"id" : 1541,
"cityId" : 19,
"cityName" : "Vienna"
} ] I need to add 3 new fields passed from the FlowFile attribute into each element? Can I do that with the update Record also? [ {
"new_att":"somevalue",
"new_att2":"somevalue2",
"new_att3":"somevalue3",
"regId" : "us",
"cId" : "SomeProduct",
"weId" : 15,
"name" : "R",
"id" : 1539,
"cityId" : 17,
"cityName" : "Moskow"
}, {
"new_att":"somevalue",
"new_att2":"somevalue2",
"new_att3":"somevalue3",
"regId" : "us",
"cId" : "SomeProduct",
"weId" : 15,
"name" : "R",
"id" : 1540,
"cityId" : 18,
"cityName" : "Berlin"
}, {
"new_att":"somevalue",
"new_att2":"somevalue2",
"new_att3":"somevalue3",
"regId" : "us",
"cId" : "SomeProduct",
"weId" : 15,
"name" : "R",
"id" : 1541,
"cityId" : 19,
"cityName" : "Vienna"
} ] Thanks, Rosa
... View more
05-17-2019
04:02 PM
I'm trying to convert nested arrays into objects depending on the number of values in the second nested array. I can't seem to get the number of the value fields and use that as a key in my spec. Now this is my input JSON file: {
"meta": {
"regId": "us",
"cId": "SomeProduct",
"weId": 15
},
"data": {
"name": "R",
"details": {
"headers": [
"id",
"cityId",
"cityName"
],
"values": [
[
1539,
17,
"Moskow"
],
[
1539,
17,
"Berlin"
],
[
1539,
17,
"Vienna"
]
]
}
}
} This my desired JSON Output:
[
{"regId": "us",
"cId": "SomeProduct",
"weId": 15,
"name":"R",
"id":1539,
"cityId":17,
"cityName":Moskow
},
{"regId": "us",
"cId": "SomeProduct",
"weId": 15,
"name":"R",
"id":1540,
"cityId":11,
"cityName":Berlin
},
{"regId": "us",
"cId": "SomeProduct",
"weId": 15,
"name":"R",
"id":151,
"cityId":18,
"cityName":Vienna
}
] This is my current spec: [
{
"operation": "shift",
"spec": {
"meta": {
"*": "&"
},
"data": {
"name": "&",
"details": {
"values": {
"*": {
"*": "@(3,headers[&])"
}
}
}
}
}
}
] I will be having multiple schemas arriving my way, is it better to have an InferAvroSchema and store the schemas into the Schema Library? What processors would that actually use? So I've got multiple files incoming in the flow with multple JSON schemas. And I need to denormalize every one of them to a queriable ORC form. Thanks in advance, N
... View more
Labels:
05-10-2019
01:09 PM
Hi all, I am a rookie at both NiFi and Python and I need your help to pass the Flow File attribute value to the script. The script is converting a nested json into csv. When I run the script locally it works. import pandas as pd import json from pandas.io.json import json_normalize src_json = "C:/Users/name/Documents/Filename.json" tgt_csv = "C:/Users/name/Documents/Filename.csv" jfile = open(src_json) jdata = json.load(jfile) How can I pass the FlowFile name to src_json and tgt_csv? Thanks, Rosa
... View more
Labels: