Member since
11-15-2017
9
Posts
0
Kudos Received
0
Solutions
03-29-2018
02:51 AM
Okay, so apparently, Correlation Attribute Name does not support expression language. Changing, Correlation Attribute Name from ${trigger.type} to trigger.type solved the issue. Thanks still!
... View more
03-28-2018
09:48 AM
Thanks a lot Matt! This very well clears the merging logic. However, currently I am facing a strange issue. Suppose I have an attribute called 'trigger.type' on the flowfile, which can be 'mouse' or 'keyboard'. Now, I want to merge flowfiles with same 'trigger.type' values so I have the Correlation Attribute Name set to ${trigger.type} However, it's merging flowfiles with different values for this attribute into the same flowfile. I can't comprehent what am I missing here? These are my settings for the 'MergeContent' processor: Merge Strategy Bin-Packing Algorithm Merge Format Binary Concatenation Attribute Strategy
Keep Only Common Attributes Correlation Attribute Name
${trigger.type} Metadata Strategy
Ignore Metadata Minimum Number of Entries
6000 Maximum Number of Entries
6000 Minimum Group Size
0 B Maximum Group Size
No value set
Max Bin Age
300 seconds Maximum number of Bins
15 Delimiter Strategy
Text Header
[ Footer
]
Demarcator
, Compression Level
1 Keep Path
false Tar Modified Time
${file.lastModifiedTime}
... View more
03-27-2018
03:30 AM
I have a doubt @Matt Clarke Does NiFi always merge the bins after 'Minimum Number of Entries'? If yes, what's the use for 'Maximum Number of Entries'? My doubt originates from the following use case. I have to merge flowfiles based on the following two conditions: A) No. of flowfiles in the bin reaches 6000 B) Bin age crosses 5 min and at least one flowfile in the bin What should be the settings for this case? Thanks!
... View more
02-06-2018
01:24 AM
Thanks a lot Bryan! I had been pulling my hair out for days now over this.
... View more
02-05-2018
03:15 AM
{
"name": "userAgreement",
"type": [
"null",
"boolean"
],
"default": null
} This is the schema for one of the fields of my json data, auto-generated using InferAvroSchema processor. The field 'userAgreement is an optional field, and when missing the value should be set to null. So the schema seems to be generated correctly. However, on partitioning this data using PartitionRecord processor, the JsonTreeReader throws this error. ..will route to failure: org.apache.nifi.schema.access.SchemaNotFoundException:
java.lang.IllegalArgumentException: Cannot set the deafult value for field 'userAgreement' to [org.apache.avro.JsonProperties$Null@5d0e5e41] because this is not a valid value for Data Type [Boolean] Any ideas what's going wrong?
... View more
Labels:
- Labels:
-
Apache NiFi
11-17-2017
02:55 AM
Precisely what I needed. Thanks!
... View more
11-16-2017
05:07 AM
I have a very simple use case but unable to come up with the right combination of processors. So, consider my flowfile coming from a HDFS json file with the following content- [{"type":"A","id":"001","content":"abc"},
{"type":"A","id":"001","content":"xyz"},
{"type":"A","id":"002","content":"sdf"},
{"type":"B","id":"004","content":"df"},
{"type":"B","id":"002","content":"dsg"},
{"type":"B","id":"002","content":"sfg"},
{"type":"B","id":"004","content":"sfg"}]
I want to finally store these in the following directory structure back in HDFS. /A/001/data.txt (data.txt-> [{"type":"A","id":"001","content":"abc"},{"type":"A","id":"001","content":"xyz"}])
/A/002/data.txt (data.txt-> [{"type":"A","id":"002","content":"sdf"}])
/B/002/data.txt (data.txt-> [{"type":"B","id":"002","content":"dsg"},{"type":"B","id":"002","content":"sfg"}])
/B/004/data.txt (data.txt-> [{"type":"B","id":"004","content":"df"},{"type":"B","id":"004","content":"sfg"}])
Any ideas?
... View more
Labels:
- Labels:
-
Apache NiFi