Member since
04-05-2016
130
Posts
93
Kudos Received
29
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3791 | 06-05-2018 01:00 AM | |
5142 | 04-10-2018 08:23 AM | |
5649 | 07-18-2017 02:16 AM | |
2913 | 07-11-2017 01:02 PM | |
3342 | 07-10-2017 02:10 AM |
06-08-2017
08:19 AM
Hello @rian amrodin
As @Wynner suggested, you should be able to route FlowFiles based on file size using 'gt()' EL function. However surprisingly, ListFTP and ListSFTP doesn't write 'file.size' attribute where it's supposed to do.. I submitted a JIRA NIFI-4041 and a fix for that. I hope it will be merged and available soon. Thanks for sharing your experience with Apache NiFi! https://issues.apache.org/jira/browse/NIFI-4041
... View more
06-05-2017
11:54 PM
@Arsalan Siddiqi I assumed UpdateRecord has been there since 1.2.0, but it's not. Sorry about that. Created another template which doesn't use UpdateRecord. Instead, I used another QueryRecord to update CSV. Confirmed it works with NiFi 1.2.0. Hope you will find it useful. https://gist.githubusercontent.com/ijokarumawak/7e20af1cd222fb2adf13acb2b0f46aed/raw/e150884f52ca186dd61433428b38d172aaa7b128/Join_CSV_Files_1.2.0.xml
... View more
06-05-2017
06:11 AM
1 Kudo
Hello @Arsalan Siddiqi I think it's possible with NiFi 1.2.0 or later, by using QueryRecord processor. The basic idea is using MergeContent to create a single FlowFile, containing all CSV files, when doing so, add a column specifying which attribute (I used 'm' column in my example). Then use QueryRecord processor to join records and produce a row having different attributes in it: select fa.t, fa.v a, fb.v b, fc.v c
from (
select t, v from FLOWFILE where m = 'a'
) fa
left join (
select t, v from FLOWFILE where m = 'b'
) fb on fa.t = fb.t
left join (
select t, v from FLOWFILE where m = 'c'
) fc on fa.t = fc.t I've created a Gist with NiFi flow template, I hope this helps: https://gist.github.com/ijokarumawak/7e20af1cd222fb2adf13acb2b0f46aed
... View more
06-05-2017
01:14 AM
1 Kudo
Hi @Eric Lloyd I found HDFS-11367 which was reported for the similar issue you encountered. By reading that JIRA and checking NiFi PutHDFS processor code that calls OutputStream.close method, I suspect there had been other exception such as TimeoutException and it causes AlreadyBeingCreatedException. I see you have configured the 'Create Filename (UpdateAttribute)' processor to have timestamp in filename as minutes resolution, and PutHDFS with 'Append' conflict resolution mode. This might overwhelm HDFS by executing lots of append requests. I'd suggest setting MergeContent with longer interval at its 'Run Schedule' such as 1 min or 30 seconds so that NiFi merge contents locally and perform less append operations. It would be not only safer but also more performant.
... View more
05-29-2017
04:32 AM
Hello @Pierre Leroy I am not sure why POST method is preferable than GET since I don't know about API specification for the HTTP endpoint. Executing "curl -POST " actually does GET request. I believe we have to '-X' to specify HTTP method with curl: # curl with POST
curl -XPOST -i https://www.data.gouv.fr/s/resources/elections-presidentielles-1965-2012-1/20150204-183729/cdsp_presi2007t2_circ.csv
# Result is the same as you got with NiFi
HTTP/1.1 405 Not Allowed So, if you'd like to download a CSV from that URL, I'd recommend to use GET HTTP Method instead.
... View more
05-23-2017
11:45 AM
You mean how to map the attribute value (term1,term2,term3) to FlowFile content? If so, I'd use ReplaceText processor, set Replacement Value as "${attribute-name}" and Replacement Strategy as "Always Replace".
... View more
05-23-2017
12:51 AM
1 Kudo
Hello, I think you can do that with EvaluateJsonPath extracting String in an JSON array, followed by UpdateAttribute removes special characters such as double quotes and square brackets by Expression Language. Here is an example NiFi template that does above conversion: https://gist.github.com/ijokarumawak/bd37690ee8aa0bf8e9e2a47cf064e0d4
... View more
04-12-2017
04:11 PM
1 Kudo
@Vadim Vaks I'm trying to do the same thing with Atlas 0.8. But I can't delete entries within inputs or outputs array with this method. With V2 API, elements didn't change. With V1 API, new elements are added even if I removed some from inputs array. The inputs had two entries before POST request, and I posted a single input entry and it got added: "inputs": [
{
"guid": "688ed1ee-222c-4416-8bf4-ba107b7fbc2c",
"typeName": "kafka_topic"
},
{
"guid": "bf3784db-fa59-4803-ad41-c5653f242f6f",
"typeName": "kafka_topic"
},
{
"guid": "688ed1ee-222c-4416-8bf4-ba107b7fbc2c",
"typeName": "kafka_topic"
}
], Please let me know how to remove elements from inputs/outputs with Atlas 0.8. Thanks!
... View more
03-31-2017
02:23 AM
@fazila when i gave command arg in execute process /root/iot-truck-streaming, i m able to see nifi data in /root/nifi_output/truck_events Does this mean ExecuteProcess was able to run the shell command but output data is written in /root/iot-truck-streaming directory? The generate shell doesn't need an argument. I haven't tried to pass an argument but IIRC it generates data to its standard output, and NiFi ExecuteProcess will read data from that standard output, then create FlowFiles containing the generated data. No FlowFile was generated when you only specify the shell path at 'Command Arguments' processor property? Also, just in case, since 'Batch Duration' is 10 sec, you need to leave ExecuteProcess running at least for 10 sec to see generated FlowFile.
... View more
03-28-2017
10:44 PM
@fazila Would you share NiFi UI screenshot around data ingestion part? Also, please try the steps explained at the answer on this thread above, "Redirect Error Stream' to capture error output of the process to see what is happening, as shown in the attached image." to see if the generate.sh have issue.
... View more