- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Nifi executeprocess does not generate realtime data
- Labels:
-
Apache NiFi
Created 10-17-2016 06:14 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, i have questions. I'm followed the REALTIME EVENT PROCESSING IN HADOOP WITH NIFI, KAFKA AND STORM and got stuck. Why execute process in Nifi does not generate realtime data although i can execute /root/iot-truck-streaming/stream-simulator/generate.sh from shell. I used the template in that tutorial and also i followed that tutorial.
Another is why i use template from ANALYZE TRAFFIC PATTERNS WITH APACHE NIFI . I can run it. But i don't see the exact output file in output folder. I just can see "trafficLocs_data_for_simulator.zip_raw=true" file in output folder although i have waited for long time.
Thanks!
Created on 10-17-2016 07:11 AM - edited 08-19-2019 02:09 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Sorry to hear that you're having trouble with our tutorial. I think there must be some error happening when the ExecuteProcess runs the shell script. But the processor ignores the stderr stream of the process by default, and currently, no bulletin error or log message are shown, then it's difficult to investigate what went wrong.
I think there's some room for improvement for this, and going to look further. In a meanwhile, please use 'Redirect Error Stream' to capture error output of the process to see what is happening, as shown in the attached image.
Thanks!
Created 10-18-2016 06:19 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I followed the tutorial once again and found what caused this.
The tutorial instructs to install NiFi via Ambari. Then the NiFi process is started by 'nifi' user on sandbox.
On the other hand, it instructs to clone the 'iot-truck-streaming' project from github, under 'root' home directory (/root).
So, even if we add execute permission to the sh file, nifi user can't access anything under /root directory.
Copying /root/iot-truck-streaming dir to /home/nifi would be a work-around:
# ssh into sandbox with root user, then: cp -rp /root/iot-truck-streaming /home/nifi
Then, specify:
/home/nifi/iot-truck-streaming/stream-simulator/generate.sh
as 'Command Arguments' on NiFi.
Created 12-20-2017 12:40 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thanks it worked!!
Created 10-18-2016 08:51 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks and it helps. Appreciate your support.
Created 03-28-2017 07:34 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@kkawamura i have run
- cp -rp /root/iot-truck-streaming /home/nifi and changed the command arg but still after completion of storm deployment there is no data emitted from kafka spout maybe because nifi does not ingest data in kafka. pls help i m stuck on it
Created 03-28-2017 07:46 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@kkawamura i have referred this link https://github.com/hortonworks-gallery/tutorials/blob/master/2015-09-25-processing-real-time-event-s... for working on kafka and storm and for ingestion in hbase and hive followed following steps of link http://saptak.in/writing/2015/09/24/ingesting-real-time-streams-hive-hbase
Created 03-28-2017 10:44 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Would you share NiFi UI screenshot around data ingestion part? Also, please try the steps explained at the answer on this thread above, "Redirect Error Stream' to capture error output of the process to see what is happening, as shown in the attached image." to see if the generate.sh have issue.
Created on 03-29-2017 04:44 AM - edited 08-19-2019 02:08 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@kkawamura my nifi service home path is /home/nifi-0.6.0.1.2.0.1-1 and i have pasted iot-streaming in /home/nifi this is shown
and also when i gave command arg in execute process /root/iot-truck-streaming, i m able to see nifi data in /root/nifi_output/truck_events. execute-process.png
Created 03-31-2017 02:23 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
when i gave command arg in execute process /root/iot-truck-streaming, i m able to see nifi data in /root/nifi_output/truck_events
Does this mean ExecuteProcess was able to run the shell command but output data is written in /root/iot-truck-streaming directory?
The generate shell doesn't need an argument. I haven't tried to pass an argument but IIRC it generates data to its standard output, and NiFi ExecuteProcess will read data from that standard output, then create FlowFiles containing the generated data.
No FlowFile was generated when you only specify the shell path at 'Command Arguments' processor property?
Also, just in case, since 'Batch Duration' is 10 sec, you need to leave ExecuteProcess running at least for 10 sec to see generated FlowFile.
- « Previous
-
- 1
- 2
- Next »