- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Nifi executeprocess does not generate realtime data
- Labels:
-
Apache NiFi
Created ‎10-17-2016 06:14 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, i have questions. I'm followed the REALTIME EVENT PROCESSING IN HADOOP WITH NIFI, KAFKA AND STORM and got stuck. Why execute process in Nifi does not generate realtime data although i can execute /root/iot-truck-streaming/stream-simulator/generate.sh from shell. I used the template in that tutorial and also i followed that tutorial.
Another is why i use template from ANALYZE TRAFFIC PATTERNS WITH APACHE NIFI . I can run it. But i don't see the exact output file in output folder. I just can see "trafficLocs_data_for_simulator.zip_raw=true" file in output folder although i have waited for long time.
Thanks!
Created on ‎10-17-2016 07:11 AM - edited ‎08-19-2019 02:09 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Sorry to hear that you're having trouble with our tutorial. I think there must be some error happening when the ExecuteProcess runs the shell script. But the processor ignores the stderr stream of the process by default, and currently, no bulletin error or log message are shown, then it's difficult to investigate what went wrong.
I think there's some room for improvement for this, and going to look further. In a meanwhile, please use 'Redirect Error Stream' to capture error output of the process to see what is happening, as shown in the attached image.
Thanks!
Created on ‎10-17-2016 07:11 AM - edited ‎08-19-2019 02:09 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Sorry to hear that you're having trouble with our tutorial. I think there must be some error happening when the ExecuteProcess runs the shell script. But the processor ignores the stderr stream of the process by default, and currently, no bulletin error or log message are shown, then it's difficult to investigate what went wrong.
I think there's some room for improvement for this, and going to look further. In a meanwhile, please use 'Redirect Error Stream' to capture error output of the process to see what is happening, as shown in the attached image.
Thanks!
Created ‎10-17-2016 07:25 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ok, here is error: "sh: /root/iot-truck-streaming/stream-simulator/generate.sh: Permission denied. "
So, how can i do?. Thanks!
Created ‎10-17-2016 07:31 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
To add executable permission to the file to every user (since I am not sure which user runs NiFi process on your environment), use following command:
chmod +x /root/iot-truck-streaming/stream-simulator/generate.sh
You might need to add sudo at the beginning of the command based on who you are logged in with:
sudo chmod +x /root/iot-truck-streaming/stream-simulator/generate.sh
Hope it works.
(Sorry, I pasted the above command wrong, updated.)
Created ‎10-17-2016 08:21 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, i don't have a clue how to create nifi user, currently there is no nifi user being showed here. I see in ambari dashboard on nifi config that nifi_user and nifi_group is both nifi!?. What do you mean" add executable permission :
sudo /root/iot-truck-streaming/stream-simulator/generate.sh
". You mean add it to shell or to processor property?.
Created ‎10-17-2016 09:34 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, i've follow your guide, executed two commands above, but there's nothing changes. Currently i'm running as anonymous user. I don't know how to create secure user. I think it relevant to the problem. I look for guide to do that but it is complicate.
Created ‎10-17-2016 11:36 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, I posted the comment without 'chmod +x' command above.. sorry about that. Please login to the sandbox machine by SSH and execute 'chmod +x /root/iot-truck-streaming/stream-simulator/generate.sh' command to add execution permission to the shell script file. Then NiFi will be able to run the sh from ExecuteProcess.
Created ‎10-17-2016 11:59 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I tried options of chmod and also chown to nifi user but error still exists even after full 777 permission on file and also folder. Please advice.
Created ‎10-18-2016 12:13 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I did read this thread and did all suggested options but error still exists as permission denied. Please advice.
Created ‎10-18-2016 12:19 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Mothilal marimuthu thanks for sharing that. I'll check it again to see if there's anything else needed.
