question Re: Nifi executeprocess does not generate realtime data in Support Questions

Nifi executeprocess does not generate realtime data

wealthyman2402 — Mon, 17 Oct 2016 13:14:34 GMT

Hi, i have questions. I'm followed the REALTIME EVENT PROCESSING IN HADOOP WITH NIFI, KAFKA AND STORM and got stuck. Why execute process in Nifi does not generate realtime data although i can execute /root/iot-truck-streaming/stream-simulator/generate.sh from shell. I used the template in that tutorial and also i followed that tutorial.

Another is why i use template from ANALYZE TRAFFIC PATTERNS WITH APACHE NIFI . I can run it. But i don't see the exact output file in output folder. I just can see "trafficLocs_data_for_simulator.zip_raw=true" file in output folder although i have waited for long time.

Thanks!

Re: Nifi executeprocess does not generate realtime data

kkawamura — Mon, 19 Aug 2019 09:09:04 GMT

Hello,

Sorry to hear that you're having trouble with our tutorial. I think there must be some error happening when the ExecuteProcess runs the shell script. But the processor ignores the stderr stream of the process by default, and currently, no bulletin error or log message are shown, then it's difficult to investigate what went wrong.

I think there's some room for improvement for this, and going to look further. In a meanwhile, please use 'Redirect Error Stream' to capture error output of the process to see what is happening, as shown in the attached image.

Thanks!

Re: Nifi executeprocess does not generate realtime data

wealthyman2402 — Mon, 17 Oct 2016 14:25:31 GMT

Ok, here is error: "sh: /root/iot-truck-streaming/stream-simulator/generate.sh: Permission denied. "

So, how can i do?. Thanks!

Re: Nifi executeprocess does not generate realtime data

kkawamura — Mon, 17 Oct 2016 14:31:42 GMT

Hi,

To add executable permission to the file to every user (since I am not sure which user runs NiFi process on your environment), use following command:

chmod +x /root/iot-truck-streaming/stream-simulator/generate.sh

You might need to add sudo at the beginning of the command based on who you are logged in with:

sudo chmod +x /root/iot-truck-streaming/stream-simulator/generate.sh

Hope it works.

(Sorry, I pasted the above command wrong, updated.)

Re: Nifi executeprocess does not generate realtime data

wealthyman2402 — Mon, 17 Oct 2016 15:21:51 GMT

Hi, i don't have a clue how to create nifi user, currently there is no nifi user being showed here. I see in ambari dashboard on nifi config that nifi_user and nifi_group is both nifi!?. What do you mean" add executable permission :

sudo /root/iot-truck-streaming/stream-simulator/generate.sh

". You mean add it to shell or to processor property?.

screenshot-24.png

Re: Nifi executeprocess does not generate realtime data

wealthyman2402 — Mon, 17 Oct 2016 16:34:42 GMT

Hi, i've follow your guide, executed two commands above, but there's nothing changes. Currently i'm running as anonymous user. I don't know how to create secure user. I think it relevant to the problem. I look for guide to do that but it is complicate.

Re: Nifi executeprocess does not generate realtime data

kkawamura — Tue, 18 Oct 2016 06:36:02 GMT

Hi, I posted the comment without 'chmod +x' command above.. sorry about that. Please login to the sandbox machine by SSH and execute 'chmod +x /root/iot-truck-streaming/stream-simulator/generate.sh' command to add execution permission to the shell script file. Then NiFi will be able to run the sh from ExecuteProcess.

Re: Nifi executeprocess does not generate realtime data

mothi86 — Tue, 18 Oct 2016 06:59:06 GMT

I tried options of chmod and also chown to nifi user but error still exists even after full 777 permission on file and also folder. Please advice.

Re: Nifi executeprocess does not generate realtime data

mothi86 — Tue, 18 Oct 2016 07:13:58 GMT

I did read this thread and did all suggested options but error still exists as permission denied. Please advice.

Re: Nifi executeprocess does not generate realtime data

kkawamura — Tue, 18 Oct 2016 07:19:43 GMT

Hi @Mothilal marimuthu thanks for sharing that. I'll check it again to see if there's anything else needed.

Re: Nifi executeprocess does not generate realtime data

kkawamura — Tue, 18 Oct 2016 13:19:48 GMT

Hello,

I followed the tutorial once again and found what caused this.

The tutorial instructs to install NiFi via Ambari. Then the NiFi process is started by 'nifi' user on sandbox.

On the other hand, it instructs to clone the 'iot-truck-streaming' project from github, under 'root' home directory (/root).

So, even if we add execute permission to the sh file, nifi user can't access anything under /root directory.

Copying /root/iot-truck-streaming dir to /home/nifi would be a work-around:

# ssh into sandbox with root user, then:
cp -rp /root/iot-truck-streaming /home/nifi

Then, specify:

/home/nifi/iot-truck-streaming/stream-simulator/generate.sh

as 'Command Arguments' on NiFi.

Re: Nifi executeprocess does not generate realtime data

mothi86 — Wed, 19 Oct 2016 03:51:11 GMT

Thanks and it helps. Appreciate your support.

Re: Nifi executeprocess does not generate realtime data

fazila_shaikh4 — Wed, 29 Mar 2017 02:34:53 GMT

@kkawamura i have run

cp -rp /root/iot-truck-streaming /home/nifi and changed the command arg but still after completion of storm deployment there is no data emitted from kafka spout maybe because nifi does not ingest data in kafka. pls help i m stuck on it

Re: Nifi executeprocess does not generate realtime data

fazila_shaikh4 — Wed, 29 Mar 2017 02:46:50 GMT

@kkawamura i have referred this link https://github.com/hortonworks-gallery/tutorials/blob/master/2015-09-25-processing-real-time-event-stream-with-apache-storm.md for working on kafka and storm and for ingestion in hbase and hive followed following steps of link http://saptak.in/writing/2015/09/24/ingesting-real-time-streams-hive-hbase

Re: Nifi executeprocess does not generate realtime data

kkawamura — Wed, 29 Mar 2017 05:44:35 GMT

@fazila

Would you share NiFi UI screenshot around data ingestion part? Also, please try the steps explained at the answer on this thread above, "Redirect Error Stream' to capture error output of the process to see what is happening, as shown in the attached image." to see if the generate.sh have issue.

Re: Nifi executeprocess does not generate realtime data

fazila_shaikh4 — Mon, 19 Aug 2019 09:08:56 GMT

@kkawamura my nifi service home path is /home/nifi-0.6.0.1.2.0.1-1 and i have pasted iot-streaming in /home/nifi this is shown

and also when i gave command arg in execute process /root/iot-truck-streaming, i m able to see nifi data in /root/nifi_output/truck_events. execute-process.png

Re: Nifi executeprocess does not generate realtime data

kkawamura — Fri, 31 Mar 2017 09:23:11 GMT

@fazila

when i gave command arg in execute process /root/iot-truck-streaming, i m able to see nifi data in /root/nifi_output/truck_events

Does this mean ExecuteProcess was able to run the shell command but output data is written in /root/iot-truck-streaming directory?

The generate shell doesn't need an argument. I haven't tried to pass an argument but IIRC it generates data to its standard output, and NiFi ExecuteProcess will read data from that standard output, then create FlowFiles containing the generated data.

No FlowFile was generated when you only specify the shell path at 'Command Arguments' processor property?

Also, just in case, since 'Batch Duration' is 10 sec, you need to leave ExecuteProcess running at least for 10 sec to see generated FlowFile.

Re: Nifi executeprocess does not generate realtime data

akshay_katti — Wed, 20 Dec 2017 20:40:07 GMT

thanks it worked!!