Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to download JSON files from live feed?

avatar

I have looked in to the following template by @Matt Burgess, which works fine for storing the data on my machine.

https://gist.github.com/mattyb149/6c09608b09ea07639e1a

I have tried to save json file from https://thingspeak.com/channels/64519/feed.json.

From InvokeHTTP ---> PutFile is not receiving any data.

I am new to NIFI, i might be missing something.

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
10 REPLIES 10

avatar
Guru

First, to get insights into what is happening you can tail -f /var/log/nifi/nifi-app.log. You can grep InvokeHTTP and PutFile to get the details for each of these processors.

Second you can right click each of these processors and view stats. That will show you bytes read and written. This should give you insights into where data is flowing and where it is not.

Your InvokeHTTP processor I believe is repeatedly reading from the url and sending flowfiles (as determined by Scheduling > Run Schedule property). This means it will be repeatedly writing to the local storage. Set the PutFile property Conflict Resolution to ignore and see what happens. (And again, use the log file for details on what is happening for each processor)

avatar

Hi @gkeys,

Thanks for a reply.

I have tried looking in to logs , it is not much of help. Process gets killed after some iterations, nothing else from logs.

jay@jay-HP-EliteBook-Folio-9470m:/usr/local/nifi/logs$ tail -f nifi-app.log | grep InvokeHTTP 2016-08-22 19:11:45,952 INFO [NiFi Web Server-186] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.standard.InvokeHTTP 2016-08-22 19:11:45,954 INFO [StandardProcessScheduler Thread-6] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling InvokeHTTP[id=7544b184-8ee4-47b8-a9ec-b13f7d9fb31a] to run

jay@jay-HP-EliteBook-Folio-9470m:/usr/local/nifi/logs$ tail -f nifi-app.log | grep PutFile 2016-08-22 19:11:45,954 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PutFile[id=83fa4ab2-778b-4017-83e0-d9171820b942] to run

Data always gets dropped at InvokeHTTP.

In InvokeHTTP it shows as 16bytes read data for each iteration, 0 bytes for write data.

In data for Invoke is 16bytes , and out it is 0.

In PutFile everything is 0.

avatar
Master Guru

Works for me with a valid SSLContext. https://community.hortonworks.com/users/12595/jayachanderit.html @Jay See

Do you have an SSLContext? Your URL is SSL, you need a working SSL connection and no firewall blocking it somewhere.

See here: https://community.hortonworks.com/content/kbentry/47854/accessing-facebook-page-data-from-apache-nif...

6862-help2.png

6863-help3.png

6861-help1.png

Works fine.

-rw-r--r-- 1 root root 17334 2016-08-22 21:25 21318561924230 [root@sandbox jdk1.8.0_91]# cat /opt/demo/data/21318561924230 {"channel":{"id":64519,"name":"Hessle IOT","description":"I am building many sensors using nodemcu and lua trying to connect everything together..\r\nHave a look at my other thingspeak pages:\r\nhttps://thingspeak.com/channels/145827\r\n\r\n\r\n\r\n","latitude":"53.723805","longitude":"-0.4319","field1":"Inside Temperature","field2":"Inside Humidity","field3":"Outside Light Level","field4":"Outside temp","field5":"Outside Humdity","field6":"Energy usage","field7":"Rain sensor","field8":"Inside gas level","created_at":"2015-11-07T14:22:02Z","updated_at":"2016-08-22T21:23:02Z","elevation":"25","last_entry_id":19221},"feeds":[{"created_at":"2016-08-22T19:43:02Z","entry_id":19122,"field1":"22","field2":"47","field3":"10.83","field4":"18.4","field5":"81.1","field6":"300","field7":"1","field8":"98"},{"created_at":"2016-08-22T19:44:02Z","entry_id":19123,"field1":"25","field2":"45","field3":"9.166","field4":"18.4","field5":"81.1","field6":"300","field7":"1","field8":"97"},{"created_at":"2016-08-22T19:45:03Z","entry_id":19124,"field1":"22","field2":"47","field3":"7.5","field4":"18.3","field5":"81.1","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:46:02Z","entry_id":19125,"field1":"24","field2":"46","field3":"6.666","field4":"18.3","field5":"81.2","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:47:03Z","entry_id":19126,"field1":"22","field2":"47","field3":"5.833","field4":"18.3","field5":"81.1","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:48:02Z","entry_id":19127,"field1":"22","field2":"47","field3":"5","field4":"18.3","field5":"81.2","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:49:02Z","entry_id":19128,"field1":"23","field2":"56","field3":"4.166","field4":"18.2","field5":"81.1","field6":"360","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:50:02Z","entry_id":19129,"field1":"22","field2":"47","field3":"3.333","field4":"18.3","field5":"81.4","field6":"300","field7":"1","field8":"97"},{"created_at":"2016-08-22T19:51:02Z","entry_id":19130,"field1":"22","field2":"47","field3":"2.5","field4":"18.2","field5":"81.2","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:52:02Z","entry_id":19131,"field1":"22","field2":"47","field3":"2.5","field4":"18.2","field5":"81.3","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:53:02Z","entry_id":19132,"field1":"22","field2":"47","field3":"1.666","field4":"18.2","field5":"81.4","field6":"300","field7":"1","field8":"94"},{"created_at":"2016-08-22T19:54:03Z","entry_id":19133,"field1":"22","field2":"47","field3":"1.666","field4":"18.1","field5":"81.4","field6":"360","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:55:02Z","entry_id":19134,"field1":"22","field2":"47","field3":"0.833","field4":"18.2","field5":"81.4","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:56:03Z","entry_id":19135,"field1":"22","field2":"47","field3":"0.833","field4":"18.1","field5":"81.5","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:57:02Z","entry_id":19136,"field1":"22","field2":"47","field3":"0.833","field4":"18.1","field5":"81.5","field6":"300","field7":"1","field8":"94"},{"created_at":"2016-08-22T19:58:03Z","entry_id":19137,"field1":"22","field2":"46","field3":"0.833","field4":"18.1","field5":"81.6","field6":"300","field7":"1","field8":"94"},{"created_at":"2016-08-22T19:59:02Z","entry_id":19138,"field1":"22","field2":"46","field3":"0","field4":"18","field5":"81.6","field6":"300","field7":"1","field8":"93"},{"created_at":"2016-08-22T20:00:03Z","entry_id":19139,"field1":"22","field2":"46","field3":"0","field4":"18","field5":"81.7","field6":"360","field7":"1","field8":"92"},{"created_at":"2016-08-22T20:01:02Z","entry_id":19140,"field1":"22","field2":"46","field3":"0","field4":"18","field5":"81.8","field6":"300","field7":"1","field8":"92"},{"created_at":"2016-08-22T20:02:02Z","entry_id":1914

avatar
Master Guru

password is changeit http://certificate.fyicenter.com/120_Java_VM_Password_for_cacerts__Java_System_Keystore.html unless you changed the default for the JVM

avatar

Thanks for detail instructions @Timothy Spann.

I have been trying to create SSL certificate. But I am getting following an errors, which am not able to fix. I have deleted and installed java. Still I am getting same error.

I have even created SSL certificate as discussed in the page http://www.batchiq.com/nifi-configuring-ssl-auth.html.

6870-2.jpg

6869-1.jpg

avatar
Master Guru

make sure you are using the JVM that is running HDF/NiFi. Also make sure that the NiFi user has read and execute Linux permissions.

/opt/jdk1.8.0_91/jre/lib/security/cacerts

Also I am not sure you want the Oracle JVM, perhaps the full JDK http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Are you on Ubuntu? run nifi.sh status or sudo service nifi status and see what JDK it is using. That is the one that needs the cert.

avatar
Master Guru

/opt/jdk1.8.0_91/bin/keytool -list -v -keystore /opt/jdk1.8.0_91/jre/lib/security/cacerts

Use the same JDK as NIFI for keytool

it will ask for the changeit password

https://www.sslshopper.com/article-most-common-java-keytool-keystore-commands.html

avatar

Second option didn't work. I have installed jdk, need to check whether that might solve the issue.

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login