question Re: How to download JSON files from live feed? in Support Questions

How to download JSON files from live feed?

jayachander_it — Sat, 20 Aug 2016 18:52:12 GMT

I have looked in to the following template by @Matt Burgess, which works fine for storing the data on my machine.

https://gist.github.com/mattyb149/6c09608b09ea07639e1a

I have tried to save json file from https://thingspeak.com/channels/64519/feed.json.

From InvokeHTTP ---> PutFile is not receiving any data.

I am new to NIFI, i might be missing something.

Re: How to download JSON files from live feed?

gkeys — Sat, 20 Aug 2016 22:28:19 GMT

First, to get insights into what is happening you can tail -f /var/log/nifi/nifi-app.log. You can grep InvokeHTTP and PutFile to get the details for each of these processors.

Second you can right click each of these processors and view stats. That will show you bytes read and written. This should give you insights into where data is flowing and where it is not.

Your InvokeHTTP processor I believe is repeatedly reading from the url and sending flowfiles (as determined by Scheduling > Run Schedule property). This means it will be repeatedly writing to the local storage. Set the PutFile property Conflict Resolution to ignore and see what happens. (And again, use the log file for details on what is happening for each processor)

Re: How to download JSON files from live feed?

jayachander_it — Tue, 23 Aug 2016 00:55:32 GMT

Hi @gkeys,

Thanks for a reply.

I have tried looking in to logs , it is not much of help. Process gets killed after some iterations, nothing else from logs.

jay@jay-HP-EliteBook-Folio-9470m:/usr/local/nifi/logs$ tail -f nifi-app.log | grep InvokeHTTP 2016-08-22 19:11:45,952 INFO [NiFi Web Server-186] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.standard.InvokeHTTP 2016-08-22 19:11:45,954 INFO [StandardProcessScheduler Thread-6] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling InvokeHTTP[id=7544b184-8ee4-47b8-a9ec-b13f7d9fb31a] to run

jay@jay-HP-EliteBook-Folio-9470m:/usr/local/nifi/logs$ tail -f nifi-app.log | grep PutFile 2016-08-22 19:11:45,954 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PutFile[id=83fa4ab2-778b-4017-83e0-d9171820b942] to run

Data always gets dropped at InvokeHTTP.

In InvokeHTTP it shows as 16bytes read data for each iteration, 0 bytes for write data.

In data for Invoke is 16bytes , and out it is 0.

In PutFile everything is 0.

Re: How to download JSON files from live feed?

TimothySpann — Mon, 19 Aug 2019 11:13:09 GMT

Works for me with a valid SSLContext. https://community.hortonworks.com/users/12595/jayachanderit.html @Jay See

Do you have an SSLContext? Your URL is SSL, you need a working SSL connection and no firewall blocking it somewhere.

See here: https://community.hortonworks.com/content/kbentry/47854/accessing-facebook-page-data-from-apache-nifi.html

Works fine.

-rw-r--r-- 1 root root 17334 2016-08-22 21:25 21318561924230 [root@sandbox jdk1.8.0_91]# cat /opt/demo/data/21318561924230 {"channel":{"id":64519,"name":"Hessle IOT","description":"I am building many sensors using nodemcu and lua trying to connect everything together..\r\nHave a look at my other thingspeak pages:\r\nhttps://thingspeak.com/channels/145827\r\n\r\n\r\n\r\n","latitude":"53.723805","longitude":"-0.4319","field1":"Inside Temperature","field2":"Inside Humidity","field3":"Outside Light Level","field4":"Outside temp","field5":"Outside Humdity","field6":"Energy usage","field7":"Rain sensor","field8":"Inside gas level","created_at":"2015-11-07T14:22:02Z","updated_at":"2016-08-22T21:23:02Z","elevation":"25","last_entry_id":19221},"feeds":[{"created_at":"2016-08-22T19:43:02Z","entry_id":19122,"field1":"22","field2":"47","field3":"10.83","field4":"18.4","field5":"81.1","field6":"300","field7":"1","field8":"98"},{"created_at":"2016-08-22T19:44:02Z","entry_id":19123,"field1":"25","field2":"45","field3":"9.166","field4":"18.4","field5":"81.1","field6":"300","field7":"1","field8":"97"},{"created_at":"2016-08-22T19:45:03Z","entry_id":19124,"field1":"22","field2":"47","field3":"7.5","field4":"18.3","field5":"81.1","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:46:02Z","entry_id":19125,"field1":"24","field2":"46","field3":"6.666","field4":"18.3","field5":"81.2","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:47:03Z","entry_id":19126,"field1":"22","field2":"47","field3":"5.833","field4":"18.3","field5":"81.1","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:48:02Z","entry_id":19127,"field1":"22","field2":"47","field3":"5","field4":"18.3","field5":"81.2","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:49:02Z","entry_id":19128,"field1":"23","field2":"56","field3":"4.166","field4":"18.2","field5":"81.1","field6":"360","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:50:02Z","entry_id":19129,"field1":"22","field2":"47","field3":"3.333","field4":"18.3","field5":"81.4","field6":"300","field7":"1","field8":"97"},{"created_at":"2016-08-22T19:51:02Z","entry_id":19130,"field1":"22","field2":"47","field3":"2.5","field4":"18.2","field5":"81.2","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:52:02Z","entry_id":19131,"field1":"22","field2":"47","field3":"2.5","field4":"18.2","field5":"81.3","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:53:02Z","entry_id":19132,"field1":"22","field2":"47","field3":"1.666","field4":"18.2","field5":"81.4","field6":"300","field7":"1","field8":"94"},{"created_at":"2016-08-22T19:54:03Z","entry_id":19133,"field1":"22","field2":"47","field3":"1.666","field4":"18.1","field5":"81.4","field6":"360","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:55:02Z","entry_id":19134,"field1":"22","field2":"47","field3":"0.833","field4":"18.2","field5":"81.4","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:56:03Z","entry_id":19135,"field1":"22","field2":"47","field3":"0.833","field4":"18.1","field5":"81.5","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:57:02Z","entry_id":19136,"field1":"22","field2":"47","field3":"0.833","field4":"18.1","field5":"81.5","field6":"300","field7":"1","field8":"94"},{"created_at":"2016-08-22T19:58:03Z","entry_id":19137,"field1":"22","field2":"46","field3":"0.833","field4":"18.1","field5":"81.6","field6":"300","field7":"1","field8":"94"},{"created_at":"2016-08-22T19:59:02Z","entry_id":19138,"field1":"22","field2":"46","field3":"0","field4":"18","field5":"81.6","field6":"300","field7":"1","field8":"93"},{"created_at":"2016-08-22T20:00:03Z","entry_id":19139,"field1":"22","field2":"46","field3":"0","field4":"18","field5":"81.7","field6":"360","field7":"1","field8":"92"},{"created_at":"2016-08-22T20:01:02Z","entry_id":19140,"field1":"22","field2":"46","field3":"0","field4":"18","field5":"81.8","field6":"300","field7":"1","field8":"92"},{"created_at":"2016-08-22T20:02:02Z","entry_id":1914

Re: How to download JSON files from live feed?

TimothySpann — Tue, 23 Aug 2016 04:19:59 GMT

password is changeit http://certificate.fyicenter.com/120_Java_VM_Password_for_cacerts__Java_System_Keystore.html unless you changed the default for the JVM

Re: How to download JSON files from live feed?

jayachander_it — Mon, 19 Aug 2019 11:12:49 GMT

Thanks for detail instructions @Timothy Spann.

I have been trying to create SSL certificate. But I am getting following an errors, which am not able to fix. I have deleted and installed java. Still I am getting same error.

I have even created SSL certificate as discussed in the page http://www.batchiq.com/nifi-configuring-ssl-auth.html.

Re: How to download JSON files from live feed?

shishir_saxena4 — Tue, 23 Aug 2016 23:13:25 GMT

@Jay See I could get it working without SSLContext Service. Please see attached. Can you please post screenshot of your InvokeHttp processor with configuration ?

Re: How to download JSON files from live feed?

TimothySpann — Tue, 23 Aug 2016 23:24:54 GMT

make sure you are using the JVM that is running HDF/NiFi. Also make sure that the NiFi user has read and execute Linux permissions.

/opt/jdk1.8.0_91/jre/lib/security/cacerts

Also I am not sure you want the Oracle JVM, perhaps the full JDK http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Are you on Ubuntu? run nifi.sh status or sudo service nifi status and see what JDK it is using. That is the one that needs the cert.

Re: How to download JSON files from live feed?

TimothySpann — Tue, 23 Aug 2016 23:27:50 GMT

/opt/jdk1.8.0_91/bin/keytool -list -v -keystore /opt/jdk1.8.0_91/jre/lib/security/cacerts

Use the same JDK as NIFI for keytool

it will ask for the changeit password

https://www.sslshopper.com/article-most-common-java-keytool-keystore-commands.html

Re: How to download JSON files from live feed?

jayachander_it — Wed, 24 Aug 2016 00:11:50 GMT

@Shishir Saxena, I haven't done much. I have just used the code from here https://gist.github.com/mattyb149/6c09608b09ea07639e1a.

Changed the link. I have tried similar to your example as well, it is not working on my machine.

Re: How to download JSON files from live feed?

jayachander_it — Wed, 24 Aug 2016 02:44:45 GMT

Second option didn't work. I have installed jdk, need to check whether that might solve the issue.