How to download JSON files from live feed?

[#1]

I have looked in to the following template by @Matt Burgess, which works fine for storing the data on my machine.

https://gist.github.com/mattyb149/6c09608b09ea07639e1a

I have tried to save json file from https://thingspeak.com/channels/64519/feed.json.

From InvokeHTTP ---> PutFile is not receiving any data.

I am new to NIFI, i might be missing something.

[#2]

First, to get insights into what is happening you can tail -f /var/log/nifi/nifi-app.log. You can grep InvokeHTTP and PutFile to get the details for each of these processors.

Second you can right click each of these processors and view stats. That will show you bytes read and written. This should give you insights into where data is flowing and where it is not.

Your InvokeHTTP processor I believe is repeatedly reading from the url and sending flowfiles (as determined by Scheduling > Run Schedule property). This means it will be repeatedly writing to the local storage. Set the PutFile property Conflict Resolution to ignore and see what happens. (And again, use the log file for details on what is happening for each processor)

[#3]

Hi @gkeys,

Thanks for a reply.

I have tried looking in to logs , it is not much of help. Process gets killed after some iterations, nothing else from logs.

jay@jay-HP-EliteBook-Folio-9470m:/usr/local/nifi/logs$ tail -f nifi-app.log | grep InvokeHTTP 2016-08-22 19:11:45,952 INFO [NiFi Web Server-186] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.standard.InvokeHTTP 2016-08-22 19:11:45,954 INFO [StandardProcessScheduler Thread-6] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling InvokeHTTP[id=7544b184-8ee4-47b8-a9ec-b13f7d9fb31a] to run

jay@jay-HP-EliteBook-Folio-9470m:/usr/local/nifi/logs$ tail -f nifi-app.log | grep PutFile 2016-08-22 19:11:45,954 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PutFile[id=83fa4ab2-778b-4017-83e0-d9171820b942] to run

Data always gets dropped at InvokeHTTP.

In InvokeHTTP it shows as 16bytes read data for each iteration, 0 bytes for write data.

In data for Invoke is 16bytes , and out it is 0.

In PutFile everything is 0.

[#4]

Works for me with a valid SSLContext. https://community.hortonworks.com/users/12595/jayachanderit.html @Jay See

Do you have an SSLContext? Your URL is SSL, you need a working SSL connection and no firewall blocking it somewhere.

See here: https://community.hortonworks.com/content/kbentry/47854/accessing-facebook-page-data-from-apache-nif...

Works fine.

-rw-r--r-- 1 root root 17334 2016-08-22 21:25 21318561924230 [root@sandbox jdk1.8.0_91]# cat /opt/demo/data/21318561924230 {"channel":{"id":64519,"name":"Hessle IOT","description":"I am building many sensors using nodemcu and lua trying to connect everything together..\r\nHave a look at my other thingspeak pages:\r\nhttps://thingspeak.com/channels/145827\r\n\r\n\r\n\r\n","latitude":"53.723805","longitude":"-0.4319","field1":"Inside Temperature","field2":"Inside Humidity","field3":"Outside Light Level","field4":"Outside temp","field5":"Outside Humdity","field6":"Energy usage","field7":"Rain sensor","field8":"Inside gas level","created_at":"2015-11-07T14:22:02Z","updated_at":"2016-08-22T21:23:02Z","elevation":"25","last_entry_id":19221},"feeds":[{"created_at":"2016-08-22T19:43:02Z","entry_id":19122,"field1":"22","field2":"47","field3":"10.83","field4":"18.4","field5":"81.1","field6":"300","field7":"1","field8":"98"},{"created_at":"2016-08-22T19:44:02Z","entry_id":19123,"field1":"25","field2":"45","field3":"9.166","field4":"18.4","field5":"81.1","field6":"300","field7":"1","field8":"97"},{"created_at":"2016-08-22T19:45:03Z","entry_id":19124,"field1":"22","field2":"47","field3":"7.5","field4":"18.3","field5":"81.1","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:46:02Z","entry_id":19125,"field1":"24","field2":"46","field3":"6.666","field4":"18.3","field5":"81.2","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:47:03Z","entry_id":19126,"field1":"22","field2":"47","field3":"5.833","field4":"18.3","field5":"81.1","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:48:02Z","entry_id":19127,"field1":"22","field2":"47","field3":"5","field4":"18.3","field5":"81.2","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:49:02Z","entry_id":19128,"field1":"23","field2":"56","field3":"4.166","field4":"18.2","field5":"81.1","field6":"360","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:50:02Z","entry_id":19129,"field1":"22","field2":"47","field3":"3.333","field4":"18.3","field5":"81.4","field6":"300","field7":"1","field8":"97"},{"created_at":"2016-08-22T19:51:02Z","entry_id":19130,"field1":"22","field2":"47","field3":"2.5","field4":"18.2","field5":"81.2","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:52:02Z","entry_id":19131,"field1":"22","field2":"47","field3":"2.5","field4":"18.2","field5":"81.3","field6":"300","field7":"1","field8":"96"},{"created_at":"2016-08-22T19:53:02Z","entry_id":19132,"field1":"22","field2":"47","field3":"1.666","field4":"18.2","field5":"81.4","field6":"300","field7":"1","field8":"94"},{"created_at":"2016-08-22T19:54:03Z","entry_id":19133,"field1":"22","field2":"47","field3":"1.666","field4":"18.1","field5":"81.4","field6":"360","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:55:02Z","entry_id":19134,"field1":"22","field2":"47","field3":"0.833","field4":"18.2","field5":"81.4","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:56:03Z","entry_id":19135,"field1":"22","field2":"47","field3":"0.833","field4":"18.1","field5":"81.5","field6":"300","field7":"1","field8":"95"},{"created_at":"2016-08-22T19:57:02Z","entry_id":19136,"field1":"22","field2":"47","field3":"0.833","field4":"18.1","field5":"81.5","field6":"300","field7":"1","field8":"94"},{"created_at":"2016-08-22T19:58:03Z","entry_id":19137,"field1":"22","field2":"46","field3":"0.833","field4":"18.1","field5":"81.6","field6":"300","field7":"1","field8":"94"},{"created_at":"2016-08-22T19:59:02Z","entry_id":19138,"field1":"22","field2":"46","field3":"0","field4":"18","field5":"81.6","field6":"300","field7":"1","field8":"93"},{"created_at":"2016-08-22T20:00:03Z","entry_id":19139,"field1":"22","field2":"46","field3":"0","field4":"18","field5":"81.7","field6":"360","field7":"1","field8":"92"},{"created_at":"2016-08-22T20:01:02Z","entry_id":19140,"field1":"22","field2":"46","field3":"0","field4":"18","field5":"81.8","field6":"300","field7":"1","field8":"92"},{"created_at":"2016-08-22T20:02:02Z","entry_id":1914

[#5]

password is changeit http://certificate.fyicenter.com/120_Java_VM_Password_for_cacerts__Java_System_Keystore.html unless you changed the default for the JVM

[#6]

Thanks for detail instructions @Timothy Spann.

I have been trying to create SSL certificate. But I am getting following an errors, which am not able to fix. I have deleted and installed java. Still I am getting same error.

I have even created SSL certificate as discussed in the page http://www.batchiq.com/nifi-configuring-ssl-auth.html.

[#7]

make sure you are using the JVM that is running HDF/NiFi. Also make sure that the NiFi user has read and execute Linux permissions.

/opt/jdk1.8.0_91/jre/lib/security/cacerts

Also I am not sure you want the Oracle JVM, perhaps the full JDK http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Are you on Ubuntu? run nifi.sh status or sudo service nifi status and see what JDK it is using. That is the one that needs the cert.

[#8]

/opt/jdk1.8.0_91/bin/keytool -list -v -keystore /opt/jdk1.8.0_91/jre/lib/security/cacerts

Use the same JDK as NIFI for keytool

it will ask for the changeit password

https://www.sslshopper.com/article-most-common-java-keytool-keystore-commands.html

[#9]

Second option didn't work. I have installed jdk, need to check whether that might solve the issue.

[#10 · Accepted solution]

@Jay See I could get it working without SSLContext Service. Please see attached. Can you please post screenshot of your InvokeHttp processor with configuration ?