question Re: Can we ingest Rss feeds using apache nifi into hive? in Archives of Support Questions (Read Only)

Can we ingest Rss feeds using apache nifi into hive?

bigspark — Fri, 22 Jan 2016 17:45:33 GMT

Any thoughts on which processors to use in nifi for ingesting rss feeds into hive tables.

http://feeds.bbci.co.uk/news/world/rss.xml

thanks.

Re: Can we ingest Rss feeds using apache nifi into hive?

nsabharwal — Fri, 22 Jan 2016 19:55:19 GMT

@surender nath reddy kudumula

Use this GetHTTP

http://www.nifi.rocks/getting-started-with-apache-nifi/

The URL we are going to grab a file from is XKCD’s rss feed, rss.xml. The GetHTTP procesor is simple to configure and just needs the URL property to be set to http://xkcd.com/rss.xml. Click on the value across from the URL property and enter http://xkcd.com/rss.xml.

Now drag down another processor, EvaluateXPath. Under the properties for this processor, set the following property-value pairs:

Destination - flowfile-attribute
Return Type - auto-detect

Re: Can we ingest Rss feeds using apache nifi into hive?

bigspark — Fri, 22 Jan 2016 23:18:49 GMT

Thank you @Neeraj Sabharwal. I ll give this a try. Why do we need EvaluateXPath here? Also i beleive at the end use puthdfs to route the data into hdfs. I beleive currently nifi doesnt have a processor to ingest directly into hive table i can see it has putsql. So i beleive best approach is to use puthdfs. Please let me know thanks..

Re: Can we ingest Rss feeds using apache nifi into hive?

nsabharwal — Fri, 22 Jan 2016 23:20:09 GMT

@surender nath reddy kudumula I will put data in HDFS and create external Hive table on top of it.

Re: Can we ingest Rss feeds using apache nifi into hive?

bigspark — Sat, 23 Jan 2016 02:21:09 GMT

thank you @Neeraj Sabharwal

Re: Can we ingest Rss feeds using apache nifi into hive?

nsabharwal — Sat, 23 Jan 2016 03:38:51 GMT

@surender nath reddy kudumula As part of the best practice, please accept the best answer to close the thread.

Re: Can we ingest Rss feeds using apache nifi into hive?

james_jones — Fri, 22 Apr 2016 03:14:31 GMT

@surender nath reddy kudumula - There is a JIRA actively being worked on to add Hive JDBC support to Nifi. https://issues.apache.org/jira/browse/NIFI-981

Re: Can we ingest Rss feeds using apache nifi into hive?

paula_ditallo — Sat, 17 Dec 2016 08:49:35 GMT

Neeraj, thanks for your advice.