In Apache NiFi 1.2, there are processors for Reading Hive data via HiveQL and Storing to Hive via HiveQL. These processors are SelectHiveQL and PutHiveQL. Configuring a HiveQL processor is simple, you need to enter your query and pick either AVRO or CSV format. AVRO is a better fit, I am waiting for ORC. Most important you need to set a Connection Pool to connect to your cluster. You can just enter a regular SQL that you are doing in Hive. For Hive to work, you must setup a HiveConnectionPool Controller Service. After configuration you will need to enable that and then you can enable your processor(s). For connecting to Hive on the Sandbox, set the Database Connection URL: jdbc://hive2://localhost:10000/default. For Hive Configuration Resources: you set the hive configuration files. You can set the Database User and Password of the user that has access you require for Hive. For documentation on the HiveConnectionPool. For a PutHiveQL, you just need to set a connection pool, batch size for updates and a character set. The defaults for this are ok. CAVEAT: Once you have it set make sure you have all the relationships terminated somewhere either in a Sink or with auto terminate. ... View more

I just tried out the new NiFi 0.7.0 version's Slack. Source I used Twitter, since it has some fun data and gets you a nice big stream. Sometimes with Twitter Feeds will be limited and Twitter will give you the 420 Enhance Your Calm Message. https://httpstatusdogs.com/420-enhance-your-calm Usually you can just wait 5-20 minutes and you will be serving again. Sometimes you might need to use a different of your apps, reset the tokens in your app or create a new app (https://apps.twitter.com/). Processing Use the Pull Key Attributes, to Find Only Tweets (remove null) Sink to Slack For the PutToSlack Processor Set the Webhook URL to the URL generated by the incoming webhook page in slack.com. Set the Webhook Text to ${twitter.msg}, this will send your Twitter message to slack. Set the Channel to #general, or a channel of your choosing. I created a slack board for receiving my messages, https://nifi-se.slack.com/messages/general/. You can easily create your own (or using your existing Slack board). Just go to slack.com. You will need to create a webhook. To set it up, in the #general channel just type incoming webhook. You will get a link the screen to create one. Apache NiFi 0.70 Final Flow Now you can start seeing tweets turn into slack messages. Apache NiFi 0.70 now has 155 processors! Let's explore some more. ... View more

su hdfs hadoop fs -mkdir /udf hadoop fs -put urldetector-1.0-jar-with-dependencies.jar /udf/ hadoop fs -put libs/url-detector-0.1.15.jar /udf/ hadoop fs -chown -R hdfs /udf hadoop fs -chgrp -R hdfs /udf hadoop fs -chmod -R 775 /udf Create Hadoop Directories and upload the two necessary libraries. CREATE FUNCTION urldetector as 'com.dataflowdeveloper.detection.URLDetector' USING JAR 'hdfs:///udf/urldetector-1.0-jar-with-dependencies.jar', JAR 'hdfs:///udf/url-detector-0.1.15.jar'; Create Hive Function with those HDFS referenced JARs select http_user_agent,urldetector(remote_host)asurls,remote_host from AccessLogs limit 100; Test the UDF via Hive QL @Description(name="urldetector", value="_FUNC_(string) - detectsurls") public final class URLDetector extends UDF{} Java Header for the UDF set hive.cli.print.header=true; add jar urldetector-1.0-jar-with-dependencies.jar;CREATE TEMPORARY FUNCTION urldetector as 'com.dataflowdeveloper.detection.URLDetector';select urldetector(description) from sample_07 limit 100; You can test with a temporary function through Hive CLI before making the function permanent. mvn compile assembly:single Build the Jar File for Deployment The library from LinkedIn (https://github.com/linkedin/URL-Detector) must be compiled and the JAR used in your code and deployed to Hive. References See: https://github.com/tspannhw/URLDetector for full source code. ... View more