su hdfs hadoop fs -mkdir /udf hadoop fs -put urldetector-1.0-jar-with-dependencies.jar /udf/ hadoop fs -put libs/url-detector-0.1.15.jar /udf/ hadoop fs -chown -R hdfs /udf hadoop fs -chgrp -R hdfs /udf hadoop fs -chmod -R 775 /udf Create Hadoop Directories and upload the two necessary libraries. CREATE FUNCTION urldetector as 'com.dataflowdeveloper.detection.URLDetector' USING JAR 'hdfs:///udf/urldetector-1.0-jar-with-dependencies.jar', JAR 'hdfs:///udf/url-detector-0.1.15.jar'; Create Hive Function with those HDFS referenced JARs select http_user_agent,urldetector(remote_host)asurls,remote_host from AccessLogs limit 100; Test the UDF via Hive QL @Description(name="urldetector", value="_FUNC_(string) - detectsurls") public final class URLDetector extends UDF{} Java Header for the UDF set hive.cli.print.header=true; add jar urldetector-1.0-jar-with-dependencies.jar;CREATE TEMPORARY FUNCTION urldetector as 'com.dataflowdeveloper.detection.URLDetector';select urldetector(description) from sample_07 limit 100; You can test with a temporary function through Hive CLI before making the function permanent. mvn compile assembly:single Build the Jar File for Deployment The library from LinkedIn (https://github.com/linkedin/URL-Detector) must be compiled and the JAR used in your code and deployed to Hive. References See: https://github.com/tspannhw/URLDetector for full source code. ... View more

Adding HDF (with Apache NiFi) to your HDP 2.5 Sandbox is very quick, painless and easy. Get the most recent Hortonworks DataFlow (download😞 wget http://d3d0kdwqv675cq.cloudfront.net/HDF/centos6/1.x/updates/1.2.0.1/HDF-1.2.0.1-1.tar.gz tar -xvf HDF-1.2.0.1-1.tar.gz cd HDF-1.2.0.1-1/nifi/ Then change the port used by NiFi in the conf/nifi.properties file to: nifi.web.http.port=8090 Install NiFi as a Linux Service bin/nifi.sh install sudo service nifi start NiFi home: /opt/HDF-1.2.0.1-1/nifi Bootstrap Config File: /opt/HDF-1.2.0.1-1/nifi/conf/bootstrap.conf 2016-07-04 02:18:00,005 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2016-07-04 02:18:00,006 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /opt/HDF-1.2.0.1-1/nifi You can check the status of single NiFi server via status command: [root@sandbox nifi]# sudo service nifi status nifi.sh: JAVA_HOME not set; results may vary Java home: NiFi home: /opt/HDF-1.2.0.1-1/nifi Bootstrap Config File: /opt/HDF-1.2.0.1-1/nifi/conf/bootstrap.conf 2016-07-04 02:18:42,527 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is currently running, listening to Bootstrap on port 43184, PID=4391 Make sure you add port 8090 to the sandbox networking. You are now ready to go. Now start flowing. ... View more