It's easier than I would have thought to add images to your SQL results tables in Apache Zeppelin. It's pretty simple to do this in HDP 2.5's version of Apache Zeppelin. You use the %html tag to output HTML instead of text. Use Case: Displaying Image with TensorFlow Inception Image Recognition Results in Same List Example SQL: SELECT user_name, handle, concat('%html <img width=50 height=60 src="', media_url, '">') as media,substring(inception,0,150) as inception, msg, sentiment, stanfordsentiment, location, time FROM twitterorc where inception not like '%Not found%' and inception is not null and trim(inception)!= '' ... View more

Use Case Before meetups, I wanted to play some music. NiFi seemed like a great choice for streaming free music through my Mac. MIDI Command Line Player For OSX brew install timidity Then you can simply play MIDI files with timidity file.mid. Microservice to Extract Links from Web Pages Java 8 Source Code: https://github.com/tspannhw/linkextractor The Spring Boot REST API accepts a URL, parses out mid files and returns JSON containing linking and descriptions. Example REST Call to Service curl -G -v "http://<urL>:8080/extract/url?url=http://www.midiworld.com/classic.htm/&type=mid" Run the Microservice java -Xms512m -Xmx2048m -Djava.net.preferIPv4Stack=true -jar target/linkextractor-0.0.1-SNAPSHOT.jar Java Snippet Using JSoup to extract links from URL (HTML) pLink = new PrintableLink(); pLink.setLink(link.attr("abs:href")); pLink.setDescr(trim(link.text(), 100)); linksReturned.add(pLink); Output hdfs dfs -ls /music/*.mid -rw-r--r-- 3 tspann hdfs 87 2016-11-07 03:51 /music/2_ase.mid -rw-r--r-- 3 tspann hdfs 99 2016-11-07 03:51 /music/4_mtking.mid -rw-r--r-- 3 tspann hdfs 105 2016-11-07 03:50 /music/EspanjaCaphriccoCatalan.mid -rw-r--r-- 3 tspann hdfs 87 2016-11-07 03:50 /music/EspanjaPrelude.mid -rw-r--r-- 3 tspann hdfs 162 2016-11-07 03:50 /music/J_M_Bach_Auf_lasst_uns_den_Herren_loben.mid -rw-r--r-- 3 tspann hdfs 93 2016-11-07 03:52 /music/adelina.mid -rw-r--r-- 3 tspann hdfs 95 2016-11-07 03:52 /music/aida_ii2.mid -rw-r--r-- 3 tspann hdfs 89 2016-11-07 03:50 /music/al_adagi.mid -rw-r--r-- 3 tspann hdfs 95 2016-11-07 03:52 /music/alborada.mid -rw-r--r-- 3 tspann hdfs 82 2016-11-07 03:52 /music/aquarium.mid -rw-r--r-- 3 tspann hdfs 105 2016-11-07 03:52 /music/barbero.mid -rw-r--r-- 3 tspann hdfs 101 2016-11-07 03:51 /music/barimyst.mid -rw-r--r-- 3 tspann hdfs 80 2016-11-07 03:52 /music/beevar2.mid -rw-r--r-- 3 tspann hdfs 111 2016-11-07 03:50 /music/biz_arls.mid -rw-r--r-- 3 tspann hdfs 94 2016-11-07 03:51 /music/blas1.mid -rw-r--r-- 3 tspann hdfs 114 2016-11-07 03:50 /music/boccher.mid -rw-r--r-- 3 tspann hdfs 78 2016-11-07 03:52 /music/bolero.mid -rw-r--r-- 3 tspann hdfs 100 2016-11-07 03:51 /music/cantique.mid -rw-r--r-- 3 tspann hdfs 88 2016-11-07 03:51 /music/carminab.mid -rw-r--r-- 3 tspann hdfs 96 2016-11-07 03:51 /music/clairdelune.mid -rw-r--r-- 3 tspann hdfs 87 2016-11-07 03:51 /music/cmveder.mid -rw-r--r-- 3 tspann hdfs 108 2016-11-07 03:52 /music/coucou.mid -rw-r--r-- 3 tspann hdfs 99 2016-11-07 03:51 /music/coup8a.mid -rw-r--r-- 3 tspann hdfs 97 2016-11-07 03:51 /music/cpf-bird.mid 2016-11-06 22:57:20.095 ERROR 28694 --- [nio-8080-exec-1] com.dataflowdeveloper.DataController : Query:http://www.midiworld.com/classic.htm/ mid,IP:192.168.1.2 Browser:nifi-agent 2016-11-06 22:57:20.313 ERROR 28694 --- [nio-8080-exec-3] com.dataflowdeveloper.DataController : Query:http://www.midiworld.com/classic.htm/ mid,IP:192.168.1.2 Browser:nifi-agent 2016-11-06 22:57:20.500 ERROR 28694 --- [nio-8080-exec-5] com.dataflowdeveloper.DataController : Query:http://www.midiworld.com/classic.htm/ mid,IP:192.168.1.2 Browser:nifi-agent ls -lt /opt/demo/midi | more total 20456 -rw-r--r-- 1 tspann staff 117731 Nov 6 22:58 appspg13.mid -rw-r--r-- 1 tspann staff 13449 Nov 6 22:58 intrlude.mid -rw-r--r-- 1 tspann staff 8777 Nov 6 22:58 latalant.mid -rw-r--r-- 1 tspann staff 1911 Nov 6 22:58 lbvar2.mid -rw-r--r-- 1 tspann staff 2230 Nov 6 22:58 lbvar4.mid -rw-r--r-- 1 tspann staff 1458 Nov 6 22:58 lbvar6ep.mid NiFi Flow GetHTTP: Call JSoup Microservice that converts HTML page full of MIDI links into JSON file of links and descriptions of MIDI files. SplitJSON: Split that big JSON file into individual link, description pairs for working with individual songs. EvaluateJSONPath: Use JSONPATH to pull out link and description as attributes. InvokeHTTP: Download the MIDI file from the link. UpdateAttribute: Give it a good file name. I just want the file name from the link (example http://sdfsdf:8080/test.mid PutFile: Store the MIDI on the OSX filesystem. ExecuteStreamCommand: Run the timidity CLI to play the MIDI file. Pass a link to the stored MIDI file to timidity player. PutHDFS: Store the MIDI on HDP 2.5 HDFS. References http://jsonpath.com/ http://macappstore.org/timidity/ ... View more

Starting My Hadoop Tools NiFi can interface directly with Hive, HDFS, HBase, Flume and Phoenix. And I can also trigger Spark and Flink through Kafka and Site-To-Site. Sometimes I need to run some Pig scripts. Apache Pig is very stable and has a lot of functions and tools that make for some smart processing. You can easily augment and add this piece to a larger pipeline or part of the process. Pig Setup I like to use Ambari to install the HDP 2.5 clients on my NiFi box to have access to all the tools I may need. Then I can just do: yum install pig Pig to Apache NiFi 1.0.0 ExecuteProcess We call a shell script that wraps the Pig script. Output of script is stored to HDFS: hdfs dfs -ls /nifi-logs Shell Script export JAVA_HOME=/opt/jdk1.8.0_101/ pig -x local -l /tmp/pig.log -f /opt/demo/pigscripts/test.pig You can run in different Pig modes like local, mapreduce and tez. You can also pass in parameters or the script. Pig Script messages = LOAD '/opt/demo/HDF/centos7/tars/nifi/nifi-1.0.0.2.0.0.0-579/logs/nifi-app.log'; warns = FILTER messages BY $0 MATCHES '.*WARN+.*'; DUMP warns store warns into 'warns.out' This is a basic example from the internet, with the NIFI 1.0 log used as the source. As an aside, I run a daily script with the schedule 1 * * * * ? to clean up my logs. Simply: /bin/rm -rf /opt/demo/HDF/centos7/tars/nifi/nifi-1.0.0.2.0.0.0-579/logs/*2016* PutHDFS Hadoop Configuration: /etc/hadoop/conf/core-site.xml Pick a directory and store away. Results HadoopVersionPigVersionUserIdStartedAtFinishedAtFeatures 2.7.3.2.5.0.0-12450.16.0.2.5.0.0-1245root2016-11-03 19:53:572016-11-03 19:53:59FILTER Success! Job Stats (time in seconds): JobIdMapsReducesMaxMapTimeMinMapTimeAvgMapTimeMedianMapTimeMaxReduceTimeMinReduceTimeAvgReduceTimeMedianReducetimeAliasFeatureOutputs job_local72884441_000110n/an/an/an/a0000messages,warnsMAP_ONLYfile:/tmp/temp1540654561/tmp-600070101, Input(s): Successfully read 30469 records from: "/opt/demo/HDF/centos7/tars/nifi/nifi-1.0.0.2.0.0.0-579/logs/nifi-app.log" Output(s): Successfully stored 1347 records in: "file:/tmp/temp1540654561/tmp-600070101" Counters: Total records written : 1347 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 Job DAG: job_local72884441_0001 Reference: http://hortonworks.com/hadoop-tutorial/hello-world-an-introduction-to-hadoop-hcatalog-hive-and-pig/#section_5 http://hortonworks.com/apache/pig/#section_2 http://hortonworks.com/blog/jsonize-anything-in-pig-with-tojson/ https://github.com/dbist/pig https://github.com/sudar/pig-samples http://hortonworks.com/hadoop-tutorial/how-to-use-basic-pig-commands/ http://hadooptutorial.info/built-in-load-store-functions-in-pig/ https://cwiki.apache.org/confluence/display/PIG/PigTutorial https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_installing_manually_book/content/validate_the_installation_pig.html http://pig.apache.org/docs/r0.16.0/start.html http://hortonworks.com/hadoop-tutorial/how-to-process-data-with-apache-pig https://github.com/alanfgates/programmingpig/tree/master/examples/ch2 ... View more