Community Articles
Find and share helpful community-sourced technical articles
Labels (1)
Super Guru

Use Case

Before meetups, I wanted to play some music. NiFi seemed like a great choice for streaming free music through my Mac.

9180-dataflow.png

MIDI Command Line Player For OSX

brew install timidity

Then you can simply play MIDI files with timidity file.mid.

Microservice to Extract Links from Web Pages

Java 8 Source Code: https://github.com/tspannhw/linkextractor

The Spring Boot REST API accepts a URL, parses out mid files and returns JSON containing linking and descriptions.

Example REST Call to Service

curl -G -v "http://<urL>:8080/extract/url?url=http://www.midiworld.com/classic.htm/&type=mid"

Run the Microservice

java -Xms512m -Xmx2048m -Djava.net.preferIPv4Stack=true -jar target/linkextractor-0.0.1-SNAPSHOT.jar

Java Snippet Using JSoup to extract links from URL (HTML)

pLink = new PrintableLink();      
pLink.setLink(link.attr("abs:href"));      
pLink.setDescr(trim(link.text(), 100));      
linksReturned.add(pLink); 

Output

hdfs dfs -ls /music/*.mid
-rw-r--r--   3 tspann hdfs         87 2016-11-07 03:51 /music/2_ase.mid
-rw-r--r--   3 tspann hdfs         99 2016-11-07 03:51 /music/4_mtking.mid
-rw-r--r--   3 tspann hdfs        105 2016-11-07 03:50 /music/EspanjaCaphriccoCatalan.mid
-rw-r--r--   3 tspann hdfs         87 2016-11-07 03:50 /music/EspanjaPrelude.mid
-rw-r--r--   3 tspann hdfs        162 2016-11-07 03:50 /music/J_M_Bach_Auf_lasst_uns_den_Herren_loben.mid
-rw-r--r--   3 tspann hdfs         93 2016-11-07 03:52 /music/adelina.mid
-rw-r--r--   3 tspann hdfs         95 2016-11-07 03:52 /music/aida_ii2.mid
-rw-r--r--   3 tspann hdfs         89 2016-11-07 03:50 /music/al_adagi.mid
-rw-r--r--   3 tspann hdfs         95 2016-11-07 03:52 /music/alborada.mid
-rw-r--r--   3 tspann hdfs         82 2016-11-07 03:52 /music/aquarium.mid
-rw-r--r--   3 tspann hdfs        105 2016-11-07 03:52 /music/barbero.mid
-rw-r--r--   3 tspann hdfs        101 2016-11-07 03:51 /music/barimyst.mid
-rw-r--r--   3 tspann hdfs         80 2016-11-07 03:52 /music/beevar2.mid
-rw-r--r--   3 tspann hdfs        111 2016-11-07 03:50 /music/biz_arls.mid
-rw-r--r--   3 tspann hdfs         94 2016-11-07 03:51 /music/blas1.mid
-rw-r--r--   3 tspann hdfs        114 2016-11-07 03:50 /music/boccher.mid
-rw-r--r--   3 tspann hdfs         78 2016-11-07 03:52 /music/bolero.mid
-rw-r--r--   3 tspann hdfs        100 2016-11-07 03:51 /music/cantique.mid
-rw-r--r--   3 tspann hdfs         88 2016-11-07 03:51 /music/carminab.mid
-rw-r--r--   3 tspann hdfs         96 2016-11-07 03:51 /music/clairdelune.mid
-rw-r--r--   3 tspann hdfs         87 2016-11-07 03:51 /music/cmveder.mid
-rw-r--r--   3 tspann hdfs        108 2016-11-07 03:52 /music/coucou.mid
-rw-r--r--   3 tspann hdfs         99 2016-11-07 03:51 /music/coup8a.mid
-rw-r--r--   3 tspann hdfs         97 2016-11-07 03:51 /music/cpf-bird.mid


2016-11-06 22:57:20.095 ERROR 28694 --- [nio-8080-exec-1] com.dataflowdeveloper.DataController     : Query:http://www.midiworld.com/classic.htm/ mid,IP:192.168.1.2 Browser:nifi-agent
2016-11-06 22:57:20.313 ERROR 28694 --- [nio-8080-exec-3] com.dataflowdeveloper.DataController     : Query:http://www.midiworld.com/classic.htm/ mid,IP:192.168.1.2 Browser:nifi-agent
2016-11-06 22:57:20.500 ERROR 28694 --- [nio-8080-exec-5] com.dataflowdeveloper.DataController     : Query:http://www.midiworld.com/classic.htm/ mid,IP:192.168.1.2 Browser:nifi-agent


ls -lt /opt/demo/midi | more
total 20456
-rw-r--r--  1 tspann  staff  117731 Nov  6 22:58 appspg13.mid
-rw-r--r--  1 tspann  staff   13449 Nov  6 22:58 intrlude.mid
-rw-r--r--  1 tspann  staff    8777 Nov  6 22:58 latalant.mid
-rw-r--r--  1 tspann  staff    1911 Nov  6 22:58 lbvar2.mid
-rw-r--r--  1 tspann  staff    2230 Nov  6 22:58 lbvar4.mid
-rw-r--r--  1 tspann  staff    1458 Nov  6 22:58 lbvar6ep.mid

NiFi Flow

GetHTTP: Call JSoup Microservice that converts HTML page full of MIDI links into JSON file of links and descriptions of MIDI files.

9174-gethttp.png

SplitJSON: Split that big JSON file into individual link, description pairs for working with individual songs.

9175-splitjson.png

EvaluateJSONPath: Use JSONPATH to pull out link and description as attributes.

9176-evaluatejsonpath.png

InvokeHTTP: Download the MIDI file from the link.

9177-invokehttp.png

UpdateAttribute: Give it a good file name.

9178-updateattribute.png

I just want the file name from the link (example http://sdfsdf:8080/test.mid

PutFile: Store the MIDI on the OSX filesystem.

ExecuteStreamCommand: Run the timidity CLI to play the MIDI file.

9179-executestreamcommand.png

Pass a link to the stored MIDI file to timidity player.

PutHDFS: Store the MIDI on HDP 2.5 HDFS.

References

http://jsonpath.com/

http://macappstore.org/timidity/

802 Views