Use Case
Before meetups, I wanted to play some music. NiFi seemed like a great choice for streaming free music through my Mac.
MIDI Command Line Player For OSX
brew install timidity
Then you can simply play MIDI files with timidity file.mid.
Microservice to Extract Links from Web Pages
Java 8 Source Code: https://github.com/tspannhw/linkextractor
The Spring Boot REST API accepts a URL, parses out mid files and returns JSON containing linking and descriptions.
Example REST Call to Service
curl -G -v "http://<urL>:8080/extract/url?url=http://www.midiworld.com/classic.htm/&type=mid"
Run the Microservice
java -Xms512m -Xmx2048m -Djava.net.preferIPv4Stack=true -jar target/linkextractor-0.0.1-SNAPSHOT.jar
Java Snippet Using JSoup to extract links from URL (HTML)
pLink = new PrintableLink();
pLink.setLink(link.attr("abs:href"));
pLink.setDescr(trim(link.text(), 100));
linksReturned.add(pLink); Output
hdfs dfs -ls /music/*.mid -rw-r--r-- 3 tspann hdfs 87 2016-11-07 03:51 /music/2_ase.mid -rw-r--r-- 3 tspann hdfs 99 2016-11-07 03:51 /music/4_mtking.mid -rw-r--r-- 3 tspann hdfs 105 2016-11-07 03:50 /music/EspanjaCaphriccoCatalan.mid -rw-r--r-- 3 tspann hdfs 87 2016-11-07 03:50 /music/EspanjaPrelude.mid -rw-r--r-- 3 tspann hdfs 162 2016-11-07 03:50 /music/J_M_Bach_Auf_lasst_uns_den_Herren_loben.mid -rw-r--r-- 3 tspann hdfs 93 2016-11-07 03:52 /music/adelina.mid -rw-r--r-- 3 tspann hdfs 95 2016-11-07 03:52 /music/aida_ii2.mid -rw-r--r-- 3 tspann hdfs 89 2016-11-07 03:50 /music/al_adagi.mid -rw-r--r-- 3 tspann hdfs 95 2016-11-07 03:52 /music/alborada.mid -rw-r--r-- 3 tspann hdfs 82 2016-11-07 03:52 /music/aquarium.mid -rw-r--r-- 3 tspann hdfs 105 2016-11-07 03:52 /music/barbero.mid -rw-r--r-- 3 tspann hdfs 101 2016-11-07 03:51 /music/barimyst.mid -rw-r--r-- 3 tspann hdfs 80 2016-11-07 03:52 /music/beevar2.mid -rw-r--r-- 3 tspann hdfs 111 2016-11-07 03:50 /music/biz_arls.mid -rw-r--r-- 3 tspann hdfs 94 2016-11-07 03:51 /music/blas1.mid -rw-r--r-- 3 tspann hdfs 114 2016-11-07 03:50 /music/boccher.mid -rw-r--r-- 3 tspann hdfs 78 2016-11-07 03:52 /music/bolero.mid -rw-r--r-- 3 tspann hdfs 100 2016-11-07 03:51 /music/cantique.mid -rw-r--r-- 3 tspann hdfs 88 2016-11-07 03:51 /music/carminab.mid -rw-r--r-- 3 tspann hdfs 96 2016-11-07 03:51 /music/clairdelune.mid -rw-r--r-- 3 tspann hdfs 87 2016-11-07 03:51 /music/cmveder.mid -rw-r--r-- 3 tspann hdfs 108 2016-11-07 03:52 /music/coucou.mid -rw-r--r-- 3 tspann hdfs 99 2016-11-07 03:51 /music/coup8a.mid -rw-r--r-- 3 tspann hdfs 97 2016-11-07 03:51 /music/cpf-bird.mid 2016-11-06 22:57:20.095 ERROR 28694 --- [nio-8080-exec-1] com.dataflowdeveloper.DataController : Query:http://www.midiworld.com/classic.htm/ mid,IP:192.168.1.2 Browser:nifi-agent 2016-11-06 22:57:20.313 ERROR 28694 --- [nio-8080-exec-3] com.dataflowdeveloper.DataController : Query:http://www.midiworld.com/classic.htm/ mid,IP:192.168.1.2 Browser:nifi-agent 2016-11-06 22:57:20.500 ERROR 28694 --- [nio-8080-exec-5] com.dataflowdeveloper.DataController : Query:http://www.midiworld.com/classic.htm/ mid,IP:192.168.1.2 Browser:nifi-agent ls -lt /opt/demo/midi | more total 20456 -rw-r--r-- 1 tspann staff 117731 Nov 6 22:58 appspg13.mid -rw-r--r-- 1 tspann staff 13449 Nov 6 22:58 intrlude.mid -rw-r--r-- 1 tspann staff 8777 Nov 6 22:58 latalant.mid -rw-r--r-- 1 tspann staff 1911 Nov 6 22:58 lbvar2.mid -rw-r--r-- 1 tspann staff 2230 Nov 6 22:58 lbvar4.mid -rw-r--r-- 1 tspann staff 1458 Nov 6 22:58 lbvar6ep.mid
NiFi Flow
GetHTTP: Call JSoup Microservice that converts HTML page full of MIDI links into JSON file of links and descriptions of MIDI files.
SplitJSON: Split that big JSON file into individual link, description pairs for working with individual songs.
EvaluateJSONPath: Use JSONPATH to pull out link and description as attributes.
InvokeHTTP: Download the MIDI file from the link.
UpdateAttribute: Give it a good file name.
I just want the file name from the link (example http://sdfsdf:8080/test.mid
PutFile: Store the MIDI on the OSX filesystem.
ExecuteStreamCommand: Run the timidity CLI to play the MIDI file.
Pass a link to the stored MIDI file to timidity player.
PutHDFS: Store the MIDI on HDP 2.5 HDFS.
References
