1. Acquire an EDI File (GetFile, GetFTP, GetHTTP, GetSFTP, Fetch...) 2. Install open source nifi-edireader on NIFI 1.0.0 Download https://github.com/BerryWorksSoftware/edireader Maven Install Berry Works EDIReader Download https://github.com/mrcsparker/nifi-edireader-bundle Maven packge nifi-edireader (must be Maven 3.3 or newer - may have to download and install separately from standard linux package) cp nifi-edireader-nar/target/nifi-edireader-nar-0.0.1.nar to your NIFI/lib Restart NiFi Service 3. Add EdiXML Processor and connect from EDI File input 4. Add extra processing, conversion or routing (TransformXML with XSLT or EvaluateXPATH) to convert to JSON 5. Land to HDFS (PutHDFS) 6. Used the Web Form linked below to generate a test EDI file. ISA*00* *00* *ZZ*SENDER ID *ZZ*RECEIVER ID *010101*0101*U*00401*000000001*0*T*! GS*IN*SENDER ID*APP RECEIVER*01010101*01010101*1*X*004010 ST*810*0001 BIG*20021208*00001**A999 N1*ST*Timothy Spann*9*122334455 N3*115 xxx ave N4*xxxtown*nj*08520 N1*BT*Hortonworks*9*122334455 N3*5470 GREAT AMERICA PARKWAY N4*santa clara*CA*95054 ITD*01*3*2**30**30*****60 FOB*PP IT1**1*EA*200**UA*EAN PID*F****Lamp IT1**4*EA*50**UA*EAN PID*F****Chair TDS*2000 CAD*****Routing ISS*30*CA CTT*50 SE*19*0001 GE*1*1 IEA*1*000000001   7. Converted to XML <?xml version="1.0" encoding="UTF-8"?> <ediroot> <interchange Standard="ANSI X.12" AuthorizationQual="00" Authorization=" " SecurityQual="00" Security=" " Date="010101" Time="0101" StandardsId="U" Version="00401" Control="000000001" AckRequest="0" TestIndicator="T"> <sender> <address Id="SENDER ID " Qual="ZZ"/> </sender> <receiver> <address Id="RECEIVER ID " Qual="ZZ"/> </receiver> <group GroupType="IN" ApplSender="SENDER ID" ApplReceiver="APP RECEIVER" Date="01010101" Time="01010101" Control="1" StandardCode="X" StandardVersion="004010"> <transaction DocType="810" Name="Invoice" Control="0001"> <segment Id="BIG"> <element Id="BIG01">20021208</element> <element Id="BIG02">00001</element> <element Id="BIG04">A999</element> </segment> <loop Id="N1"> <segment Id="N1"> <element Id="N101">ST</element> <element Id="N102">Timothy Spann</element> <element Id="N103">9</element> <element Id="N104">122334455</element> </segment> <segment Id="N3"> <element Id="N301">115 xxx ave</element> </segment> <segment Id="N4"> <element Id="N401">xxxstown</element> <element Id="N402">nj</element> <element Id="N403">08520</element> </segment> </loop> <loop Id="N1"> <segment Id="N1"> <element Id="N101">BT</element> <element Id="N102">Hortonworks</element> <element Id="N103">9</element> <element Id="N104">122334455</element> </segment> <segment Id="N3"> <element Id="N301">5470 GREAT AMERICA PARKWAY</element> </segment> <segment Id="N4"> <element Id="N401">santa clara</element> <element Id="N402">CA</element> <element Id="N403">95054</element> </segment> </loop> <segment Id="ITD"> <element Id="ITD01">01</element> <element Id="ITD02">3</element> <element Id="ITD03">2</element> <element Id="ITD05">30</element> <element Id="ITD07">30</element> <element Id="ITD12">60</element> </segment> <segment Id="FOB"> <element Id="FOB01">PP</element> </segment> <loop Id="IT1"> <segment Id="IT1"> <element Id="IT102">1</element> <element Id="IT103">EA</element> <element Id="IT104">200</element> <element Id="IT106">UA</element> <element Id="IT107">EAN</element> </segment> <loop Id="PID"> <segment Id="PID"> <element Id="PID01">F</element> <element Id="PID05">Lamp</element> </segment> </loop> </loop> <loop Id="IT1"> <segment Id="IT1"> <element Id="IT102">4</element> <element Id="IT103">EA</element> <element Id="IT104">50</element> <element Id="IT106">UA</element> <element Id="IT107">EAN</element> </segment> <loop Id="PID"> <segment Id="PID"> <element Id="PID01">F</element> <element Id="PID05">Chair</element> </segment> </loop> </loop> <segment Id="TDS"> <element Id="TDS01">2000</element> </segment> <segment Id="CAD"> <element Id="CAD05">Routing</element> </segment> <loop Id="ISS"> <segment Id="ISS"> <element Id="ISS01">30</element> <element Id="ISS02">CA</element> </segment> </loop> <segment Id="CTT"> <element Id="CTT01">50</element> </segment> </transaction> </group> </interchange> </ediroot> Resources https://github.com/mrcsparker/nifi-edireader-bundle https://github.com/BerryWorksSoftware/edireader https://en.wikipedia.org/wiki/Electronic_data_interchange https://en.wikipedia.org/wiki/EDIFACT https://en.wikipedia.org/wiki/FORTRAS http://databene.org/edifatto.html https://sourceforge.net/projects/edifatto/ https://secure.edidev.net/edidev-ca/samples/vbNetGen/WebFrmNetGen.aspx (Generate example EDI)   ... View more

I ran the same flow myself and examined the AVRO file in HDFS using AVRO Cli. Even though I didn't specify SNAPPY compression, it was there in the file. java -jar avro-tools-1.8.0.jar getmeta 23568764174290.avro log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. avro.codec snappyavro.schema {"type":"record","name":"people","doc":"Schema generated by Kite","fields":[{"name":"id","type":"long","doc":"Type inferred from '2'"},{"name":"first_name","type":"string","doc":"Type inferred from 'Gregory'"},{"name":"last_name","type":"string","doc":"Type inferred from 'Vasquez'"},{"name":"email","type":"string","doc":"Type inferred from 'gvasquez1@pcworld.com'"},{"name":"gender","type":"string","doc":"Type inferred from 'Male'"},{"name":"ip_address","type":"string","doc":"Type inferred from '32.8.254.252'"},{"name":"company_name","type":"string","doc":"Type inferred from 'Janyx'"},{"name":"domain_name","type":"string","doc":"Type inferred from 'free.fr'"},{"name":"file_name","type":"string","doc":"Type inferred from 'NonMauris.xls'"},{"name":"mac_address","type":"string","doc":"Type inferred from '03-FB-66-0F-20-A3'"},{"name":"user_agent","type":"string","doc":"Type inferred from '\"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7;'"},{"name":"lat","type":"string","doc":"Type inferred from ' like Gecko) Version/5.0.4 Safari/533.20.27\"'"},{"name":"long","type":"double","doc":"Type inferred from '26.98829'"}]} It's hard coded in NIFI. https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/ConvertCSVToAvro.java It always adds SnappyCompression to every AVRO file. No options. writer.setCodec(CodecFactory.snappyCodec()); Make sure you have a schema set: Record schema Record Schema: ${inferred.avro.schema} If you can make everything Strings and convert to other types later, you will be happier. References: https://www.linkedin.com/pulse/converting-csv-avro-apache-nifi-jeremy-dyer https://community.hortonworks.com/questions/44063/nifi-avro-to-csv-or-json-to-csvnifi-convert-avro-t.html https://community.hortonworks.com/articles/28341/converting-csv-to-avro-with-apache-nifi.html ... View more

I had a few hours in the morning before the conference schedule kicked in, so I decided to write a little HDF 2.0 flow to grab all the tweets about Strata Hadoop conference. First up, I used GetTwitter to read tweets and filtered on these terms: strata,stratahadoop,strataconf,NIFI,FutureOfData,ApacheNiFi,Hortonworks,Hadoop,ApacheHive,HBase,ApacheSpark,ApacheTez,MachineLearning,ApachePhoenix,ApacheCalcite,ApacheStorm,ApacheAtlas,ApacheKnox,Apache Ranger, HDFS, Apache Pig, Accumulo, Apache Flume, Sqoop, Apache Falcon Input: InvokeHttp: I used this to download the first image URL from tweets. GetTwitter: This is our primary source of data and the most important. You must have a twitter account, a twitter developer account and create a twitter application. Then you can access the keywords and hashtags above. So far I've ingested 14,211 tweets into Phoenix. This included many times I've shut it down for testing and moving things around. I've had this run live as I've added pieces. I do not recommend this development process, but it's good for exploring data. Processing: RouteOnAttribute: To only process tweets with an actual messages, sometimes they are damaged or missing. Don't waste our time. ExecuteStreamCommand: To call shell scripts that call TensorFlow C++ binaries and Python scripts. Many ways to do this, but this is the easiest. UpdateAttribute: To change the file name for files I downloaded to HDFS. For output sinks: PutHDFS: Saved to HDFS in a few different directories (the first attached image); the raw JSON tweet, a limited set of fields such as handle, message, geolocation and a fully processed file that I added TensorFlow Inception v3 image recognition for images attached to Strata tweets and sentiment analysis using VADER on the text of the tweet. PutSQL: I upserted all tweets that were enriched with HDF called TensorFlow & Python Sentiment Analysis into a Phoenix Table; PutSlack: https://nifi-se.slack.com/messages/general/ Visualization: There are a ton of ways to look at this data now. I used Apache Zeppelin since it was part of my HDP 2.5 cluster and it's so easy to use. I added a few tables, charts and did quick SQL exploration of the data in Phoenix. Linux Shell Scripts source /usr/local/lib/bazel/bin/bazel-complete.bash export JAVA_HOME=/opt/jdk1.8.0_101/ /bin/rm -rf /tmp/$@ hdfs dfs -get /twitter/rawimage/$@ /tmp/ /opt/demo/tensorflow/bazel-bin/tensorflow/examples/label_image/label_image --image="/tmp/$@" --output_layer="softmax:0" --input_layer="Mul:0" --input_std=128 --input_mean=128 --graph=/opt/demo/tensorflow/tensorflow/examples/label_image/data/tensorflow_inception_graph.pb --labels=/opt/demo/tensorflow/tensorflow/examples/label_image/data/imagenet_comp_graph_label_strings.txt 2>&1| cut -c48- /bin/rm -rf /tmp/$@ python /opt/demo/sentiment/sentiment2.py "$@" Python Script If you have Python 2.7 installed, in previous articles I have shown how to install PiP and NLTK. Very easy to do some simple Sentiment Analysis. I also have a version where I just return the polarity_scores (compound, negative, neutral and positive). from nltk.sentiment.vader import SentimentIntensityAnalyzer import sys sid = SentimentIntensityAnalyzer() ss = sid.polarity_scores(sys.argv[1]) if ss['compound'] == 0.00: print('Neutral') elif ss['compound'] < 0.00: print ('Negative') else: print('Positive') NIFI 1.0.0 Flow Template tweetnyc.xml Resources: http://conferences.oreilly.com/strata/hadoop-big-data-ny/public/schedule/grid/public/ http://hortonworks.com/downloads/#dataflow http://futureofdata.io/ http://www.meetup.com/futureofdata-princeton/ ... View more