Adding Stanford CoreNLP To Big Data Pipelines (Apache NiFi 1.1/HDF 2.1) Part 2 of 2

[#1]

Sentiment CoreNLP Processor

[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -
Adding annotator tokenize[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.TokenizerAnnotator
- No tokenizer type provided. Defaulting to PTBTokenizer.[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -
Adding annotator ssplit[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -
Adding annotator parse[pool-1-thread-1] INFO edu.stanford.nlp.parser.common.ParserGrammar
- Loading parser from serialized file edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz
... done [0.4 sec].[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -
Adding annotator sentimentFILE:Header,Header2,Header3Value,Value2,Value3Value4,Value5,Value6Attribute: {"names":"NEGATIVE"}

Service Source Code

JUnit Test for Processor

To Add Sentiment Analysis to Your NiFi Data Flow, just add the custom processor, CoreNLPProcessor. You can downloada pre-built NAR from the github listed below. Add to your NiFi/lib directory and restart each node.

The results of the run will be an attribute named sentiment:

You can see how easy it is to add to your dataflows.

If you would like to add more features to this processor, please fork the github below.

This is not an official NiFi processor, just one I wrote in a couple of hours for my own use and for testing.

There are four easy ways to add Sentiment Analysis to your Big Data pipelines: executescript of Python NLP scripts, call my custom processor, make a REST call to a Stanford CoreNLP sentiment server, make a REST call to a public sentiment as a service and send a message via Kafka (or JMS) to Spark or Storm to run other JVM sentiment analysis tools.

Download a release

• https://github.com/tspannhw/nifi-corenlp-processor/releases/tag/v1.0
• sentimentanalysiscustomprocessor.xml
• http://stanfordnlp.github.io/CoreNLP
• https://github.com/tspannhw/neural-sentiment
• https://github.com/tspannhw/nlp-utilities
• https://community.hortonworks.com/content/kbentry/81222/adding-stanford-corenlp-to-big-data-pipeline...
• https://community.hortonworks.com/content/repo/81187/nifi-corenlp-processor-example-processor-for-do...
• https://community.hortonworks.com/repos/79537/various-utilities-and-examples-for-working-with-va.htm...
• https://community.hortonworks.com/articles/76935/using-sentiment-analysis-and-nlp-tools-with-hdp-25....
• https://community.hortonworks.com/questions/20791/sentiment-analysis-with-hdp.html
• https://community.hortonworks.com/articles/30213/us-presidential-election-tweet-analysis-using-hdfn....
• https://community.hortonworks.com/articles/52415/processing-social-media-feeds-in-stream-with-apach....
• https://community.hortonworks.com/articles/81222/adding-stanford-corenlp-to-big-data-pipelines-apac....
• https://community.hortonworks.com/content/kbentry/67983/apache-hive-with-apache-hivemall.html