Adding Stanford CoreNLP To Big Data Pipelines (Apache NiFi 1.1/HDF 2.1) Part 2 of 2

[#1]

Sentiment CoreNLP Processor

[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -
Adding annotator tokenize[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.TokenizerAnnotator
- No tokenizer type provided. Defaulting to PTBTokenizer.[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -
Adding annotator ssplit[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -
Adding annotator parse[pool-1-thread-1] INFO edu.stanford.nlp.parser.common.ParserGrammar
- Loading parser from serialized file edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz
... done [0.4 sec].[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -
Adding annotator sentimentFILE:Header,Header2,Header3Value,Value2,Value3Value4,Value5,Value6Attribute: {"names":"NEGATIVE"}

Service Source Code

12062-corenlpsourcecode.png

JUnit Test for Processor

12063-corenlpjunit.png

To Add Sentiment Analysis to Your NiFi Data Flow, just add the custom processor, CoreNLPProcessor. You can downloada pre-built NAR from the github listed below. Add to your NiFi/lib directory and restart each node.

12064-corenlpaddprocessor.png

The results of the run will be an attribute named sentiment:

12067-corenlp-results.png

You can see how easy it is to add to your dataflows.

12069-corenlpoverview.png

If you would like to add more features to this processor, please fork the github below.

This is not an official NiFi processor, just one I wrote in a couple of hours for my own use and for testing.

There are four easy ways to add Sentiment Analysis to your Big Data pipelines: executescript of Python NLP scripts, call my custom processor, make a REST call to a Stanford CoreNLP sentiment server, make a REST call to a public sentiment as a service and send a message via Kafka (or JMS) to Spark or Storm to run other JVM sentiment analysis tools.

Download a release

• https://github.com/tspannhw/nifi-corenlp-processor/releases/tag/v1.0
• sentimentanalysiscustomprocessor.xml
• http://stanfordnlp.github.io/CoreNLP
• https://github.com/tspannhw/neural-sentiment
• https://github.com/tspannhw/nlp-utilities
• https://community.hortonworks.com/content/kbentry/81222/adding-stanford-corenlp-to-big-data-pipeline...
• https://community.hortonworks.com/content/repo/81187/nifi-corenlp-processor-example-processor-for-do...
• https://community.hortonworks.com/repos/79537/various-utilities-and-examples-for-working-with-va.htm...
• https://community.hortonworks.com/articles/76935/using-sentiment-analysis-and-nlp-tools-with-hdp-25....
• https://community.hortonworks.com/questions/20791/sentiment-analysis-with-hdp.html
• https://community.hortonworks.com/articles/30213/us-presidential-election-tweet-analysis-using-hdfn....
• https://community.hortonworks.com/articles/52415/processing-social-media-feeds-in-stream-with-apach....
• https://community.hortonworks.com/articles/81222/adding-stanford-corenlp-to-big-data-pipelines-apac....
• https://community.hortonworks.com/content/kbentry/67983/apache-hive-with-apache-hivemall.html