This sounds like it may be a build problem. https://github.com/simonellistonball/spark-samples... has a working sample with sbt scripts to build against the Hortonworks repository, which has been testing on HDP 2.3.2. Note that the Kafka consumer API has changed a bit recently, so it's important to be aware of versions in Kafka. Also, I note that you're running in local model, we would recommend that you only use local mode for testing, and that you use --master yarn-client for running on a proper cluster. ... View more

There are a range of common NLP systems that work well on the platform. OpenNLP is a java native library which integrates well with, for example map reduce, and of course NLTK being a python system works well with pyspark. There are also native spark elements which are connected to NLP tasks: Latent Dirichlet Allocation for topic detection is one example. Of course the NLTK components also work well with Hive to do things like Tokenisation, and Part of Speech tagging. Stanford CoreNLP also provides a good toolkit of NLP functions. There is also a spark-package to integrate this with SparkML pipelines. Solr provides a number of useful tools that apply in the NLP space as well, such as stemming, synonym handling etc as part of its indexing and querying, so provides some building blocks for simple NLP analysis. There are also a number of commercial and partner solutions which handle NLP tasks. We are also looking to build tools for Entity Resolution on Spark, which will add to this. ... View more