I am new to spark and Kafka. I am following the tutorial at http://henning.kropponline.de/2016/12/25/simple-spark-streaming-kafka-example-in-a-zeppelin-notebook...
In the artilces, kafka and spark packages are
I installed hdp 2.6. What are the version of those packages should I use?
how could I find it out?
1. If you have access to Ambari, you can see the components version details on Stacks and Version Page
On Ambari screen, top right, Hit the Admin tab > Stacks and Version
Or directly using http://ambarihost:port/#/main/admin/stack/services
Replace ambarihost:port with details from your cluster.
If you dont have access to ambari:
2. Try out the hdp-select command. This will list all packages installed.
You can grep for the package you are looking for:
hdp-select status | grep kafka hdp-select status | grep spark
3. Alternatively, you can also use the standard yum commands to get the similar result.
However, using yum will list multiple versions of packages if they exist on your system.
yum list installed | grep spark yum list installed | grep kafka
Release notes page for each version of HDP includes versions packaged for each project. Following is for HDP 2.6.1. Kafka version is 0.10.1.2 and Apache Spark is 2.1.1 as well as Apache Spark 1.6.3.
to find version of Spark from command line, type spark-submit --version.