Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera kafka pyspark KafkaProducer (ImportError: No module named kafka)

Cloudera kafka pyspark KafkaProducer (ImportError: No module named kafka)

Explorer

Need help on Kafka on Cloudera.
I wrote a program in pySpark in PyCharm it works good.

 

from kafka import KafkaProducer
from kafka.errors import KafkaError
producer = KafkaProducer(bootstrap_servers=['192.168.56.103:9092'])
tes = producer.send('my-first-topic', "this message from pyspark")
producer.flush()

 

but when I run in my Linux Cloudera machine I get

File "/home/cloudera/kafka/kproducer.py", line 1, in <module>
from kafka import KafkaProducer
ImportError: No module named kafka

 

using command spark2-submit kproducer.py

1 REPLY 1
Highlighted

Re: Cloudera kafka pyspark KafkaProducer (ImportError: No module named kafka)

Contributor

Hey @AndyTech,

 

Thanks for reaching out to the Cloudera community.

 

This issue is due to the missing "kafka-python" module in your Python installation. You have to manually install the "kafka-python" module using the mentioned command in the edge node and all the hosts on which Spark job executes.

 

$ pip install kafka-python

Don't have an account?
Coming from Hortonworks? Activate your account here