Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Running a spark streaming job in production

Running a spark streaming job in production

Expert Contributor

How does one go about deploying a Spark streaming job on a CDH cluster? Does the job need to be deployed as a CSD parcel?

 

Thanks!

1 REPLY 1

Re: Running a spark streaming job in production

Expert Contributor

CSD and parcel are two different things.  CSDs manage third party processes, parcels is a mechanism to distrubte the software.  You could use a parcel to deploy your Spark code to all workers, but this would be overkill.  Instead you can package a jar if using java or spark or submit a python file from the command line[1].

 

1.  https://spark.apache.org/docs/1.6.1/submitting-applications.html