Reply
Expert Contributor
Posts: 139
Registered: ‎07-21-2014

Running a spark streaming job in production

How does one go about deploying a Spark streaming job on a CDH cluster? Does the job need to be deployed as a CSD parcel?

 

Thanks!

Cloudera Employee
Posts: 97
Registered: ‎05-10-2016

Re: Running a spark streaming job in production

CSD and parcel are two different things.  CSDs manage third party processes, parcels is a mechanism to distrubte the software.  You could use a parcel to deploy your Spark code to all workers, but this would be overkill.  Instead you can package a jar if using java or spark or submit a python file from the command line[1].

 

1.  https://spark.apache.org/docs/1.6.1/submitting-applications.html