Support Questions
Find answers, ask questions, and share your expertise

Local configuration of pyCharm - Spark



I have a question regarding the setup of a development environment.

I’m using an AWS where I installed the last version of Hortonworks and wants to do some POC around kafka and spark streaming.

As the access to AWS is not free I wanted to create a dev env on my local laptop based on PyCharm and spark 1.6.

Usually I use a maven repository but I’m looking for a simpler solution. Can this works:

  1. Install localy the exact spark version than AWS
  2. Setup the PyCharm to point to the local spark
  3. Do the development locally then deploy and run the code on the AWS