About kehtg

kehtg · ‎12-29-2016

For the beginners, like myself: If you added a new processor, or change the processor name, you will need to add or change the name in the .Processor file in <Home Dir>/Documents/nifi/ChakraProcessor/HWX/nifi-demo-processors/src/main/resources/META-INF/services. If you don't do this, the processor will not be loaded.

kehtg · ‎03-15-2016

Hi Guys, Thanks a lot for your help. Just to summarize the issue so that someone else knows what to do: If using HDP sandbox 2.4, for the tutorial "A Lap Around Apache Spark", there is no need to install spark, everything is in. When I had problem starting spark, it was because there is another spark instance running. Started because I was running the "Spark on Zeppelin" tutorial. To stop the spark session, use the "yarn application -list" and "yarn application -kill" command to kill of other spark on yarn sessions. All is well now. Takes again for your help.

kehtg · ‎03-14-2016

So sorry, forgot to attached the ps ax output. Please find attached: ps-axoutput.txt

kehtg · ‎03-14-2016

Hi @glupu , Only saw your answer after I posted. As mentioned, you are right, the spark session got started because I was running the spark on zeppelin tutorial. Attached is the ps ax | grep spark output, before and after I ran the zeppelin notebook. 4 processes got started. Killing them does not seem to help as 2 of them will just get restarted. And after killing them, even Zeppeline stopped working properly. Would you have any idea how I can gracefully stop the spark session started by zeppelin? Thanks a lot.

kehtg · ‎03-14-2016

Hi Guys, Thanks for helping me with this. I have confirmed the issue. There is another spark session running and somehow that caused the problem. The session was started when I ran the spark tutorial on Zeppelin. Would you guys know how I can stop the spark session without having to restart the sandbox?

kehtg · ‎03-14-2016

Hi @Shivaji, Thanks. A new instance of the spark shell actually started on port 4041 since 4040 is already in use. I have also access the URL through port 4041 to see that the job is created. The problem I have is that I do not see that the job gets completed and it keeps going. Even starting a spark shell (./bin/spark-shell --master yarn-client --driver-memory 512m--executor-memory 512m) goes into such a "loop" and I am not able to see the scala prompt to start trying out the commands. Or could the problem I am facing be from YARN and not SPARK? Thanks again for your help.

kehtg · ‎03-14-2016

Hi Artem, Thanks for your reply. Yes, I did see that Spark is already available and started in sandbox 2.4. But how do I start the Spark shell using the already started instance? I just followed the instructions in the tutorial to try getting to the Scala prompt but encountered the same issue. Hope you can point me in the correct direction. Thanks a lot.

kehtg · ‎03-13-2016

output.txtHi, I am new to HDP and have downloaded the HDP sandbox 2.4 and trying out the tutorial http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/ But I am not able to complete the Spark Pi example as a message keeps looping through the message "INFO Client: Application report for application_1457875462527_0006 (state: ACCEPTED)" and does not show the result as expected. I have tried following the tutorial with and without the installation of spark_2_3_4_1_10-master and spark_2_3_4_1_10-python as I see that HDP 2.4 already come with spark 2.4.0.0-169. Attached is the output from the command. Hope someone can help. Thanks,

Online	Offline
Last Visited	‎12-29-2016 07:39 AM

Member Since	‎03-13-2016 02:12 PM
Last Visited	‎12-29-2016 07:39 AM
Posts	8
Kudos received	10

Cloudera Community

Re: Unable to complete Spark Pi example tutorial

Re: Build Custom Nifi Processor

Re: Unable to complete Spark Pi example tutorial

Re: Unable to complete Spark Pi example tutorial

Re: Unable to complete Spark Pi example tutorial

Re: Unable to complete Spark Pi example tutorial

Re: Unable to complete Spark Pi example tutorial

Re: Unable to complete Spark Pi example tutorial

Unable to complete Spark Pi example tutorial