question Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell. in Support Questions

How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Fri, 05 Feb 2016 05:09:17 GMT

I'm new to Storm so, please suggest me where to start typing this piece of code and methods to save and execute it on the hortonworks shell. Any suggestions to get started will be helpful. Thanks in advance for your time.

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

nsabharwal — Fri, 05 Feb 2016 05:12:17 GMT

@keerthana gajarajakumar

See this http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/validate_installation_storm.html

You can use the above for smoke test.

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Fri, 05 Feb 2016 05:15:12 GMT

thanks @Neeraj Sabharwal -- are u suggesting that I can modify this code on the hortonworks shell?

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

nsabharwal — Fri, 05 Feb 2016 05:16:57 GMT

@keerthana gajarajakumar You can run smoke test to just test.

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Fri, 05 Feb 2016 05:19:04 GMT

ok. Is there a way for me to code a program from scratch using the shell?

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Fri, 05 Feb 2016 05:44:43 GMT

@Artem Ervits -- Hi Artem.. Can you help me out with this question?

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

nsabharwal — Fri, 05 Feb 2016 05:57:57 GMT

@keerthana gajarajakumar You can follow this turotrial step by step

http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

aervits — Fri, 05 Feb 2016 07:58:12 GMT

@keerthana gajarajakumar

this is how I started learning Storm. Link. Feel free to try it and report any issues. Michael Noll's blog is an excellent resource on Storm and now Kafka. I built my first Hadoop cluster thanks to his site. He has some beginner posts Here

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

aervits — Fri, 05 Feb 2016 08:00:33 GMT

@keerthana gajarajakumar also if you are in NYC we are having a meetup with Storm user group on the 17th, welcome!

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Fri, 05 Feb 2016 08:05:00 GMT

Thanks for the suggestion @Artem Ervits -- I wont be in NY during that time, but thanks for letting me know:)

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

aervits — Fri, 05 Feb 2016 08:09:09 GMT

@keerthana gajarajakumar here's a sample in python. Link if you notice it is in the examples folder of the Storm project.

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Fri, 05 Feb 2016 08:38:04 GMT

@Artem Ervits -- Artem.. Once I customize the ExclamationTopology.java file to meet my needs how can I make the shell point to it on my system and execute? Any thoughts?

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Fri, 05 Feb 2016 08:41:22 GMT

Yes Artem I noticed this. Now I want my program to spit out spout and bolt to stdout so if I customize this code to do that how can I run it on the shell? Am I clear or is my question confusing Artem?

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

aervits — Fri, 05 Feb 2016 08:43:49 GMT

If you use LocalCluster you can execute in IntelliJ or any other IDE. Just make sure in your pom you set scope compile. When you are ready to run on Sandbox, you can scp the jar to sandbox and run the job by specifying storm jar command. Here's more info Link. Just make sure when you compile for submission to real cluster and not LocalCluster, set scope back to provided for Storm dependency. The documentation in the link I gave you is comprehensive. Read through it, it will save you a lot of time. @keerthana gajarajakumar

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Fri, 05 Feb 2016 09:18:26 GMT

Thanks Neeraj.

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Fri, 05 Feb 2016 09:23:06 GMT

Totally appreciate your feedback. Just noticed this reply of urs. Thanks!

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

aervits — Fri, 05 Feb 2016 09:39:55 GMT

@keerthana gajarajakumar sure thing, let me know how it works out.

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Fri, 05 Feb 2016 15:40:21 GMT

@Artem Ervits -- I created the StormTest.jar file and it's saved in system within C:/.... Now how do I copy this onto the sandbox? I'm having difficulty pointing to this location from the hortonworks shell.

[root@sandbox ~]# cp C:/Users/.IdeaIC15/config/StormTest.jar /usr/hdp

cp: cannot stat `C:/Users/.IdeaIC15/config/StormTest.jar': No such file or directory

How do I fix this issue? I would like to run this jar file from the shell. Your thoughts will be helpful.

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

aervits — Fri, 05 Feb 2016 19:44:35 GMT

@keerthana gajarajakumar you need to have a tool like putty or cygwin and then use SCP tool to upload jar. scp -P 2222 -r file root@127.0.0.1:

Your file will then be in /root directory

Re: How can I create a simple topology? I want to write this program using python on the hortonworks shell.

keerthanakumar2 — Sat, 06 Feb 2016 00:41:20 GMT

Oh ya. Didn't think about that option. Thanks!