How to run Rscript on Storm multilang

filipe_mendes · ‎03-08-2017

HDP version: 2.4.2.0-258

Storm version: 0.10.0.2.4

Storm package version: 1.2

R version: 3.3.2

Cluster: 5 Nodes

Overview: We are tried to integrate R with Storm based on Storm package available on CRAN (https://cran.r-project.org/web/packages/Storm/index.html) and also the example available on GitHub for the same package (https://github.com/allenday/R-Storm). R, Storm package and dependecies were installed in all nodes. The $PATH was also configure. We tried to execute the example from GitHub without success. A new Storm Topology was adapted to test this example. There are some errors capted from the worker, seems to be a problem with the premutebolt (R error?) and storm multilang.

Any ideia about how to fixe this errors, or a detailed tutorial for Rscript and Storm integration?

Best regards

2017-03-08 11:09:56.219 b.s.d.executor [INFO] Processing received message FOR 4 TUPLE: source: spout:5, stream: default, id: {}, [i am at two with nature]
2017-03-08 11:09:56.223 permutebolt [INFO] Error: invalid assignment for reference class field ‘input’, should be from class “list” or a subclass (was class “character”)
Execution halted


2017-03-08 11:09:56.224 b.s.d.executor [INFO] Execute done TUPLE source: spout:5, stream: default, id: {}, [i am at two with nature] TASK: 4 DELTA: 
2017-03-08 11:09:56.224 b.s.d.executor [INFO] Processing received message FOR 4 TUPLE: source: spout:5, stream: default, id: {}, [the cow jumped over the moon]
2017-03-08 11:09:56.227 b.s.d.executor [INFO] Execute done TUPLE source: spout:5, stream: default, id: {}, [the cow jumped over the moon] TASK: 4 DELTA: 3
2017-03-08 11:09:56.240 b.s.t.ShellBolt [ERROR] Halting process: ShellBolt died.
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read.
Serializer Exception:




	at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:101) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
2017-03-08 11:09:56.249 b.s.d.executor [ERROR] 
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read.
Serializer Exception:




	at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:101) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
2017-03-08 11:09:56.263 b.s.d.executor [INFO] Processing received message FOR 4 TUPLE: source: spout:5, stream: default, id: {}, [i am at two with nature]
2017-03-08 11:09:56.266 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: pid:31526, name:permutebolt exitCode:1, errorString: 
	at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at backtype.storm.daemon.executor$fn__5492$fn__5505$fn__5556.invoke(executor.clj:813) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at backtype.storm.util$async_loop$fn__644.invoke(util.clj:479) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: pid:31526, name:permutebolt exitCode:1, errorString: 
	at backtype.storm.task.ShellBolt.execute(ShellBolt.java:136) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at backtype.storm.daemon.executor$fn__5492$tuple_action_fn__5494.invoke(executor.clj:684) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at backtype.storm.daemon.executor$mk_task_receiver$fn__5415.invoke(executor.clj:431) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at backtype.storm.disruptor$clojure_handler$reify__4991.onEvent(disruptor.clj:58) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
	... 6 more