Member since
06-18-2015
55
Posts
34
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1348 | 03-04-2016 02:39 AM | |
| 1905 | 12-29-2015 09:42 AM |
02-15-2016
08:14 AM
1 Kudo
@Artem Ervits : Can you please help me with issue. What settings should I change to view the logs in Web UI ? Thanks
... View more
02-15-2016
08:08 AM
1 Kudo
@Neeraj Sabharwal :I am able to view jobs in Spark history server but unable to view current running jobs in Spark Web UI even I though I am specifying --master yarn-client or --master yarn When the job finishes I could see the " This webpage is not available ERR_CONNECTION_REFUSED " And When job running it displays the page but doesnt show any running jobs . Am I missing any settings ?
... View more
02-15-2016
04:24 AM
1 Kudo
@Neeraj Sabharwal : I am running my spark below : spark-shell --properties-file /TestDivya/Spark/Oracle.properties --jars /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --driver-class-path /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --packages com.databricks:spark-csv_2.10:1.1.0 --master yarn-client -i /TestDivya/Spark/Test.scala Do I need to modify my options to view jobs in Spark UI ?
... View more
02-15-2016
02:58 AM
1 Kudo
Hi, I am running HDP-2.3.4 cluster on EC2.When I go to http://ec2-xx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com:4041/jobs/ ,it doesnt display any currently running jobs. Whereas in spark-shell ,it shows the spark UI started at 4041 16/02/14 21:45:13 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041
16/02/14 21:45:13 INFO Utils: Successfully started service 'SparkUI' on port 4041.
16/02/14 21:45:13 INFO SparkUI: Started SparkUI at http://xxx.xx.xx.xxx:4041 But if I go to history server at It shows all the old jobs ran in the past . I am new bee to Spark. P.S. Attaching the screen shots of Spark Web UI and History UI Thanks, Divya
... View more
Labels:
- Labels:
-
Apache Spark
02-15-2016
02:37 AM
1 Kudo
Thanks Ali for the help
... View more
02-04-2016
04:07 AM
1 Kudo
16/02/03 22:55:25 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$connect$1(JettyUtils.scala:237)
at org.apache.spark.ui.JettyUtils$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.ui.JettyUtils$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.util.Utils$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:247)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
at org.apache.spark.SparkContext$anonfun$13.apply(SparkContext.scala:474)
at org.apache.spark.SparkContext$anonfun$13.apply(SparkContext.scala:474)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:474)
at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017)
at $line3.$read$iwC$iwC.<init>(<console>:9)
at $line3.$read$iwC.<init>(<console>:18)
at $line3.$read.<init>(<console>:20)
at $line3.$read$.<init>(<console>:24)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.<init>(<console>:7)
at $line3.$eval$.<clinit>(<console>)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply$mcZ$sp(SparkILoop.scala:991)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:685)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/02/03 22:55:25 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@7509bf4e: java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$connect$1(JettyUtils.scala:237)
at org.apache.spark.ui.JettyUtils$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.ui.JettyUtils$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.util.Utils$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:247)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
at org.apache.spark.SparkContext$anonfun$13.apply(SparkContext.scala:474)
at org.apache.spark.SparkContext$anonfun$13.apply(SparkContext.scala:474)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:474)
at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017)
at $line3.$read$iwC$iwC.<init>(<console>:9)
at $line3.$read$iwC.<init>(<console>:18)
at $line3.$read.<init>(<console>:20)
at $line3.$read$.<init>(<console>:24)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.<init>(<console>:7)
at $line3.$eval$.<clinit>(<console>)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply$mcZ$sp(SparkILoop.scala:991)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:685)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/st
[ec2-user@ip-172-31-29-201 ~]$ netstat -pnat | grep 404 (No info could be read for "-p": geteuid()=1000 but you should be root.)
tcp 1 0 xxx.xx.xx.xxx:34047 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp 1 0 xxx.xx.xx.xxx:34049 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp 1 0 xxx.xx.xx.xxx:34046 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp 1 0 xxx.xx.xx.xxx:34048 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp6 0 0 :::4040 :::* LISTEN -
tcp6 0 0 :::4041 :::* LISTEN -
tcp6 1 0 xxx.xx.xx.xxx:34043 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp6 1 0 xxx.xx.xx.xxx:34045 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp6 1 0 xxx.xx.xx.xxx:34042 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp6 1 0 xxx.xx.xx.xxx:34044 xxx.xx.xx.xxx:6188 CLOSE_WAIT Any pointers Appreciated .
... View more
Labels:
- Labels:
-
Apache Spark
02-01-2016
04:08 AM
2 Kudos
@Pradeep Allu If your parameter parameter file is in hdfs then make sure your pig script should also be in hdfs not in local file system. Try placing your pig script also in HDFS .
... View more
01-28-2016
07:21 AM
1 Kudo
@Artem Ervits I tried doing that but still getting page cannot be displayed . Do I need to make any settings in config files aka YARN-default.xml ,etc.. I try to look up for port 8042 in configuration files of YARN but could not find the property itself arn.nodemanager.webapp.address ${yarn.nodemanager.hostname}:8042
... View more
01-21-2016
02:32 AM
1 Kudo
Hi, I am newbee to EC2 and HDP 2.3.4 . When I try to view the application logs as its taking the internal IP of EC2 of one of the datanode,I get "web page cant be displayed error". When I searched the yarn-site.xml I couldnt find the port 8042 where I can make changes for log files. Can somebody could help me configuring the Application log address. Refer to the screenshot attached
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
12-29-2015
09:42 AM
Finally resolved the issue the input data was not correct format so when
I was using Timestamp/DateType its was returning empty result set.
... View more