Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

livy server with yarn cant connect

Highlighted

livy server with yarn cant connect

Super Collaborator

Hi:

today i donwload , configure and start livy server 0.2 i set this configuration but from notebook hue when I execute somenthing, it execute on local mode why??

livy.server.host = xxxxxxxxx
livy.server.port = 8998
livy.server.session.factory = yarn

also this environment

export SPARK_HOME=/usr/hdp/current/spark-client/
export HADOOP_CONF_DIR=/etc/hadoop/conf
16/08/11 00:24:33 INFO LivyServer: Using spark-submit version 1.6.0

16/08/11 00:24:33 WARN RequestLogHandler: !RequestLog

16/08/11 00:24:33 INFO WebServer: Starting server on http://xxxxxxx:8998

16/08/11 00:24:58 INFO InteractiveSession: Creating LivyClient for sessionId: 0

16/08/11 00:24:58 WARN RSCConf: Your hostname, xxxxxxx, resolves to a loopback address, but we couldn't find any external IP address!

16/08/11 00:24:58 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.

16/08/11 00:24:58 INFO SessionManager: Registering new session 0

16/08/11 00:24:59 INFO ContextLauncher: 16/08/11 00:24:59 INFO RSCDriver: Connecting to: xxxxxxx:39592

16/08/11 00:24:59 INFO ContextLauncher: 16/08/11 00:24:59 INFO RSCDriver: Starting RPC server...

16/08/11 00:24:59 INFO ContextLauncher: 16/08/11 00:24:59 WARN RSCConf: Your hostname, xxxxxxxxx, resolves to a loopback address, but we couldn't find any external IP address!

16/08/11 00:24:59 INFO ContextLauncher: 16/08/11 00:24:59 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.

16/08/11 00:24:59 INFO ContextLauncher: 16/08/11 00:24:59 INFO RSCDriver: Received job request 07ddeb9b-35ee-43ea-aa93-62b8c52bb670

16/08/11 00:24:59 INFO ContextLauncher: 16/08/11 00:24:59 INFO RSCDriver: SparkContext not yet up, queueing job request.

16/08/11 00:25:00 INFO ContextLauncher: 16/08/11 00:25:00 INFO SparkContext: Running Spark version 1.6.0

16/08/11 00:25:00 INFO ContextLauncher: 16/08/11 00:25:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/08/11 00:25:00 INFO ContextLauncher: 16/08/11 00:25:00 INFO SecurityManager: Changing view acls to: root

16/08/11 00:25:00 INFO ContextLauncher: 16/08/11 00:25:00 INFO SecurityManager: Changing modify acls to: root

16/08/11 00:25:00 INFO ContextLauncher: 16/08/11 00:25:00 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)

16/08/11 00:25:00 INFO ContextLauncher: 16/08/11 00:25:00 INFO Utils: Successfully started service 'sparkDriver' on port 43740.

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO Slf4jLogger: Slf4jLogger started

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO Remoting: Starting remoting

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.1.246.19:38730]

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 38730.

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO SparkEnv: Registering MapOutputTracker

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO SparkEnv: Registering BlockManagerMaster

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-d58ba9b0-c327-435a-97a5-94d1690e065b

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO MemoryStore: MemoryStore started with capacity 1983.0 MB

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO SparkEnv: Registering OutputCommitCoordinator

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO Server: jetty-8.y.z-SNAPSHOT

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO Utils: Successfully started service 'SparkUI' on port 4040.

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO SparkUI: Started SparkUI at http://10.1.246.19:4040

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO HttpFileServer: HTTP File server directory is /tmp/spark-169f91e3-6967-44a0-a328-eaf46a234a4e/httpd-fd528842-1fe5-488d-b388-c49d347e6e3a

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO HttpServer: Starting HTTP Server

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO Server: jetty-8.y.z-SNAPSHOT

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO AbstractConnector: Started SocketConnector@0.0.0.0:36775

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO Utils: Successfully started service 'HTTP file server' on port 36775.

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO SparkContext: Added JAR file:/opt/livy-server-0.2.0/rsc-jars/livy-api-0.2.0.jar at http://10.1.246.19:36775/jars/livy-api-0.2.0.jar with timestamp 1470867901795

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO SparkContext: Added JAR file:/opt/livy-server-0.2.0/rsc-jars/livy-rsc-0.2.0.jar at http://10.1.246.19:36775/jars/livy-rsc-0.2.0.jar with timestamp 1470867901797

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO SparkContext: Added JAR file:/opt/livy-server-0.2.0/rsc-jars/netty-all-4.0.23.Final.jar at http://10.1.246.19:36775/jars/netty-all-4.0.23.Final.jar with timestamp 1470867901800

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO SparkContext: Added JAR file:/opt/livy-server-0.2.0/repl-jars/livy-repl-0.2.0.jar at http://10.1.246.19:36775/jars/livy-repl-0.2.0.jar with timestamp 1470867901801

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO SparkContext: Added JAR file:/opt/livy-server-0.2.0/repl-jars/commons-codec-1.9.jar at http://10.1.246.19:36775/jars/commons-codec-1.9.jar with timestamp 1470867901801

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO SparkContext: Added JAR file:/opt/livy-server-0.2.0/repl-jars/livy-core-0.2.0.jar at http://10.1.246.19:36775/jars/livy-core-0.2.0.jar with timestamp 1470867901802

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO Executor: Starting executor ID driver on host localhost

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 50381.

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO NettyBlockTransferService: Server created on 50381

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO BlockManagerMaster: Trying to register BlockManager

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO BlockManagerMasterEndpoint: Registering block manager localhost:50381 with 1983.0 MB RAM, BlockManagerId(driver, localhost, 50381)

16/08/11 00:25:01 INFO ContextLauncher: 16/08/11 00:25:01 INFO BlockManagerMaster: Registered BlockManager

16/08/11 00:25:02 INFO ContextLauncher: 16/08/11 00:25:02 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.

16/08/11 00:25:02 INFO ContextLauncher: 16/08/11 00:25:02 INFO EventLoggingListener: Logging events to hdfs:///spark-history/local-1470867901834

16/08/11 00:25:02 INFO RSCClient: Received result for 07ddeb9b-35ee-43ea-aa93-62b8c52bb670

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 310.1 KB, free 310.1 KB)

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 26.4 KB, free 336.5 KB)

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:50381 (size: 26.4 KB, free: 1983.0 MB)

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO SparkContext: Created broadcast 0 from textFile at NativeMethodAccessorImpl.java:-2

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO FileInputFormat: Total input paths to process : 0

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO FileOutputCommitter: File Output Committer Algorithm version is 1

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO SparkContext: Starting job: saveAsTextFile at NativeMethodAccessorImpl.java:-2

16/08/11 00:25:05 INFO ContextLauncher: 16/08/11 00:25:05 INFO DAGScheduler: Job 0 finished: saveAsTextFile at NativeMethodAccessorImpl.java:-2, took 0.001895 s

2 REPLIES 2

Re: livy server with yarn cant connect

Expert Contributor

@Roberto Sancho

Did you find solution and what was the fix to run it in yarn-client mode.

Re: livy server with yarn cant connect

Rising Star

@Roberto Sancho check you zeppelin config. It is there where export MASTER=yarn-client is defined in the zeppelin-env. This would not be set in livy I do not believe but in zeppelin process. I hope this helps.

Don't have an account?
Coming from Hortonworks? Activate your account here