Member since
02-09-2017
41
Posts
2
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1172 | 06-13-2017 02:52 AM | |
| 3170 | 02-10-2017 11:33 AM |
08-05-2019
02:59 AM
Hi,@Geoffrey Shelton Okot,I got it , and i am looking forward to arrival of CDP .
... View more
08-02-2019
02:55 AM
Looking forward to a good reply, thank you!
... View more
08-02-2019
02:54 AM
I have found that the version of HDP has not been updated since 2019.
... View more
03-29-2019
01:33 AM
I know, this is the problem that is displayed. During the installation process, I still use my local repo. I guess it may be a bug after Ambari upgrade!
... View more
03-29-2019
01:08 AM
Ambari Version is 2.6.1.0 and HDP Version is 2.6.4.0 .
... View more
03-28-2019
10:28 AM
When adding the service, I found that the local repo was not being used. Instead, a remote hdp repo is used.
... View more
03-28-2019
07:16 AM
Prepare to upgrade to Spark-2.2.0, but when Ambari adds services, only Spark 2.1.1.
... View more
09-06-2018
09:42 AM
https://community.hortonworks.com/articles/2659/exploring-apache-flink-with-hdp.html
... View more
05-25-2018
01:44 PM
1 Kudo
Hi,@Simran Kaur, I have the same problem, did you solve it?
... View more
11-13-2017
03:53 AM
This problem is caused by two reasons:
(1) Each node did not add this ### Linux user and added it to the yarn user group.
(2) nodemanager container directory permissions are not normal, this is due to the machine partition is not uniform.
Solve as follows
Execute on each machine --->
useradd -M ###
usermod -a -G supergroup ###
Finally, check each node machine node node nm directory permissions are the same!
... View more
06-13-2017
02:52 AM
Hi,@Sergey Soldatov,This question is solved! Thank you very much!
... View more
06-12-2017
01:40 AM
Hi,Team.When I run a program on a spark cluster to save RDD to phoenix, I always report this error. as follows: 17/06/12 09:35:46 ERROR kerberos.SaveAsPhoenixApp$: save to phoenix error :java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:232)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1083)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1074)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:994)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:985)
at org.apache.phoenix.spark.ProductRDDFunctions.saveToPhoenix(ProductRDDFunctions.scala:51)
at com.yjf.phoenix.kerberos.SaveAsPhoenixApp$.saveRDDToPhoenixSIMPLELOG(SaveAsPhoenixApp.scala:144)
at com.yjf.phoenix.kerberos.SaveAsPhoenixApp$.saveAsphoenixProcess(SaveAsPhoenixApp.scala:93)
at com.yjf.phoenix.kerberos.SaveAsPhoenixApp$.main(SaveAsPhoenixApp.scala:156)
at com.yjf.phoenix.kerberos.SaveAsPhoenixApp.main(SaveAsPhoenixApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
... 29 more
Ask for help, thank you very much!
... View more
Labels:
- Labels:
-
Apache Phoenix
-
Apache Spark
06-06-2017
08:30 AM
hi,@Akash S,Can you save the results of spark to the inside of phoenix?
... View more
06-06-2017
03:13 AM
error:
17/06/06 10:39:54 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
17/06/06 10:39:54 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
17/06/06 10:39:54 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
17/06/06 10:39:54 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
17/06/06 10:39:54 INFO zookeeper.ZooKeeper: Client environment:os.version=4.1.6-1.el6.x86_64
17/06/06 10:39:54 INFO zookeeper.ZooKeeper: Client environment:user.name=root
17/06/06 10:39:54 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
17/06/06 10:39:54 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/spark
17/06/06 10:39:54 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=bg-js-sz1-ib1:2181 sessionTimeout=90000 watcher=hconnection-0x157d3a450x0, quorum=bg-js-sz1-ib1:2181, baseZNode=/hbase
17/06/06 10:39:54 INFO zookeeper.ClientCnxn: Opening socket connection to server bg-js-sz1-ib1/120.132.23.12:2181. Will not attempt to authenticate using SASL (unknown error)
17/06/06 10:39:54 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /120.132.23.12:55725, server: bg-js-sz1-ib1/120.132.23.12:2181
17/06/06 10:39:54 INFO zookeeper.ClientCnxn: Session establishment complete on server bg-js-sz1-ib1/120.132.23.12:2181, sessionid = 0x25b5cf27b987009, negotiated timeout = 60000
17/06/06 10:39:54 WARN ipc.RpcControllerFactory: Cannot load configured "hbase.rpc.controllerfactory.class" (org.apache.hadoop.hbase.ipc.controller.ClientRpcControllerFactory) from hbase-site.xml, falling back to use default RpcControllerFactory
17/06/06 10:39:54 INFO metrics.Metrics: Initializing metrics system: phoenix
17/06/06 10:39:54 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
17/06/06 10:39:54 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
17/06/06 10:39:54 INFO impl.MetricsSystemImpl: phoenix metrics system started
17/06/06 10:39:54 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
17/06/06 10:40:43 INFO client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=48474 ms ago, cancelled=false, msg=
17/06/06 10:41:03 INFO client.RpcRetryingCaller: Call exception, tries=11, retries=35, started=68544 ms ago, cancelled=false, msg=
code:
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.phoenix.spark._
import org.slf4j.LoggerFactory
/**
* Created by root on 20170606.
* Update date:
* Time: 9:40 AM
* Project: ${PROJECT_HOME}
* Package: ${PACKAGE_HOME}
* Describle : save data to phoenix
*
* Result of Test: local test ok,remote test error
* Command:
**
*
spark-submit \
--master yarn-client \
--driver-memory 2g \
--num-executors 3 \
--driver-cores 2 \
--executor-cores 2 \
--executor-memory 2g \
--class com.yjf.phoenix.kerberos.SaveAsPhoenixApp \
--jars /home/spark/lib/phoenix-spark-4.7.0-HBase-0.98.jar,\
/home/spark/lib/hbase-common-1.2.0-cdh5.8.0.jar,\
/home/spark/lib/hbase-client-1.2.0-cdh5.8.0.jar,\
/home/spark/lib/hbase-server-1.2.0-cdh5.8.0.jar,\
/home/spark/lib/hbase-protocol-1.2.0-cdh5.8.0.jar,\
/home/spark/lib/htrace-core-3.2.0-incubating.jar,\
/home/spark/lib/guava-12.0.1.jar,\
/home/spark/lib/phoenix-core-4.7.0-HBase-0.98.jar \
--principal log2hadoop \
--keytab /home/spark/conf/log2hadoop_bgp.keytab \
/home/spark/lib/test-phonenix.jar yarn-client
*
*
*
*
* Email: jifei.yang@ngaa.com.cn
* Status：线上正在使用
* Machine ip:
* ^^集群----->192.168.1.1（001业务统计）
* ^^集群----->127.0.0.1 （002业务统计）
*
* Attention：
*
*
*/
object SaveAsPhoenixApp {
private val logger=LoggerFactory.getLogger(SaveAsPhoenixApp.getClass)
private val quorum="bg-js-sz1-ib1,bg-js-sz2-ib3,bg-js-sz1-ib2"
private val krb5 = "/home/spark/conf/krb5.conf"
private val user = "log2hadoop@EEENN.NET" //用户名称
private val keyPath = "/home/spark/conf/log2hadoop.keytab" //文件
def saveAsphoenixProcess(master:String):Unit={
System.setProperty("java.security.krb5.conf",krb5);
val conf=new SparkConf().setAppName("PhoenixSparkRDDApp")
.set("spark.app.id","PhoenixSparkRDDApp_id")
.setMaster(master)
val sc=new SparkContext(conf)
//save error
val dataSet = List(
("00059","www.tudou.com","134.0.0.22","2017-06-23 12:00:12",75,"2017-08-25"),
("00069","www.youku.com","197.1.0.23","2017-06-03 13:00:12",96,"2017-07-16"),
("00079","www.nggaa.com","187.0.6.32","2017-06-13 13:00:12",88,"2017-05-06"),
("00089","www.wec.com","138.0.3.62","2017-06-12 15:00:12",129,"2017-12-19")
)
val endData=sc.parallelize(dataSet)
val configuration = new Configuration()
//hbase
configuration.set("hbase.zookeeper.quorum", quorum)
configuration.set("hbase.rootdir", "/hbase")
configuration.set("hadoop.security.authentication", "kerberos")
configuration.set("hbase.security.authentication", "kerberos")
configuration.set("hbase.security.authorization", "true")
configuration.set("hbase.master.kerberos.principal", "hbase/_HOST@EEENN.NET")
configuration.set("hbase.thrift.kerberos.principal", "hbase/_HOST@EEENN.NET")
configuration.set("hbase.regionserver.kerberos.principal", "hbase/_HOST@EEENN.NET")
configuration.set("hbase.zookeeper.property.clientPort", "2181")
configuration.set("hbase.rpc.timeout", "10000")
configuration.set("hbase.client.retries.number", "5")
configuration.set("hbase.client.pause", "5000")
configuration.set("hbase.client.scanner.timeout.period", "50000")
logger.warn("开始认证")
try{
UserGroupInformation.setConfiguration(configuration)
UserGroupInformation.loginUserFromKeytab(user, keyPath)
logger.warn("认证成功!")
}catch {
case ex:Exception=>logger.error(s"auth login error :${ex.toString}")
}
logger.warn("save as phoenix")
val hbConf: Configuration = HBaseConfiguration.create(configuration)
endData.saveToPhoenix(
"PH_HOT_IP",
Array("CREATETIME","DOMAIN","IP","IPFIRSTTIME","IPACTIVENESS","DAY"),
conf=hbConf
)
logger.warn("save as phoenix ok !!!")
sc.stop()
}
def main(args: Array[String]): Unit = {
val Array(master)=args
saveAsphoenixProcess(master)
}
}
... View more
Labels:
- Labels:
-
Apache Phoenix
-
Apache Spark
03-10-2017
02:31 AM
This question bothers me for 2 days! ：）
... View more
03-10-2017
02:30 AM
Hi,Dale Bradman，I have the same mistakes as you, do you ask? Can you help me? Thank you!
... View more
03-09-2017
06:44 PM
Hi, Vivian ,Please help me, thank you! This question bothers me for 3 days!
... View more
03-09-2017
01:18 AM
Hi, Romainr,your problem solved? I have not solved yet. Help me! Thank you!
... View more
03-09-2017
01:16 AM
Hi, Vivian,My friend, your problem solved? I have not solved yet. Help me! Thank you!
... View more
02-27-2017
05:29 AM
Hi,@Mohamed Ashiq I have the same question, may I ask you to solve this problem? Thank you!
... View more
02-13-2017
08:10 PM
Thank you, you are right, when I create a kadmin user on each linux machine, you can successfully submit the task!
... View more
02-10-2017
11:33 AM
My problem is solved！ Thank you, spark submitted tasks, I do not need certification of the cluster, but the need for the implementation of the spark machine! Therefore, I need to remove these two parameters: spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn-client \
--executor-memory 1G \
--num-executors 1 \
--num-executors 2 \
--driver-memory 1g \
--executor-cores 1 \
#--principal kadmin/admin@NGAA.COM \
#--keytab /home/test/sparktest/princpal/sparkjob.keytab \
/opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar 12
Submit again to succeed! 17/02/10 16:18:33 INFO spark.SparkContext: Running Spark version 1.6.0
17/02/10 16:18:34 INFO spark.SecurityManager: Changing view acls to: root,hdfs
17/02/10 16:18:34 INFO spark.SecurityManager: Changing modify acls to: root,hdfs
17/02/10 16:18:34 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, hdfs); users with modify permissions: Set(root, hdfs)
17/02/10 16:18:34 INFO util.Utils: Successfully started service 'sparkDriver' on port 53300.
17/02/10 16:18:35 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/02/10 16:18:35 INFO Remoting: Starting remoting
17/02/10 16:18:35 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.10.100.53:59243]
17/02/10 16:18:35 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.10.100.53:59243]
17/02/10 16:18:35 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 59243.
17/02/10 16:18:35 INFO spark.SparkEnv: Registering MapOutputTracker
17/02/10 16:18:35 INFO spark.SparkEnv: Registering BlockManagerMaster
17/02/10 16:18:35 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-1521d8d2-ce43-4c6e-8068-af08ed953b77
17/02/10 16:18:35 INFO storage.MemoryStore: MemoryStore started with capacity 530.3 MB
17/02/10 16:18:35 INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/02/10 16:18:36 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
17/02/10 16:18:36 INFO ui.SparkUI: Started SparkUI at http://10.10.100.53:4040
17/02/10 16:18:36 INFO spark.SparkContext: Added JAR file:/opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar at spark://10.10.100.53:53300/jars/spark-examples.jar with timestamp 1486714716370
17/02/10 16:18:36 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/10.10.100.51:8032
17/02/10 16:18:37 INFO yarn.Client: Requesting a new application from cluster with 4 NodeManagers
17/02/10 16:18:37 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/02/10 16:18:37 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
17/02/10 16:18:37 INFO yarn.Client: Setting up container launch context for our AM
17/02/10 16:18:37 INFO yarn.Client: Setting up the launch environment for our AM container
17/02/10 16:18:37 INFO yarn.Client: Preparing resources for our AM container
17/02/10 16:18:38 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://hadoop2:8020/user/hdfs/.sparkStaging/application_1486705141135_0008
17/02/10 16:18:38 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 52 for hdfs on 10.10.100.52:8020
17/02/10 16:18:39 INFO hive.metastore: Trying to connect to metastore with URI thrift://hadoop1:9083
17/02/10 16:18:39 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/02/10 16:18:39 INFO hive.metastore: Connected to metastore.
17/02/10 16:18:39 INFO hive.metastore: Closed a connection to metastore, current connections: 0
17/02/10 16:18:39 INFO yarn.Client: Uploading resource file:/tmp/spark-f6434659-beb9-437c-b233-8667c48702b9/__spark_conf__2828602694267011736.zip -> hdfs://hadoop2:8020/user/hdfs/.sparkStaging/application_1486705141135_0008/__spark_conf__2828602694267011736.zip
17/02/10 16:18:40 INFO spark.SecurityManager: Changing view acls to: root,hdfs
17/02/10 16:18:40 INFO spark.SecurityManager: Changing modify acls to: root,hdfs
17/02/10 16:18:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, hdfs); users with modify permissions: Set(root, hdfs)
17/02/10 16:18:40 INFO yarn.Client: Submitting application 8 to ResourceManager
17/02/10 16:18:40 INFO impl.YarnClientImpl: Submitted application application_1486705141135_0008
17/02/10 16:18:41 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:41 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.hdfs
start time: 1486714720230
final status: UNDEFINED
tracking URL: http://hadoop1:8088/proxy/application_1486705141135_0008/
user: hdfs
17/02/10 16:18:42 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:43 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:44 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:45 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:46 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:47 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:48 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:48 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
17/02/10 16:18:48 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> hadoop1, PROXY_URI_BASES -> http://hadoop1:8088/proxy/application_1486705141135_0008), /proxy/application_1486705141135_0008
17/02/10 16:18:48 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/02/10 16:18:49 INFO yarn.Client: Application report for application_1486705141135_0008 (state: RUNNING)
17/02/10 16:18:49 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: N/A
ApplicationMaster host: 10.10.100.53
ApplicationMaster RPC port: 0
queue: root.users.hdfs
start time: 1486714720230
final status: UNDEFINED
tracking URL: http://hadoop1:8088/proxy/application_1486705141135_0008/
user: hdfs
17/02/10 16:18:49 INFO cluster.YarnClientSchedulerBackend: Application application_1486705141135_0008 has started running.
17/02/10 16:18:49 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 47910.
17/02/10 16:18:49 INFO netty.NettyBlockTransferService: Server created on 47910
17/02/10 16:18:49 INFO storage.BlockManager: external shuffle service port = 7337
17/02/10 16:18:49 INFO storage.BlockManagerMaster: Trying to register BlockManager
17/02/10 16:18:49 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.10.100.53:47910 with 530.3 MB RAM, BlockManagerId(driver, 10.10.100.53, 47910)
17/02/10 16:18:49 INFO storage.BlockManagerMaster: Registered BlockManager
17/02/10 16:18:49 INFO scheduler.EventLoggingListener: Logging events to hdfs://hadoop2:8020/user/spark/applicationHistory/application_1486705141135_0008
17/02/10 16:18:49 WARN spark.SparkContext: Dynamic Allocation and num executors both set, thus dynamic allocation disabled.
17/02/10 16:18:58 INFO cluster.YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (hadoop4:49020) with ID 1
17/02/10 16:18:58 INFO storage.BlockManagerMasterEndpoint: Registering block manager hadoop4:48173 with 530.3 MB RAM, BlockManagerId(1, hadoop4, 48173)
17/02/10 16:19:01 INFO cluster.YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (hadoop2:52352) with ID 2
17/02/10 16:19:01 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/02/10 16:19:01 INFO storage.BlockManagerMasterEndpoint: Registering block manager hadoop2:39922 with 530.3 MB RAM, BlockManagerId(2, hadoop2, 39922)
17/02/10 16:19:01 INFO spark.SparkContext: Starting job: reduce at SparkPi.scala:36
17/02/10 16:19:01 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:36) with 12 output partitions
17/02/10 16:19:01 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:36)
17/02/10 16:19:01 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/02/10 16:19:01 INFO scheduler.DAGScheduler: Missing parents: List()
17/02/10 16:19:01 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32), which has no missing parents
17/02/10 16:19:01 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1904.0 B, free 1904.0 B)
17/02/10 16:19:02 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1202.0 B, free 3.0 KB)
17/02/10 16:19:02 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.10.100.53:47910 (size: 1202.0 B, free: 530.3 MB)
17/02/10 16:19:02 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
17/02/10 16:19:02 INFO scheduler.DAGScheduler: Submitting 12 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32)
17/02/10 16:19:02 INFO cluster.YarnScheduler: Adding task set 0.0 with 12 tasks
17/02/10 16:19:02 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, hadoop2, partition 0,PROCESS_LOCAL, 2034 bytes)
17/02/10 16:19:02 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, hadoop4, partition 1,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:03 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop4:48173 (size: 1202.0 B, free: 530.3 MB)
17/02/10 16:19:04 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop2:39922 (size: 1202.0 B, free: 530.3 MB)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, hadoop4, partition 2,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 2231 ms on hadoop4 (1/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, hadoop2, partition 3,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, hadoop4, partition 4,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 2369 ms on hadoop2 (2/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 127 ms on hadoop4 (3/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, hadoop2, partition 5,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 108 ms on hadoop2 (4/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, hadoop4, partition 6,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 114 ms on hadoop4 (5/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, hadoop2, partition 7,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 85 ms on hadoop2 (6/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, hadoop4, partition 8,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 103 ms on hadoop4 (7/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, hadoop2, partition 9,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 89 ms on hadoop2 (8/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, hadoop4, partition 10,PROCESS_LOCAL, 2039 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, hadoop2, partition 11,PROCESS_LOCAL, 2040 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 109 ms on hadoop4 (9/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 83 ms on hadoop2 (10/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 90 ms on hadoop4 (11/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 77 ms on hadoop2 (12/12)
17/02/10 16:19:04 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:36) finished in 2.695 s
17/02/10 16:19:04 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/02/10 16:19:04 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:36, took 3.293783 s
Pi is roughly 3.1438333333333333
17/02/10 16:19:05 INFO ui.SparkUI: Stopped Spark web UI at http://10.10.100.53:4040
17/02/10 16:19:05 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
17/02/10 16:19:05 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
17/02/10 16:19:05 INFO cluster.YarnClientSchedulerBackend: Asking each executor to shut down
17/02/10 16:19:05 INFO cluster.YarnClientSchedulerBackend: Stopped
17/02/10 16:19:05 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/02/10 16:19:05 INFO storage.MemoryStore: MemoryStore cleared
17/02/10 16:19:05 INFO storage.BlockManager: BlockManager stopped
17/02/10 16:19:05 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
17/02/10 16:19:05 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/02/10 16:19:05 INFO spark.SparkContext: Successfully stopped SparkContext
17/02/10 16:19:05 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/02/10 16:19:05 INFO util.ShutdownHookManager: Shutdown hook called
17/02/10 16:19:05 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-f6434659-beb9-437c-b233-8667c48702b9
17/02/10 16:19:05 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
... View more
02-10-2017
08:13 AM
@rguruvannagari Hi, Spark submitted the task has been certified kadmin, do you think should be how to deal with it? Thank you
... View more
02-10-2017
06:35 AM
@mqureshi Thanks. The principal is kadmin,and I suspect that the yarn is missing the group. [root@hadoop1 princpal]# klist -kt sparkjob.keytab
Keytab name: FILE:sparkjob.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
3 02/06/17 19:01:40 kadmin/admin@NGAA.COM
3 02/06/17 19:01:40 kadmin/admin@NGAA.COM
3 02/06/17 19:01:40 kadmin/admin@NGAA.COM
3 02/06/17 19:01:40 kadmin/admin@NGAA.COM
3 02/06/17 19:01:40 kadmin/admin@NGAA.COM
3 02/06/17 19:01:40 kadmin/admin@NGAA.COM
... View more
02-10-2017
06:24 AM
@Neeraj Sabharwal Hi, I encountered a similar mistake, running spark, the user can not find!Please help me, thank you! spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn-client \
--executor-memory 1G \
--num-executors 1 \
--num-executors 2 \
--driver-memory 1g \
--executor-cores 1 \
--principal kadmin/admin@NGAA.COM \
--keytab /home/test/sparktest/princpal/sparkjob.keytab \
/opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar 12 error messages: 17/02/10 13:54:16 INFO security.UserGroupInformation: Login successful for user kadmin/admin@NGAA.COM using keytab file /home/test/sparktest/princpal/sparkjob.keytab
17/02/10 13:54:16 INFO spark.SparkContext: Running Spark version 1.6.0
17/02/10 13:54:16 INFO spark.SecurityManager: Changing view acls to: root,kadmin
17/02/10 13:54:16 INFO spark.SecurityManager: Changing modify acls to: root,kadmin
17/02/10 13:54:16 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, kadmin); users with modify permissions: Set(root, kadmin)
17/02/10 13:54:17 INFO util.Utils: Successfully started service 'sparkDriver' on port 56214.
17/02/10 13:54:17 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/02/10 13:54:17 INFO Remoting: Starting remoting
17/02/10 13:54:18 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.10.100.51:40936]
17/02/10 13:54:18 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.10.100.51:40936]
17/02/10 13:54:18 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 40936.
17/02/10 13:54:18 INFO spark.SparkEnv: Registering MapOutputTracker
17/02/10 13:54:18 INFO spark.SparkEnv: Registering BlockManagerMaster
17/02/10 13:54:18 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-cf37cdde-4eab-4804-b84b-b5f937828aa7
17/02/10 13:54:18 INFO storage.MemoryStore: MemoryStore started with capacity 530.3 MB
17/02/10 13:54:18 INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/02/10 13:54:19 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
17/02/10 13:54:19 INFO ui.SparkUI: Started SparkUI at http://10.10.100.51:4040
17/02/10 13:54:19 INFO spark.SparkContext: Added JAR file:/opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar at spark://10.10.100.51:56214/jars/spark-examples.jar with timestamp 1486706059601
17/02/10 13:54:19 INFO yarn.Client: Attempting to login to the Kerberos using principal: kadmin/admin@NGAA.COM and keytab: /home/test/sparktest/princpal/sparkjob.keytab
17/02/10 13:54:19 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/10.10.100.51:8032
17/02/10 13:54:20 INFO yarn.Client: Requesting a new application from cluster with 4 NodeManagers
17/02/10 13:54:20 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/02/10 13:54:20 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
17/02/10 13:54:20 INFO yarn.Client: Setting up container launch context for our AM
17/02/10 13:54:20 INFO yarn.Client: Setting up the launch environment for our AM container
17/02/10 13:54:21 INFO yarn.Client: Credentials file set to: credentials-79afe260-414b-4df7-8242-3cd1a279dbc7
17/02/10 13:54:21 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://hadoop2:8020/user/kadmin/.sparkStaging/application_1486705141135_0002
17/02/10 13:54:21 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 44 for kadmin on 10.10.100.52:8020
17/02/10 13:54:21 INFO yarn.Client: Renewal Interval set to 86400061
17/02/10 13:54:21 INFO yarn.Client: Preparing resources for our AM container
17/02/10 13:54:21 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://hadoop2:8020/user/kadmin/.sparkStaging/application_1486705141135_0002
17/02/10 13:54:21 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 45 for kadmin on 10.10.100.52:8020
17/02/10 13:54:22 INFO hive.metastore: Trying to connect to metastore with URI thrift://hadoop1:9083
17/02/10 13:54:22 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/02/10 13:54:22 INFO hive.metastore: Connected to metastore.
17/02/10 13:54:22 INFO hive.metastore: Closed a connection to metastore, current connections: 0
17/02/10 13:54:23 INFO yarn.Client: To enable the AM to login from keytab, credentials are being copied over to the AM via the YARN Secure Distributed Cache.
17/02/10 13:54:23 INFO yarn.Client: Uploading resource file:/home/test/sparktest/princpal/sparkjob.keytab -> hdfs://hadoop2:8020/user/kadmin/.sparkStaging/application_1486705141135_0002/sparkjob.keytab
17/02/10 13:54:23 INFO yarn.Client: Uploading resource file:/tmp/spark-79d08367-6f8d-4cb3-813e-d450e90a3128/__spark_conf__4615276915023723512.zip -> hdfs://hadoop2:8020/user/kadmin/.sparkStaging/application_1486705141135_0002/__spark_conf__4615276915023723512.zip
17/02/10 13:54:23 INFO spark.SecurityManager: Changing view acls to: root,kadmin
17/02/10 13:54:23 INFO spark.SecurityManager: Changing modify acls to: root,kadmin
17/02/10 13:54:23 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, kadmin); users with modify permissions: Set(root, kadmin)
17/02/10 13:54:23 INFO yarn.Client: Submitting application 2 to ResourceManager
17/02/10 13:54:23 INFO impl.YarnClientImpl: Submitted application application_1486705141135_0002
17/02/10 13:54:24 INFO yarn.Client: Application report for application_1486705141135_0002 (state: FAILED)
17/02/10 13:54:24 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1486705141135_0002 failed 2 times due to AM Container for appattempt_1486705141135_0002_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://hadoop1:8088/proxy/application_1486705141135_0002/Then, click on links to logs of each attempt.
Diagnostics: Application application_1486705141135_0002 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is kadmin
main : requested yarn user is kadmin
User kadmin not found
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.kadmin
start time: 1486706063635
final status: FAILED
tracking URL: http://hadoop1:8088/cluster/app/application_1486705141135_0002
user: kadmin
17/02/10 13:54:24 INFO yarn.Client: Deleting staging directory .sparkStaging/application_1486705141135_0002
17/02/10 13:54:24 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:541)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/02/10 13:54:25 INFO ui.SparkUI: Stopped Spark web UI at http://10.10.100.51:4040
17/02/10 13:54:25 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
17/02/10 13:54:25 INFO cluster.YarnClientSchedulerBackend: Asking each executor to shut down
17/02/10 13:54:25 INFO cluster.YarnClientSchedulerBackend: Stopped
17/02/10 13:54:25 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/02/10 13:54:25 ERROR util.Utils: Uncaught exception in thread main
java.lang.NullPointerException
at org.apache.spark.network.shuffle.ExternalShuffleClient.close(ExternalShuffleClient.java:152)
at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1231)
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:96)
at org.apache.spark.SparkContext$$anonfun$stop$12.apply$mcV$sp(SparkContext.scala:1767)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1230)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1766)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:613)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/02/10 13:54:25 INFO spark.SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:541)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/02/10 13:54:25 INFO storage.DiskBlockManager: Shutdown hook called
17/02/10 13:54:25 INFO util.ShutdownHookManager: Shutdown hook called
17/02/10 13:54:25 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-79d08367-6f8d-4cb3-813e-d450e90a3128/userFiles-58912a50-d060-42ec-8665-7a74c1be9a7b
17/02/10 13:54:25 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-79d08367-6f8d-4cb3-813e-d450e90a3128 Thanks
... View more
02-10-2017
06:18 AM
Hi,teams: When running the spark program, the user can not find!Please help me, thank you! spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn-client \
--executor-memory 1G \
--num-executors 1 \
--num-executors 2 \
--driver-memory 1g \
--executor-cores 1 \
--principal kadmin/admin@NGAA.COM \
--keytab /home/test/sparktest/princpal/sparkjob.keytab \
/opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar 12 error messages： 17/02/10 13:54:16 INFO security.UserGroupInformation: Login successful for user kadmin/admin@NGAA.COM using keytab file /home/test/sparktest/princpal/sparkjob.keytab
17/02/10 13:54:16 INFO spark.SparkContext: Running Spark version 1.6.0
17/02/10 13:54:16 INFO spark.SecurityManager: Changing view acls to: root,kadmin
17/02/10 13:54:16 INFO spark.SecurityManager: Changing modify acls to: root,kadmin
17/02/10 13:54:16 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, kadmin); users with modify permissions: Set(root, kadmin)
17/02/10 13:54:17 INFO util.Utils: Successfully started service 'sparkDriver' on port 56214.
17/02/10 13:54:17 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/02/10 13:54:17 INFO Remoting: Starting remoting
17/02/10 13:54:18 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.10.100.51:40936]
17/02/10 13:54:18 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.10.100.51:40936]
17/02/10 13:54:18 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 40936.
17/02/10 13:54:18 INFO spark.SparkEnv: Registering MapOutputTracker
17/02/10 13:54:18 INFO spark.SparkEnv: Registering BlockManagerMaster
17/02/10 13:54:18 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-cf37cdde-4eab-4804-b84b-b5f937828aa7
17/02/10 13:54:18 INFO storage.MemoryStore: MemoryStore started with capacity 530.3 MB
17/02/10 13:54:18 INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/02/10 13:54:19 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
17/02/10 13:54:19 INFO ui.SparkUI: Started SparkUI at http://10.10.100.51:4040
17/02/10 13:54:19 INFO spark.SparkContext: Added JAR file:/opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar at spark://10.10.100.51:56214/jars/spark-examples.jar with timestamp 1486706059601
17/02/10 13:54:19 INFO yarn.Client: Attempting to login to the Kerberos using principal: kadmin/admin@NGAA.COM and keytab: /home/test/sparktest/princpal/sparkjob.keytab
17/02/10 13:54:19 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/10.10.100.51:8032
17/02/10 13:54:20 INFO yarn.Client: Requesting a new application from cluster with 4 NodeManagers
17/02/10 13:54:20 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/02/10 13:54:20 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
17/02/10 13:54:20 INFO yarn.Client: Setting up container launch context for our AM
17/02/10 13:54:20 INFO yarn.Client: Setting up the launch environment for our AM container
17/02/10 13:54:21 INFO yarn.Client: Credentials file set to: credentials-79afe260-414b-4df7-8242-3cd1a279dbc7
17/02/10 13:54:21 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://hadoop2:8020/user/kadmin/.sparkStaging/application_1486705141135_0002
17/02/10 13:54:21 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 44 for kadmin on 10.10.100.52:8020
17/02/10 13:54:21 INFO yarn.Client: Renewal Interval set to 86400061
17/02/10 13:54:21 INFO yarn.Client: Preparing resources for our AM container
17/02/10 13:54:21 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://hadoop2:8020/user/kadmin/.sparkStaging/application_1486705141135_0002
17/02/10 13:54:21 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 45 for kadmin on 10.10.100.52:8020
17/02/10 13:54:22 INFO hive.metastore: Trying to connect to metastore with URI thrift://hadoop1:9083
17/02/10 13:54:22 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/02/10 13:54:22 INFO hive.metastore: Connected to metastore.
17/02/10 13:54:22 INFO hive.metastore: Closed a connection to metastore, current connections: 0
17/02/10 13:54:23 INFO yarn.Client: To enable the AM to login from keytab, credentials are being copied over to the AM via the YARN Secure Distributed Cache.
17/02/10 13:54:23 INFO yarn.Client: Uploading resource file:/home/test/sparktest/princpal/sparkjob.keytab -> hdfs://hadoop2:8020/user/kadmin/.sparkStaging/application_1486705141135_0002/sparkjob.keytab
17/02/10 13:54:23 INFO yarn.Client: Uploading resource file:/tmp/spark-79d08367-6f8d-4cb3-813e-d450e90a3128/__spark_conf__4615276915023723512.zip -> hdfs://hadoop2:8020/user/kadmin/.sparkStaging/application_1486705141135_0002/__spark_conf__4615276915023723512.zip
17/02/10 13:54:23 INFO spark.SecurityManager: Changing view acls to: root,kadmin
17/02/10 13:54:23 INFO spark.SecurityManager: Changing modify acls to: root,kadmin
17/02/10 13:54:23 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, kadmin); users with modify permissions: Set(root, kadmin)
17/02/10 13:54:23 INFO yarn.Client: Submitting application 2 to ResourceManager
17/02/10 13:54:23 INFO impl.YarnClientImpl: Submitted application application_1486705141135_0002
17/02/10 13:54:24 INFO yarn.Client: Application report for application_1486705141135_0002 (state: FAILED)
17/02/10 13:54:24 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1486705141135_0002 failed 2 times due to AM Container for appattempt_1486705141135_0002_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://hadoop1:8088/proxy/application_1486705141135_0002/Then, click on links to logs of each attempt.
Diagnostics: Application application_1486705141135_0002 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is kadmin
main : requested yarn user is kadmin
User kadmin not found
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.kadmin
start time: 1486706063635
final status: FAILED
tracking URL: http://hadoop1:8088/cluster/app/application_1486705141135_0002
user: kadmin
17/02/10 13:54:24 INFO yarn.Client: Deleting staging directory .sparkStaging/application_1486705141135_0002
17/02/10 13:54:24 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:541)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/02/10 13:54:25 INFO ui.SparkUI: Stopped Spark web UI at http://10.10.100.51:4040
17/02/10 13:54:25 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
17/02/10 13:54:25 INFO cluster.YarnClientSchedulerBackend: Asking each executor to shut down
17/02/10 13:54:25 INFO cluster.YarnClientSchedulerBackend: Stopped
17/02/10 13:54:25 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/02/10 13:54:25 ERROR util.Utils: Uncaught exception in thread main
java.lang.NullPointerException
at org.apache.spark.network.shuffle.ExternalShuffleClient.close(ExternalShuffleClient.java:152)
at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1231)
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:96)
at org.apache.spark.SparkContext$$anonfun$stop$12.apply$mcV$sp(SparkContext.scala:1767)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1230)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1766)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:613)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/02/10 13:54:25 INFO spark.SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:541)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/02/10 13:54:25 INFO storage.DiskBlockManager: Shutdown hook called
17/02/10 13:54:25 INFO util.ShutdownHookManager: Shutdown hook called
17/02/10 13:54:25 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-79d08367-6f8d-4cb3-813e-d450e90a3128/userFiles-58912a50-d060-42ec-8665-7a74c1be9a7b
17/02/10 13:54:25 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-79d08367-6f8d-4cb3-813e-d450e90a3
key poit: main : run as user is kadmin
main : requested yarn user is kadmin
User kadmin not found Thanks
... View more
- Tags:
- Hadoop Core
- Spark
- YARN
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
