Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spark submit to kafka not working

Highlighted

spark submit to kafka not working

New Contributor

We have a kerberized hdp 3.1.4 cluster and able to access all the services using keytab, even sample spark jobs able to submit without any issue, but we are seeing failures when spark jobs with kafkawordcount

Below is the log

 

spark-secure-kafka-app]$ spark2-submit \
> --num-executors 2 \
> --master yarn \
> --deploy-mode cluster \
> --class com.cloudera.spark.examples.DirectKafkaWordCount \
> ./target/spark-secure-kafka-app-1.0-SNAPSHOT-jar-with-dependencies.jar
-bash: spark2-submit: command not found
[hraj@stg2139 spark-secure-kafka-app]$ spark-submit --num-executors 2 --master yarn --deploy-mode cluster --class com.cloudera.spark.examples.DirectKafkaWordCount ./target/spark-secure-kafka-app-1.0-SNAPSHOT-jar-with-dependencies.jar
19/12/09 08:14:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/12/09 08:14:08 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
19/12/09 08:14:08 INFO RMProxy: Connecting to ResourceManager at stg-r640-01.txn.prod.example.com/10.135.93.35:8050
19/12/09 08:14:08 INFO Client: Requesting a new application from cluster with 4 NodeManagers
19/12/09 08:14:08 INFO Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.4.0-315/0/resource-types.xml
19/12/09 08:14:08 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (270336 MB per container)
19/12/09 08:14:08 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
19/12/09 08:14:08 INFO Client: Setting up container launch context for our AM
19/12/09 08:14:08 INFO Client: Setting up the launch environment for our AM container
19/12/09 08:14:08 INFO Client: Preparing resources for our AM container
19/12/09 08:14:08 INFO HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-823996690_1, ugi=ezlaketest@HDPSTG.txn.PROD.example.COM (auth:KERBEROS)]]
19/12/09 08:14:08 INFO DFSClient: Created token for ezlaketest: HDFS_DELEGATION_TOKEN owner=ezlaketest@HDPSTG.txn.PROD.example.COM, renewer=yarn, realUser=, issueDate=1575900848964, maxDate=1576505648964, sequenceNumber=1080, masterKeyId=136 on 10.135.93.35:8020
19/12/09 08:14:09 INFO KMSClientProvider: New token created: (Kind: kms-dt, Service: kms://http@stg-r640-02.txn.prod.example.com:9292/kms, Ident: (kms-dt owner=ezlaketest, renewer=yarn, realUser=, issueDate=1575900849121, maxDate=1576505649121, sequenceNumber=93, masterKeyId=14))
19/12/09 08:14:10 INFO metastore: Trying to connect to metastore with URI thrift://stg-r640-01.txn.prod.example.com:9083
19/12/09 08:14:10 INFO metastore: Connected to metastore.
19/12/09 08:14:12 INFO Client: Use hdfs cache file as spark.yarn.archive for HDP, hdfsCacheFile:hdfs://stg-r640-01.txn.prod.example.com:8020/hdp/apps/3.1.4.0-315/spark2/spark2-hdp-yarn-archive.tar.gz
19/12/09 08:14:12 INFO Client: Source and destination file systems are the same. Not copying hdfs://stg-r640-01.txn.prod.example.com:8020/hdp/apps/3.1.4.0-315/spark2/spark2-hdp-yarn-archive.tar.gz
19/12/09 08:14:12 INFO Client: Distribute hdfs cache file as spark.sql.hive.metastore.jars for HDP, hdfsCacheFile:hdfs://stg-r640-01.txn.prod.example.com:8020/hdp/apps/3.1.4.0-315/spark2/spark2-hdp-hive-archive.tar.gz
19/12/09 08:14:12 INFO Client: Source and destination file systems are the same. Not copying hdfs://stg-r640-01.txn.prod.example.com:8020/hdp/apps/3.1.4.0-315/spark2/spark2-hdp-hive-archive.tar.gz
19/12/09 08:14:12 INFO Client: Uploading resource file:/example/home/hraj/spark-secure-kafka-app/target/spark-secure-kafka-app-1.0-SNAPSHOT-jar-with-dependencies.jar -> hdfs://stg-r640-01.txn.prod.example.com:8020/user/ezlaketest/.sparkStaging/application_1574862672514_0068/spark-secure-kafka-app-1.0-SNAPSHOT-jar-with-dependencies.jar
19/12/09 08:14:13 INFO Client: Uploading resource file:/tmp/spark-8a0a36d3-d361-4b93-abe0-263450193201/__spark_conf__7495589881279546089.zip -> hdfs://stg-r640-01.txn.prod.example.com:8020/user/ezlaketest/.sparkStaging/application_1574862672514_0068/__spark_conf__.zip
19/12/09 08:14:13 INFO SecurityManager: Changing view acls to: hraj,ezlaketest
19/12/09 08:14:13 INFO SecurityManager: Changing modify acls to: hraj,ezlaketest
19/12/09 08:14:13 INFO SecurityManager: Changing view acls groups to:
19/12/09 08:14:13 INFO SecurityManager: Changing modify acls groups to:
19/12/09 08:14:13 INFO SecurityManager: SecurityManager: authentication disabled; ui acls enabled; users with view permissions: Set(hraj, ezlaketest); groups with view permissions: Set(); users with modify permissions: Set(hraj, ezlaketest); groups with modify permissions: Set()
19/12/09 08:14:13 INFO Client: Submitting application application_1574862672514_0068 to ResourceManager
19/12/09 08:14:13 INFO YarnClientImpl: Submitted application application_1574862672514_0068
19/12/09 08:14:14 INFO Client: Application report for application_1574862672514_0068 (state: ACCEPTED)
19/12/09 08:14:14 INFO Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1575900853543
final status: UNDEFINED
tracking URL: http://stg-r640-01.txn.prod.example.com:8088/proxy/application_1574862672514_0068/
user: ezlaketest
19/12/09 08:14:15 INFO Client: Application report for application_1574862672514_0068 (state: ACCEPTED)
19/12/09 08:14:16 INFO Client: Application report for application_1574862672514_0068 (state: ACCEPTED)
19/12/09 08:14:17 INFO Client: Application report for application_1574862672514_0068 (state: ACCEPTED)
19/12/09 08:14:18 INFO Client: Application report for application_1574862672514_0068 (state: ACCEPTED)
19/12/09 08:14:19 INFO Client: Application report for application_1574862672514_0068 (state: ACCEPTED)
19/12/09 08:14:20 INFO Client: Application report for application_1574862672514_0068 (state: ACCEPTED)
19/12/09 08:14:21 INFO Client: Application report for application_1574862672514_0068 (state: ACCEPTED)
19/12/09 08:14:22 INFO Client: Application report for application_1574862672514_0068 (state: FAILED)
19/12/09 08:14:22 INFO Client:
client token: N/A
diagnostics: Application application_1574862672514_0068 failed 2 times due to AM Container for appattempt_1574862672514_0068_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2019-12-09 08:14:22.481]Exception from container-launch.
Container id: container_e20_1574862672514_0068_02_000001
Exit code: 1
Exception message: Launch container failed
Shell output: main : command provided 1
main : run as user is ezlaketest
main : requested yarn user is ezlaketest
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file /u01/hadoop/yarn/local/nmPrivate/application_1574862672514_0068/container_e20_1574862672514_0068_02_000001/container_e20_1574862672514_0068_02_000001.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
Getting exit code file...
Creating script paths...


[2019-12-09 08:14:22.484]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/u01/hadoop/yarn/local/filecache/15/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/12/09 08:14:20 INFO SignalUtils: Registered signal handler for TERM
19/12/09 08:14:20 INFO SignalUtils: Registered signal handler for HUP
19/12/09 08:14:20 INFO SignalUtils: Registered signal handler for INT
19/12/09 08:14:20 INFO SecurityManager: Changing view acls to: ezlaketest
19/12/09 08:14:20 INFO SecurityManager: Changing modify acls to: ezlaketest
19/12/09 08:14:20 INFO SecurityManager: Changing view acls groups to:
19/12/09 08:14:20 INFO SecurityManager: Changing modify acls groups to:
19/12/09 08:14:20 INFO SecurityManager: SecurityManager: authentication disabled; ui acls enabled; users with view permissions: Set(ezlaketest); groups with view permissions: Set(); users with modify permissions: Set(ezlaketest); groups with modify permissions: Set()
19/12/09 08:14:21 INFO ApplicationMaster: Preparing Local resources
19/12/09 08:14:22 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1574862672514_0068_000002
19/12/09 08:14:22 INFO ApplicationMaster: Starting the user application in a separate Thread
19/12/09 08:14:22 INFO ApplicationMaster: Waiting for spark context initialization...

Usage: DirectKafkaWordCount <brokers> <topics>
<brokers> is a list of one or more Kafka brokers
<topics> is a list of one or more kafka topics to consume from
<ssl> true if using SSL, false otherwise.


19/12/09 08:14:22 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Shutdown hook called before final status was reported.)
19/12/09 08:14:22 INFO ApplicationMaster: Deleting staging directory hdfs://stg-r640-01.txn.prod.example.com:8020/user/ezlaketest/.sparkStaging/application_1574862672514_0068
19/12/09 08:14:22 INFO ShutdownHookManager: Shutdown hook called


[2019-12-09 08:14:22.486]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/u01/hadoop/yarn/local/filecache/15/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/12/09 08:14:20 INFO SignalUtils: Registered signal handler for TERM
19/12/09 08:14:20 INFO SignalUtils: Registered signal handler for HUP
19/12/09 08:14:20 INFO SignalUtils: Registered signal handler for INT
19/12/09 08:14:20 INFO SecurityManager: Changing view acls to: ezlaketest
19/12/09 08:14:20 INFO SecurityManager: Changing modify acls to: ezlaketest
19/12/09 08:14:20 INFO SecurityManager: Changing view acls groups to:
19/12/09 08:14:20 INFO SecurityManager: Changing modify acls groups to:
19/12/09 08:14:20 INFO SecurityManager: SecurityManager: authentication disabled; ui acls enabled; users with view permissions: Set(ezlaketest); groups with view permissions: Set(); users with modify permissions: Set(ezlaketest); groups with modify permissions: Set()
19/12/09 08:14:21 INFO ApplicationMaster: Preparing Local resources
19/12/09 08:14:22 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1574862672514_0068_000002
19/12/09 08:14:22 INFO ApplicationMaster: Starting the user application in a separate Thread
19/12/09 08:14:22 INFO ApplicationMaster: Waiting for spark context initialization...

Usage: DirectKafkaWordCount <brokers> <topics>
<brokers> is a list of one or more Kafka brokers
<topics> is a list of one or more kafka topics to consume from
<ssl> true if using SSL, false otherwise.


19/12/09 08:14:22 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Shutdown hook called before final status was reported.)
19/12/09 08:14:22 INFO ApplicationMaster: Deleting staging directory hdfs://stg-r640-01.txn.prod.example.com:8020/user/ezlaketest/.sparkStaging/application_1574862672514_0068
19/12/09 08:14:22 INFO ShutdownHookManager: Shutdown hook called


For more detailed output, check the application tracking page: http://stg-r640-01.txn.prod.example.com:8088/cluster/app/application_1574862672514_0068 Then click on links to logs of each attempt.
. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1575900853543
final status: FAILED
tracking URL: http://stg-r640-01.txn.prod.example.com:8088/cluster/app/application_1574862672514_0068
user: ezlaketest
Exception in thread "main" org.apache.spark.SparkException: Application application_1574862672514_0068 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1269)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1627)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
19/12/09 08:14:22 INFO ShutdownHookManager: Shutdown hook called
19/12/09 08:14:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-8a0a36d3-d361-4b93-abe0-263450193201
19/12/09 08:14:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-fad200be-4db9-4e1f-b01f-b32fa0bcb5f7

 

Note: We are able to list/create/delete/discribe topics using kafka-topics.sh

 

Also below spark jobs working fine

 

spark-submit --conf spark.ego.uname=ezlaketest --conf spark.ego.keytab=./ezlaketest.keytab --class org.apache.spark.examples.SparkPi /usr/hdp/3.1.4.0-315/spark2/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar

 

spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --num-executors 1 --driver-memory 512m --executor-memory 512m --executor-cores 1 /usr/hdp/3.1.4.0-315/spark2/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar 10

Don't have an account?
Coming from Hortonworks? Activate your account here