Member since
05-15-2017
3
Posts
0
Kudos Received
0
Solutions
05-21-2018
01:13 AM
There is a SQL I use spark as hive.execution.engine. I run it successfully in hive editor but failed in a workflow. 2018-05-21 15:29:38,583 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql. metadata.HiveException: Failed to create spark client. org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client. at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:119) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:129) at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:101) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10316) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10109) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:494) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1279) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1396) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1208) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1198) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:383) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:318) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:416) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:432) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:726) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:693) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:628) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:339) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:316) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:64) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:234) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) What should I do to find the deep reason?
... View more
05-16-2017
03:29 AM
You're right, the reason is that I didn't initialize a SparkContext until receiving a message from kafka.
... View more
05-15-2017
11:43 PM
My start script: #!/bin/bash set -ux class=com.palmaplus.amapdata.Main spark-submit \ --master yarn \ --deploy-mode cluster \ --num-executors 12 \ --driver-cores 1 \ --executor-cores 1 \ --driver-memory 4G \ --executor-memory 2G \ --conf spark.default.parallelism=24 \ --conf spark.shuffle.compress=false \ --conf spark.storage.memoryFraction=0.2 \ --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j-spark.properties -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/hadoop-yarn" \ --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j-spark.properties -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/hadoop-yarn" \ --class $class \ --files "/data/projects/superdb/conf/application.conf,/data/projects/superdb/conf/brand-blacklist.txt" \ /data/projects/superdb/jar/amap-data-1.0-SNAPSHOT-jar-with-dependencies.jar \ exit 0 =====================================console log below=============================== 17/05/16 14:37:11 INFO yarn.Client: Application report for application_1494307194668_0014 (state: ACCEPTED) 17/05/16 14:37:12 INFO yarn.Client: Application report for application_1494307194668_0014 (state: ACCEPTED) 17/05/16 14:37:13 INFO yarn.Client: Application report for application_1494307194668_0014 (state: FAILED) 17/05/16 14:37:13 INFO yarn.Client: client token: N/A diagnostics: Application application_1494307194668_0014 failed 2 times due to AM Container for appattempt_1494307194668_0014_000002 exited with exitCode: 15 For more detailed output, check application tracking page:http://master:8088/proxy/application_1494307194668_0014/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_1494307194668_0014_02_000001 Exit code: 15 Stack trace: ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:601) at org.apache.hadoop.util.Shell.run(Shell.java:504) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 15 Failing this attempt. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.users.hdfs start time: 1494916610862 final status: FAILED tracking URL: http://master:8088/cluster/app/application_1494307194668_0014 user: hdfs Exception in thread "main" org.apache.spark.SparkException: Application application_1494307194668_0014 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1213) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 17/05/16 14:37:13 INFO util.ShutdownHookManager: Shutdown hook called 17/05/16 14:37:13 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-d231fd6a-4752-4dc1-b041-080756d9c5aa + exit 0 ================yarn logs -applicationId application_1494307194668_0014========================= 17/05/16 14:39:14 INFO client.RMProxy: Connecting to ResourceManager at master/10.0.25.5:8032 Container: container_1494307194668_0014_02_000001 on slave2_8041 ================================================================== LogType:stderr Log Upload Time:Tue May 16 14:37:14 +0800 2017 LogLength:1742 Log Contents: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/yarn/nm/usercache/hdfs/filecache/43/__spark_libs__8927180205054833354.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/data/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/05/16 14:37:11 INFO SignalUtils: Registered signal handler for TERM 17/05/16 14:37:11 INFO SignalUtils: Registered signal handler for HUP 17/05/16 14:37:11 INFO SignalUtils: Registered signal handler for INT 17/05/16 14:37:12 INFO ApplicationMaster: Preparing Local resources 17/05/16 14:37:12 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1494307194668_0014_000002 17/05/16 14:37:12 INFO SecurityManager: Changing view acls to: yarn,hdfs 17/05/16 14:37:12 INFO SecurityManager: Changing modify acls to: yarn,hdfs 17/05/16 14:37:12 INFO SecurityManager: Changing view acls groups to: 17/05/16 14:37:12 INFO SecurityManager: Changing modify acls groups to: 17/05/16 14:37:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); groups with view permissions: Set(); users with modify permissions: Set(yarn, hdfs); groups with modify permissions: Set() 17/05/16 14:37:12 INFO ApplicationMaster: Starting the user application in a separate Thread 17/05/16 14:37:12 INFO ApplicationMaster: Waiting for spark context initialization... LogType:stdout Log Upload Time:Tue May 16 14:37:14 +0800 2017 LogLength:0 Log Contents: Container: container_1494307194668_0014_01_000001 on slave3_8041 ================================================================== LogType:stderr Log Upload Time:Tue May 16 14:37:13 +0800 2017 LogLength:1742 Log Contents: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/yarn/nm/usercache/hdfs/filecache/44/__spark_libs__8927180205054833354.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/data/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/05/16 14:36:58 INFO SignalUtils: Registered signal handler for TERM 17/05/16 14:36:58 INFO SignalUtils: Registered signal handler for HUP 17/05/16 14:36:58 INFO SignalUtils: Registered signal handler for INT 17/05/16 14:37:06 INFO ApplicationMaster: Preparing Local resources 17/05/16 14:37:07 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1494307194668_0014_000001 17/05/16 14:37:07 INFO SecurityManager: Changing view acls to: yarn,hdfs 17/05/16 14:37:07 INFO SecurityManager: Changing modify acls to: yarn,hdfs 17/05/16 14:37:07 INFO SecurityManager: Changing view acls groups to: 17/05/16 14:37:07 INFO SecurityManager: Changing modify acls groups to: 17/05/16 14:37:07 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); groups with view permissions: Set(); users with modify permissions: Set(yarn, hdfs); groups with modify permissions: Set() 17/05/16 14:37:07 INFO ApplicationMaster: Starting the user application in a separate Thread 17/05/16 14:37:07 INFO ApplicationMaster: Waiting for spark context initialization... LogType:stdout Log Upload Time:Tue May 16 14:37:13 +0800 2017 LogLength:0 Log Contents: Any help plz.
... View more
