Oozie job submit fails in HDP-3.0 with below error: Error: E0723 : E0723: Unsupported action type, node [hive] type [org.apache.oozie.service.ActionService] . It looks like failure is because Oozie is not aware of Hive action type. When I checked value of oozie.service.ActionService.executor.ext.classes in oozie-site.xml, I found that it was set to below value: <property> <name>oozie.service.ActionService.executor.ext.classes</name> <value> org.apache.oozie.action.email.EmailActionExecutor, org.apache.oozie.action.hadoop.ShellActionExecutor, org.apache.oozie.action.hadoop.SqoopActionExecutor, org.apache.oozie.action.hadoop.DistcpActionExecutor</value> </property> Note - Hive action type and other supported action types like Spark/Spark2 is missing in this. I'm still working with our engineering to find out why this is happening with recent Oozie version, earlier it used to work without any issues. I will keep you posted on this. . To fix this issue: Please modify oozie.service.ActionService.executor.ext.classes to have org.apache.oozie.action.hadoop.HiveActionExecutor class. Modified property: <property> <name>oozie.service.ActionService.executor.ext.classes</name> <value> org.apache.oozie.action.email.EmailActionExecutor, org.apache.oozie.action.hadoop.ShellActionExecutor, org.apache.oozie.action.hadoop.SqoopActionExecutor, org.apache.oozie.action.hadoop.DistcpActionExecutor, org.apache.oozie.action.hadoop.HiveActionExecutor</value> </property> After you modify this, please restart Oozie services via Ambari and try to resubmit your workflow. It should work without any issue. . I hope this will save your valuable time in troubleshooting! . Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more

When Oozie launcher(map only mapreduce job) gets scheduled to run on RHEL7 node in a mixed OS environment, it may get failed with below ERROR(stderr section of Oozie launcher logs): Container: container_e1XX_XXXXXXX_0X_00000X on XXXXXX_XXX_XXXXXX LogAggregationType: AGGREGATED =============================================================================================================== LogType:stderr Log Upload Time:Tue XXXXXXXXXXXX LogLength:XX Log Contents: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. /usr/bin/env: bash: No such file or directory Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. . Why it failed? This happens because "/bin" and "/sbin" missing in your $PATH in container launch environment. $PATH variable gets derived from nodemanager's env and nodemanager get's the env from ambari-agent's /var/lib/ambari-agent/ambari-env.sh. . How to fix this? To fix this, add "/bin" and "/sbin" in /var/lib/ambari-agent/ambari-env.sh, restart ambari-agent followed by nodemanager restart. . Note - It may get failed with "ln: command not found" error, please follow the same resolution mentioned above in this case as well. . Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more

Troubleshooting Oozie job is a pain! It kills your time and patience 🙂 . Here are few steps which can save your valuable time: . 1. Always check Oozie launcher's stderr section to see if there is any error. Please find an useful article here to see how to check Oozie launcher logs. . 2. Check stdout logs to see if Oozie has launched any child job which has some error and because of which launcher got failed. Expand the stdout section and search for string "Submitted application" to see what all child jobs got triggered by launcher. . 3. Few situations are complex to troubleshoot. Child job gets completed successfully. There is no error in the stderr section and still your launcher gets failed with "Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]" error. . Sample stdout logs: 2016-12-06 09:03:39,986 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 100% reduce 0% 2016-12-06 09:03:39,991 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1378)) - Job job_XXXXXX_YYYY completed successfully . . . 2016-12-06 09:03:40,228 DEBUG [main] hive.TableDefWriter (TableDefWriter.java:getLoadDataStmt(252)) - Load statement: LOAD DATA INPATH 'hdfs://XXXXXXX' OVERWRITE INTO TABLE `XXXXXX` 65695 [main] INFO org.apache.sqoop.hive.HiveImport - Loading uploaded data into Hive 2016-12-06 09:03:40,229 INFO [main] hive.HiveImport (HiveImport.java:importTable(195)) - Loading uploaded data into Hive . 65711 [main] DEBUG org.apache.sqoop.hive.HiveImport - Using in-process Hive instance. 2016-12-06 09:03:40,245 DEBUG [main] hive.HiveImport (HiveImport.java:executeScript(326)) - Using in-process Hive instance. [Loaded org.apache.sqoop.util.SubprocessSecurityManager from file:/dataXXX/hadoop/yarn/local/filecache/693/sqoop-1.4.6.2.3.4.0-3485.jar] [Loaded org.apache.sqoop.util.ExitSecurityException from file:/dataXXX/hadoop/yarn/local/filecache/693/sqoop-1.4.6.2.3.4.0-3485.jar] [Loaded com.cloudera.sqoop.util.ExitSecurityException from file:/dataXXX/hadoop/yarn/local/filecache/693/sqoop-1.4.6.2.3.4.0-3485.jar] 65714 [main] DEBUG org.apache.sqoop.util.SubprocessSecurityManager - Installing subprocess security manager 2016-12-06 09:03:40,248 DEBUG [main] util.SubprocessSecurityManager (SubprocessSecurityManager.java:install(59)) - Installing subprocess security manager [Loaded org.apache.hadoop.hive.ql.metadata.HiveException from file:/dataXXX/hadoop/yarn/local/filecache/778/hive-exec-1.2.1.2.3.4.0-3485.jar] [Loaded org.apache.hadoop.hive.ql.security.authorization.plugin.HiveMetastoreClientFactory from file:/dataXXX/hadoop/yarn/local/filecache/778/hive-exec-1.2.1.2.3.4.0-3485.jar] . . . [Loaded org.apache.oozie.action.hadoop.JavaMainException from file:/dataXXX/hadoop/yarn/local/filecache/365/oozie-sharelib-oozie-4.2.0.2.3.4.0-3485.jar] [Loaded org.apache.oozie.action.hadoop.LauncherMainException from file:/dataXXX/hadoop/yarn/local/filecache/365/oozie-sharelib-oozie-4.2.0.2.3.4.0-3485.jar] Intercepting System.exit(1) <<< Invocation of Main class completed <<< Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] Oozie Launcher failed, finishing Hadoop job gracefully . How to troubleshoot this? By default, when Yarn application gets finished, nodemanager deletes temporary data from local container directories. In case of above issue, we will have to retain it for some time and check hive.log inside container directory. Below are the detailed steps to do this: 1. Please add below property in yarn-site.xml to retain container directory after application is finished. yarn.nodemanager.delete.debug-delay-sec=1800 ( I have set it for 30 minutes. you can change the value as per your convenience ) 2. Restart required services via Ambari. 3. Rerun the Oozie job. 4. Goto the failed launcher job logs and find the Node manager where launcher was run ( which is failed ) 5. Expand launch container section of the application logs. 6. Find value of PWD 7. Login to the node manager and cd to $PWD ( value obtained in step 6 ) 8. find file with name hive.log inside container's directory e.g. find . -name hive.log 9. hive.log should have actual error which is not visible in application logs. . Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more

This has been tested on Ambari 2.6.2.0 and DLM 1.1.2.0 If there is broken symlink or unwanted directory under /var/lib/ambari-server/resources on Ambari Server, you get below error while installing mpack(management pack) for Beacon service. [root@XXXXXX ~]# ambari-server install-mpack --mpack /root/beacon-ambari-mpack-1.1.2.0-37.tar.gz --verbose Using python /usr/bin/python Installing management pack INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Installing management pack /root/beacon-ambari-mpack-1.1.2.0-37.tar.gz INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Download management pack to temp location /var/lib/ambari-server/data/tmp/beacon-ambari-mpack-1.1.2.0-37.tar.gz INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Expand management pack at temp location /var/lib/ambari-server/data/tmp/beacon-ambari-mpack-1.1.2.0-37/ INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Stage management pack beacon-engine.mpack-1.1.0.0 to staging location /var/lib/ambari-server/resources/mpacks/beacon-engine.mpack-1.1.0.0 INFO: Processing artifact BEACON-common-services of type service-definitions in /var/lib/ambari-server/resources/mpacks/beacon-engine.mpack-1.1.0.0/common-services INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Symlink: /var/lib/ambari-server/resources/common-services/BEACON/1.1.0 INFO: Processing artifact BEACON-addon-services of type stack-addon-service-definitions in /var/lib/ambari-server/resources/mpacks/beacon-engine.mpack-1.1.0.0/addon-services INFO: Loading properties from /etc/ambari-server/conf/ambari.properties Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 952, in <module> mainBody() File "/usr/sbin/ambari-server.py", line 922, in mainBody main(options, args, parser) File "/usr/sbin/ambari-server.py", line 874, in main action_obj.execute() File "/usr/sbin/ambari-server.py", line 78, in execute self.fn(*self.args, **self.kwargs) File "/usr/lib/ambari-server/lib/ambari_server/setupMpacks.py", line 896, in install_mpack (mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode) File "/usr/lib/ambari-server/lib/ambari_server/setupMpacks.py", line 794, in _install_mpack process_stack_addon_service_definitions_artifact(artifact, artifact_source_dir, options) File "/usr/lib/ambari-server/lib/ambari_server/setupMpacks.py", line 554, in process_stack_addon_service_definitions_artifact sudo.symlink(source_service_version_path, dest_link) File "/usr/lib/ambari-server/lib/resource_management/core/sudo.py", line 124, in symlink os.symlink(source, link_name) OSError: [Errno 17] File exists Please follow below steps to fix this: 1. Make sure that there is no backup directory under /var/lib/ambari-server/resources like common-services.backup or stacks.old etc. If exists, please move it some other location. . 2. Delete or move below directories to other location /var/lib/ambari-server/resources/common-services/BEACON /var/lib/ambari-server/resources/mpacks . 3. Check if there is any broken symlink for BEACON under stacks directory. If exists, unlink it. unlink /var/lib/ambari-server/resources/stacks/HDP/2.6/services/BEACON . 4 Reinstall mpack using command mentioned in the Hortonworks docs. e.g. ambari-server install-mpack --mpack /root/beacon-ambari-mpack-1.1.2.0-37.tar.gz --verbose . Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more

Spark job fails with below error when byte code for any particular method grows beyond 64KB spark.sql.codegen.wholeStage is enabled by default for internal optimization in Spark2 which can cause these kind of issues in some corner cases. Below is the detailed stack trace for your reference: org.codehaus.janino.JaninoRuntimeException: Code of method "processNext()V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator" grows beyond 64 KB at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:949) at org.codehaus.janino.CodeContext.write(CodeContext.java:857) at org.codehaus.janino.UnitCompiler.writeShort(UnitCompiler.java:11072) at org.codehaus.janino.UnitCompiler.load(UnitCompiler.java:10744) at org.codehaus.janino.UnitCompiler.load(UnitCompiler.java:10729) at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3824) at org.codehaus.janino.UnitCompiler.access$9100(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$12.visitLocalVariableAccess(UnitCompiler.java:3796) at org.codehaus.janino.UnitCompiler$12.visitLocalVariableAccess(UnitCompiler.java:3762) at org.codehaus.janino.Java$LocalVariableAccess.accept(Java.java:3675) at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563) at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3762) at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3820) [....] Output truncated at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:782) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) . How to fix this? This can be fixed by setting spark.sql.codegen.wholeStage=false in custom spark2-defaults configuration via Ambari and restart required services OR adding --conf spark.sql.codegen.wholeStage=false in spark-shell or spark-submit command. . Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more

Due to conflict in Jackson jar versions, Oozie job with spark2 action(spark action with spark2 sharelib) may get failed with below error: 2018-06-05 16:53:04,567 [Thread-20] INFO org.apache.spark.SparkContext - Created broadcast 0 from showString at NativeMethodAccessorImpl.java:0 Traceback (most recent call last): File "/grid/9/hadoop/yarn/local/usercache/XXXX/appcache/application_1528131553123_0280/container_e81_1528131553123_0280_01_000002/stg_gl_account_classification_master.py", line 9, in <module> gacm.show() File "/grid/9/hadoop/yarn/local/usercache/XXXX/appcache/application_1528131553123_0280/container_e81_1528131553123_0280_01_000002/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 318, in show File "/grid/9/hadoop/yarn/local/usercache/XXXX/appcache/application_1528131553123_0280/container_e81_1528131553123_0280_01_000002/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__ File "/grid/9/hadoop/yarn/local/usercache/XXXX/appcache/application_1528131553123_0280/container_e81_1528131553123_0280_01_000002/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco File "/grid/9/hadoop/yarn/local/usercache/XXXX/appcache/application_1528131553123_0280/container_e81_1528131553123_0280_01_000002/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o35.showString. : java.lang.ExceptionInInitializerError at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:225) at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:308) at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38) at org.apache.spark.sql.Dataset$anonfun$org$apache$spark$sql$Dataset$execute$1$1.apply(Dataset.scala:2386) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57) at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2788) at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$execute$1(Dataset.scala:2385) at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$collect(Dataset.scala:2392) at org.apache.spark.sql.Dataset$anonfun$head$1.apply(Dataset.scala:2128) at org.apache.spark.sql.Dataset$anonfun$head$1.apply(Dataset.scala:2127) at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2818) at org.apache.spark.sql.Dataset.head(Dataset.scala:2127) at org.apache.spark.sql.Dataset.take(Dataset.scala:2342) at org.apache.spark.sql.Dataset.showString(Dataset.scala:248) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745) Caused by: com.fasterxml.jackson.databind.JsonMappingException: Jackson version is too old 2.4.4 at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:56) at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:549) at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82) at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala) ... 27 more . Why this error? By default, 'oozie' directory in Oozie sharelib has jackson jars with 2.4.4 version and spark2 sharelib has latest versions of jackson jars. . To fix this error, please follow below steps: Step 1: Move older jackson jars from default oozie sharelib to other directory: hadoop fs -mv /user/oozie/share/lib/lib_<ts>/oozie/jackson*/user/oozie/share/lib/lib_<ts>/oozie.old . Step 2: Update oozie sharelib: oozie admin -oozie http://<oozie-server-hostname>:11000/oozie -sharelibupdate . Please check this article for more details about oozie spark2 action. . Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more