When Oozie launcher(map only mapreduce job) gets scheduled to run on RHEL7 node in a mixed OS environment, it may get failed with below ERROR(stderr section of Oozie launcher logs): Container: container_e1XX_XXXXXXX_0X_00000X on XXXXXX_XXX_XXXXXX LogAggregationType: AGGREGATED =============================================================================================================== LogType:stderr Log Upload Time:Tue XXXXXXXXXXXX LogLength:XX Log Contents: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. /usr/bin/env: bash: No such file or directory Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. . Why it failed? This happens because "/bin" and "/sbin" missing in your $PATH in container launch environment. $PATH variable gets derived from nodemanager's env and nodemanager get's the env from ambari-agent's /var/lib/ambari-agent/ambari-env.sh. . How to fix this? To fix this, add "/bin" and "/sbin" in /var/lib/ambari-agent/ambari-env.sh, restart ambari-agent followed by nodemanager restart. . Note - It may get failed with "ln: command not found" error, please follow the same resolution mentioned above in this case as well. . Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more

Troubleshooting Oozie job is a pain! It kills your time and patience 🙂 . Here are few steps which can save your valuable time: . 1. Always check Oozie launcher's stderr section to see if there is any error. Please find an useful article here to see how to check Oozie launcher logs. . 2. Check stdout logs to see if Oozie has launched any child job which has some error and because of which launcher got failed. Expand the stdout section and search for string "Submitted application" to see what all child jobs got triggered by launcher. . 3. Few situations are complex to troubleshoot. Child job gets completed successfully. There is no error in the stderr section and still your launcher gets failed with "Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]" error. . Sample stdout logs: 2016-12-06 09:03:39,986 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 100% reduce 0% 2016-12-06 09:03:39,991 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1378)) - Job job_XXXXXX_YYYY completed successfully . . . 2016-12-06 09:03:40,228 DEBUG [main] hive.TableDefWriter (TableDefWriter.java:getLoadDataStmt(252)) - Load statement: LOAD DATA INPATH 'hdfs://XXXXXXX' OVERWRITE INTO TABLE `XXXXXX` 65695 [main] INFO org.apache.sqoop.hive.HiveImport - Loading uploaded data into Hive 2016-12-06 09:03:40,229 INFO [main] hive.HiveImport (HiveImport.java:importTable(195)) - Loading uploaded data into Hive . 65711 [main] DEBUG org.apache.sqoop.hive.HiveImport - Using in-process Hive instance. 2016-12-06 09:03:40,245 DEBUG [main] hive.HiveImport (HiveImport.java:executeScript(326)) - Using in-process Hive instance. [Loaded org.apache.sqoop.util.SubprocessSecurityManager from file:/dataXXX/hadoop/yarn/local/filecache/693/sqoop-1.4.6.2.3.4.0-3485.jar] [Loaded org.apache.sqoop.util.ExitSecurityException from file:/dataXXX/hadoop/yarn/local/filecache/693/sqoop-1.4.6.2.3.4.0-3485.jar] [Loaded com.cloudera.sqoop.util.ExitSecurityException from file:/dataXXX/hadoop/yarn/local/filecache/693/sqoop-1.4.6.2.3.4.0-3485.jar] 65714 [main] DEBUG org.apache.sqoop.util.SubprocessSecurityManager - Installing subprocess security manager 2016-12-06 09:03:40,248 DEBUG [main] util.SubprocessSecurityManager (SubprocessSecurityManager.java:install(59)) - Installing subprocess security manager [Loaded org.apache.hadoop.hive.ql.metadata.HiveException from file:/dataXXX/hadoop/yarn/local/filecache/778/hive-exec-1.2.1.2.3.4.0-3485.jar] [Loaded org.apache.hadoop.hive.ql.security.authorization.plugin.HiveMetastoreClientFactory from file:/dataXXX/hadoop/yarn/local/filecache/778/hive-exec-1.2.1.2.3.4.0-3485.jar] . . . [Loaded org.apache.oozie.action.hadoop.JavaMainException from file:/dataXXX/hadoop/yarn/local/filecache/365/oozie-sharelib-oozie-4.2.0.2.3.4.0-3485.jar] [Loaded org.apache.oozie.action.hadoop.LauncherMainException from file:/dataXXX/hadoop/yarn/local/filecache/365/oozie-sharelib-oozie-4.2.0.2.3.4.0-3485.jar] Intercepting System.exit(1) <<< Invocation of Main class completed <<< Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] Oozie Launcher failed, finishing Hadoop job gracefully . How to troubleshoot this? By default, when Yarn application gets finished, nodemanager deletes temporary data from local container directories. In case of above issue, we will have to retain it for some time and check hive.log inside container directory. Below are the detailed steps to do this: 1. Please add below property in yarn-site.xml to retain container directory after application is finished. yarn.nodemanager.delete.debug-delay-sec=1800 ( I have set it for 30 minutes. you can change the value as per your convenience ) 2. Restart required services via Ambari. 3. Rerun the Oozie job. 4. Goto the failed launcher job logs and find the Node manager where launcher was run ( which is failed ) 5. Expand launch container section of the application logs. 6. Find value of PWD 7. Login to the node manager and cd to $PWD ( value obtained in step 6 ) 8. find file with name hive.log inside container's directory e.g. find . -name hive.log 9. hive.log should have actual error which is not visible in application logs. . Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more

This has been tested on Ambari 2.6.2.0 and DLM 1.1.2.0 If there is broken symlink or unwanted directory under /var/lib/ambari-server/resources on Ambari Server, you get below error while installing mpack(management pack) for Beacon service. [root@XXXXXX ~]# ambari-server install-mpack --mpack /root/beacon-ambari-mpack-1.1.2.0-37.tar.gz --verbose Using python /usr/bin/python Installing management pack INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Installing management pack /root/beacon-ambari-mpack-1.1.2.0-37.tar.gz INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Download management pack to temp location /var/lib/ambari-server/data/tmp/beacon-ambari-mpack-1.1.2.0-37.tar.gz INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Expand management pack at temp location /var/lib/ambari-server/data/tmp/beacon-ambari-mpack-1.1.2.0-37/ INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Stage management pack beacon-engine.mpack-1.1.0.0 to staging location /var/lib/ambari-server/resources/mpacks/beacon-engine.mpack-1.1.0.0 INFO: Processing artifact BEACON-common-services of type service-definitions in /var/lib/ambari-server/resources/mpacks/beacon-engine.mpack-1.1.0.0/common-services INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Symlink: /var/lib/ambari-server/resources/common-services/BEACON/1.1.0 INFO: Processing artifact BEACON-addon-services of type stack-addon-service-definitions in /var/lib/ambari-server/resources/mpacks/beacon-engine.mpack-1.1.0.0/addon-services INFO: Loading properties from /etc/ambari-server/conf/ambari.properties Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 952, in <module> mainBody() File "/usr/sbin/ambari-server.py", line 922, in mainBody main(options, args, parser) File "/usr/sbin/ambari-server.py", line 874, in main action_obj.execute() File "/usr/sbin/ambari-server.py", line 78, in execute self.fn(*self.args, **self.kwargs) File "/usr/lib/ambari-server/lib/ambari_server/setupMpacks.py", line 896, in install_mpack (mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode) File "/usr/lib/ambari-server/lib/ambari_server/setupMpacks.py", line 794, in _install_mpack process_stack_addon_service_definitions_artifact(artifact, artifact_source_dir, options) File "/usr/lib/ambari-server/lib/ambari_server/setupMpacks.py", line 554, in process_stack_addon_service_definitions_artifact sudo.symlink(source_service_version_path, dest_link) File "/usr/lib/ambari-server/lib/resource_management/core/sudo.py", line 124, in symlink os.symlink(source, link_name) OSError: [Errno 17] File exists Please follow below steps to fix this: 1. Make sure that there is no backup directory under /var/lib/ambari-server/resources like common-services.backup or stacks.old etc. If exists, please move it some other location. . 2. Delete or move below directories to other location /var/lib/ambari-server/resources/common-services/BEACON /var/lib/ambari-server/resources/mpacks . 3. Check if there is any broken symlink for BEACON under stacks directory. If exists, unlink it. unlink /var/lib/ambari-server/resources/stacks/HDP/2.6/services/BEACON . 4 Reinstall mpack using command mentioned in the Hortonworks docs. e.g. ambari-server install-mpack --mpack /root/beacon-ambari-mpack-1.1.2.0-37.tar.gz --verbose . Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more