Support Questions
Find answers, ask questions, and share your expertise

ERROR / ISSUE WHILE RUNNING IMPALA QUERY IN SHELL SCRIPT FROM OOZIE ACTION

ERROR / ISSUE WHILE RUNNING IMPALA QUERY IN SHELL SCRIPT FROM OOZIE ACTION

Contributor

ISSUE:

After upgrading from cdh 5.4.2 to cdh 5.5.1 our most of the oozie workflow actions started failing one among them is to run a simple shell script which will invalidate metadata,refresh,compute stats tables from impala through oozie action, that used to work perfectly in cdh 5.4.2.

But now we are facing this issue as our databases are not getting invalidated through impala, because oozie shell action to do this is failing.This is creating a huge problem for our front-end queries. We are currently running "invalidate metadata" manually through impala-shell.


We tried running empty oozie shell action multiple times without impala queries which works fine without errors. But as we include an impala query in the shell script, the oozie shell action fails
in most instances and works in few instances.
The configuration and settings are similar to our previous distribution of CDH. There is no change in servers or roles and configs.

Also have a complete fresh install of CDH 5.5.1 on other cluster with similar configuration, same oozie shell action with shell script consisting of impala queries and it works perfectly fine, so the only difference between two clusters is one is upgraded and the other is fresh install and setup of CDH 5.5.1 with same configurations.

 

Another thing we noticed it started working fine for a week and again we see the same issue.

 

Please help ASAP!!!!

 

So, the issue is with upgraded cluster.

Following are the contains of shell script:

#!/bin/sh


if [ ! -d "/tmp/ooziePythonEgg" ]; then
mkdir /tmp/ooziePythonEgg
fi

export PYTHON_EGG_CACHE=/tmp/ooziePythonEgg


impala-shell -q "

SET QUERY_TIMEOUT_S=500 ;

use database;

REFRESH accnt_status_dmn;
REFRESH campaign_dmn;
REFRESH cntct_dmn;
REFRESH currency_dmn;
REFRESH day_dmn;
REFRESH employee_dmn;
REFRESH hr_dmn;
REFRESH lctn_dmn;
REFRESH lead_dmn;


COMPUTE STATS accnt_status_dmn;
COMPUTE STATS campaign_dmn;
COMPUTE STATS cntct_dmn;
COMPUTE STATS currency_dmn;
COMPUTE STATS day_dmn;
COMPUTE STATS employee_dmn;
COMPUTE STATS hr_dmn;
COMPUTE STATS lctn_dmn;
COMPUTE STATS lead_dmn;


"


Following is the oozie shell action:

<workflow-app name="Refresh-metadata" xmlns="uri:oozie:workflow:0.4">
<global>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>oozie.launcher.mapred.child.java.opts</name>
<value>-Xmx5124M</value>
</property>
</configuration>
</global>

<start to="Refresh_metadata"/>

<action name="Refresh_metadata">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>${hiveSiteDir}</job-xml>
<exec>${nameNode}/user/xyz/workflow_invalidate/invalidate.sh</exec>
<file>${nameNode}/user/xyz/workflow_invalidate/invalidate.sh#invalidate.sh</file>
<file>${hiveSiteDir}#hive-oozie-site.xml</file>
<capture-output/>
</shell>
<ok to="workflowEnd"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="workflowEnd"/>

</workflow-app>


Please see the following error message and logs for that action:


ERROR CODE: JA018

ERROR message: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]


syslog:


Log Upload Time: Wed Jan 27 18:24:09 +0000 2016

Log Length: 25569
2016-01-27 18:23:43,137 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1453585092595_50327_000001
2016-01-27 18:23:43,536 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2016-01-27 18:23:43,536 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@323c1a28)
2016-01-27 18:23:43,566 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 10.4.2.102:8032, Ident: (owner=cdhadmin, renewer=oozie mr token, realUser=oozie, issueDate=1453919021929, maxDate=1454523821929, sequenceNumber=69240, masterKeyId=5)
2016-01-27 18:23:44,129 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-01-27 18:23:44,252 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2016-01-27 18:23:44,253 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2016-01-27 18:23:44,294 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2016-01-27 18:23:44,295 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2016-01-27 18:23:44,296 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2016-01-27 18:23:44,297 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2016-01-27 18:23:44,297 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2016-01-27 18:23:44,298 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2016-01-27 18:23:44,299 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2016-01-27 18:23:44,299 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2016-01-27 18:23:44,341 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://site-xxxx.com:8020]
2016-01-27 18:23:44,362 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://site-xxxx.com:8020]
2016-01-27 18:23:44,413 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://site-xxxx.ccmteam.com:8020]
2016-01-27 18:23:44,425 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2016-01-27 18:23:44,467 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2016-01-27 18:23:44,670 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2016-01-27 18:23:44,728 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2016-01-27 18:23:44,728 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2016-01-27 18:23:44,740 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1453585092595_50327 to jobTokenSecretManager
2016-01-27 18:23:44,851 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1453585092595_50327 because: not enabled; too much RAM;
2016-01-27 18:23:44,873 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1453585092595_50327 = 0. Number of splits = 1
2016-01-27 18:23:44,874 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1453585092595_50327 = 0
2016-01-27 18:23:44,874 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1453585092595_50327Job Transitioned from NEW to INITED
2016-01-27 18:23:44,875 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1453585092595_50327.
2016-01-27 18:23:44,904 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2016-01-27 18:23:44,913 INFO [Socket Reader #1 for port 42899] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 42899
2016-01-27 18:23:44,932 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2016-01-27 18:23:44,932 INFO [IPC Server listener on 42899] org.apache.hadoop.ipc.Server: IPC Server listener on 42899: starting
2016-01-27 18:23:44,932 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2016-01-27 18:23:44,933 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at site-xxxxxx
2016-01-27 18:23:44,991 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-01-27 18:23:45,005 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2016-01-27 18:23:45,012 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2016-01-27 18:23:45,021 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2016-01-27 18:23:45,026 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2016-01-27 18:23:45,026 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2016-01-27 18:23:45,029 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2016-01-27 18:23:45,029 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2016-01-27 18:23:45,037 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 37785
2016-01-27 18:23:45,037 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2016-01-27 18:23:45,065 INFO [main] org.mortbay.log: Extract jar:file:/vol1/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/hadoop-yarn-common-2.6.0-cdh5.5.1.jar!/webapps/mapreduce to ./tmp/Jetty_0_0_0_0_37785_mapreduce____z1e0q1/webapp
2016-01-27 18:23:45,372 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:37785
2016-01-27 18:23:45,372 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 37785
2016-01-27 18:23:45,673 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2016-01-27 18:23:45,678 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2016-01-27 18:23:45,679 INFO [Socket Reader #1 for port 45305] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 45305
2016-01-27 18:23:45,684 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2016-01-27 18:23:45,685 INFO [IPC Server listener on 45305] org.apache.hadoop.ipc.Server: IPC Server listener on 45305: starting
2016-01-27 18:23:45,706 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2016-01-27 18:23:45,706 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2016-01-27 18:23:45,706 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2016-01-27 18:23:45,796 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at site-xxxxx
2016-01-27 18:23:45,863 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:8192, vCores:14>
2016-01-27 18:23:45,863 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.cdhadmin
2016-01-27 18:23:45,867 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2016-01-27 18:23:45,868 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
2016-01-27 18:23:45,874 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1453585092595_50327Job Transitioned from INITED to SETUP
2016-01-27 18:23:45,876 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2016-01-27 18:23:45,881 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2016-01-27 18:23:45,890 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1453585092595_50327Job Transitioned from SETUP to RUNNING
2016-01-27 18:23:45,922 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1453585092595_50327_m_000000 Task Transitioned from NEW to SCHEDULED
2016-01-27 18:23:45,923 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1453585092595_50327_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2016-01-27 18:23:45,924 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:4096, vCores:1>
2016-01-27 18:23:45,946 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1453585092595_50327, File: hdfs://site-xxxxx.com:8020/user/cdhadmin/.staging/job_1453585092595_50327/job_1453585092595_50327_1.jhist
2016-01-27 18:23:46,218 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://site-xxxx.com:8020]
2016-01-27 18:23:46,866 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2016-01-27 18:23:46,902 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1453585092595_50327: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:191488, vCores:88> knownNMs=7
2016-01-27 18:23:47,920 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2016-01-27 18:23:47,948 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1453585092595_50327_01_000002 to attempt_1453585092595_50327_m_000000_0
2016-01-27 18:23:47,951 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2016-01-27 18:23:47,990 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Job jar is not present. Not adding any jar to the list of resources.
2016-01-27 18:23:48,002 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/cdhadmin/.staging/job_1453585092595_50327/job.xml
2016-01-27 18:23:48,019 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #1 tokens and #1 secret keys for NM use for launching container
2016-01-27 18:23:48,019 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 2
2016-01-27 18:23:48,019 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
2016-01-27 18:23:48,065 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1453585092595_50327_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2016-01-27 18:23:48,071 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1453585092595_50327_01_000002 taskAttempt attempt_1453585092595_50327_m_000000_0
2016-01-27 18:23:48,076 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1453585092595_50327_m_000000_0
2016-01-27 18:23:48,078 INFO [ContainerLauncher #0] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : site-xxxxxx:8041
2016-01-27 18:23:48,130 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1453585092595_50327_m_000000_0 : 13562
2016-01-27 18:23:48,133 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1453585092595_50327_m_000000_0] using containerId: [container_1453585092595_50327_01_000002 on NM: [site-xxxxx:8041]
2016-01-27 18:23:48,137 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1453585092595_50327_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2016-01-27 18:23:48,137 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1453585092595_50327_m_000000 Task Transitioned from SCHEDULED to RUNNING
2016-01-27 18:23:48,954 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1453585092595_50327: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:191488, vCores:88> knownNMs=7
2016-01-27 18:23:49,553 INFO [Socket Reader #1 for port 45305] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1453585092595_50327 (auth:SIMPLE)
2016-01-27 18:23:49,576 INFO [IPC Server handler 0 on 45305] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1453585092595_50327_m_000002 asked for a task
2016-01-27 18:23:49,577 INFO [IPC Server handler 0 on 45305] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1453585092595_50327_m_000002 given task: attempt_1453585092595_50327_m_000000_0
2016-01-27 18:23:56,678 INFO [IPC Server handler 2 on 45305] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1453585092595_50327_m_000000_0 is : 1.0
2016-01-27 18:24:01,668 INFO [IPC Server handler 3 on 45305] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1453585092595_50327_m_000000_0 is : 1.0
2016-01-27 18:24:01,838 INFO [IPC Server handler 3 on 45305] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1453585092595_50327_m_000000_0
2016-01-27 18:24:01,839 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1453585092595_50327_m_000000_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING
2016-01-27 18:24:01,839 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1453585092595_50327_m_000000_0 given a go for committing the task output.
2016-01-27 18:24:01,841 INFO [IPC Server handler 1 on 45305] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1453585092595_50327_m_000000_0
2016-01-27 18:24:01,842 INFO [IPC Server handler 1 on 45305] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1453585092595_50327_m_000000_0:true
2016-01-27 18:24:02,010 INFO [IPC Server handler 8 on 45305] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1453585092595_50327_m_000000_0 is : 1.0
2016-01-27 18:24:02,014 INFO [IPC Server handler 10 on 45305] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1453585092595_50327_m_000000_0
2016-01-27 18:24:02,017 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1453585092595_50327_m_000000_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_FINISHING_CONTAINER
2016-01-27 18:24:02,031 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1453585092595_50327_m_000000_0
2016-01-27 18:24:02,032 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1453585092595_50327_m_000000 Task Transitioned from RUNNING to SUCCEEDED
2016-01-27 18:24:02,033 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
2016-01-27 18:24:02,034 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1453585092595_50327Job Transitioned from RUNNING to COMMITTING
2016-01-27 18:24:02,035 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT
2016-01-27 18:24:02,101 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent
2016-01-27 18:24:02,102 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1453585092595_50327Job Transitioned from COMMITTING to SUCCEEDED
2016-01-27 18:24:02,104 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2016-01-27 18:24:02,104 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2016-01-27 18:24:02,104 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
2016-01-27 18:24:02,104 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2016-01-27 18:24:02,104 INFO [Thread-69] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2016-01-27 18:24:02,104 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2016-01-27 18:24:02,106 INFO [Thread-69] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
2016-01-27 18:24:02,179 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://site-cdh02.dev.slc1.ccmteam.com:8020/user/cdhadmin/.staging/job_1453585092595_50327/job_1453585092595_50327_1.jhist to hdfs://site-xxxxx.com:8020/user/history/done_intermediate/cdhadmin/job_1453585092595_50327-1453919022079-cdhadmin-oozie%3Alauncher%3AT%3Dshell%3AW%3D15002%2Dudm%2Dto%2Dprime%2Dincrem-1453919042098-1-0-SUCCEEDED-root.cdhadmin-1453919025870.jhist_tmp
2016-01-27 18:24:02,214 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://site-xxxxx.com:8020/user/history/done_intermediate/cdhadmin/job_1453585092595_50327-1453919022079-cdhadmin-oozie%3Alauncher%3AT%3Dshell%3AW%3D15002%2Dudm%2Dto%2Dprime%2Dincrem-1453919042098-1-0-SUCCEEDED-root.cdhadmin-1453919025870.jhist_tmp
2016-01-27 18:24:02,217 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://site-xxxxx.com:8020/user/cdhadmin/.staging/job_1453585092595_50327/job_1453585092595_50327_1_conf.xml to hdfs://site-xxxx.com:8020/user/history/done_intermediate/cdhadmin/job_1453585092595_50327_conf.xml_tmp
2016-01-27 18:24:02,249 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://site-xxxx:8020/user/history/done_intermediate/cdhadmin/job_1453585092595_50327_conf.xml_tmp
2016-01-27 18:24:02,253 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://site-xxxx:8020/user/history/done_intermediate/cdhadmin/job_1453585092595_50327.summary_tmp to hdfs://site-xxxxx:8020/user/history/done_intermediate/cdhadmin/job_1453585092595_50327.summary
2016-01-27 18:24:02,255 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://site-xxxxx:8020/user/history/done_intermediate/cdhadmin/job_1453585092595_50327_conf.xml_tmp to hdfs://site-xxxx:8020/user/history/done_intermediate/cdhadmin/job_1453585092595_50327_conf.xml
2016-01-27 18:24:02,257 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://site-xxxxx:8020/user/history/done_intermediate/cdhadmin/job_1453585092595_50327-1453919022079-cdhadmin-oozie%3Alauncher%3AT%3Dshell%3AW%3D15002%2Dudm%2Dto%2Dprime%2Dincrem-1453919042098-1-0-SUCCEEDED-root.cdhadmin-1453919025870.jhist_tmp to hdfs://site-xxxx.com:8020/user/history/done_intermediate/cdhadmin/job_1453585092595_50327-1453919022079-cdhadmin-oozie%3Alauncher%3AT%3Dshell%3AW%3D15002%2Dudm%2Dto%2Dprime%2Dincrem-1453919042098-1-0-SUCCEEDED-root.cdhadmin-1453919025870.jhist
2016-01-27 18:24:02,258 INFO [Thread-69] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2016-01-27 18:24:02,259 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1453585092595_50327_m_000000_0
2016-01-27 18:24:02,259 INFO [Thread-69] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : site-xxxxx.com:8041
2016-01-27 18:24:02,275 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1453585092595_50327_m_000000_0 TaskAttempt Transitioned from SUCCESS_FINISHING_CONTAINER to SUCCEEDED
2016-01-27 18:24:02,276 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to
2016-01-27 18:24:02,276 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://xxxxx.com:19888/jobhistory/job/job_1453585092595_50327
2016-01-27 18:24:02,281 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
2016-01-27 18:24:03,283 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2016-01-27 18:24:03,285 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://xxxxx.com:8020 /user/cdhadmin/.staging/job_1453585092595_50327
2016-01-27 18:24:03,287 INFO [Thread-69] org.apache.hadoop.ipc.Server: Stopping server on 45305
2016-01-27 18:24:03,331 INFO [IPC Server listener on 45305] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 45305
2016-01-27 18:24:03,332 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2016-01-27 18:24:03,333 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
2016-01-27 18:24:03,333 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder

 


stderr:

SLF4J: Found binding in [jar:file:/vol1/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/vol2/yarn/nm/filecache/12724/OozieNotification-0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Jan 27, 2016 6:23:51 PM com.google.inject.servlet.InternalServletModule$BackwardsCompatibleServletContextProvider get
WARNING: You are attempting to use a deprecated API (specifically, attempting to @Inject ServletContext inside an eagerly created singleton. While we allow this for backwards compatibility, be warned that this MAY have unexpected behavior if you have more than one injector (with ServletModule) running in the same JVM. Please consult the Guice documentation at http://code.google.com/p/google-guice/wiki/Servlets for more information.
Jan 27, 2016 6:23:51 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class
Jan 27, 2016 6:23:51 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
Jan 27, 2016 6:23:51 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class
Jan 27, 2016 6:23:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
Jan 27, 2016 6:23:51 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"
Jan 27, 2016 6:23:52 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
Jan 27, 2016 6:23:52 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest"
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

2 REPLIES 2

Re: ERROR / ISSUE WHILE RUNNING IMPALA QUERY IN SHELL SCRIPT FROM OOZIE ACTION

New Contributor

Please try to add this line in your shell script 

We had a similar problem it solved.

 

export HADOOP_USER_NAME=<USER>

 

thanks

Chenna

Re: ERROR / ISSUE WHILE RUNNING IMPALA QUERY IN SHELL SCRIPT FROM OOZIE ACTION

Contributor

Hi,

 

I tried setting export HADOOP_USER_NAME=${username}, but still it fails with same error as I mentioned on my previous post.