Reply
New Contributor
Posts: 3
Registered: ‎10-19-2017

Sqoop with Oozie

Hi,

 

I have installed cloudera 5.12.1 with Hue as notebook. When I tried to execute a (sqoop, shell, even Spark2) command the application just give me this error:

Sqoop Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

Spark2 [workflor] Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [1]

Those are the only output I obteined after problem generated.

The sqoop command works perfect in command line directly from the server but with Hue failed every time.

 

 

-- import --connect jdbc:oracle:thin:@asdf:1522:pro1 --username ssss --password xxxx --query "select  a.MO_NUM, TO_CHAR(a.MO_HOROCUP, 'yyyy-MM-dd HH24:mi:ss'), TO_CHAR(a.MO_HOLIBER, 'yyyy-MM-dd HH24:mi:ss'), a.MO_COD, a.MO_NUM, d.REFER_EST, d.NOMBRE from TDSAFG a , TSASCXA b, TQWDF c, TSDSDS d where a.MO_NUM = '12' and a.MO_COD not like '%*' and a.MO_HOROCUP between TO_DATE('26-10-17 05:30:00', 'DD-MM-YY HH:mi:ss') and TO_DATE('27-10-17 01:00:00', 'DD-MM-YY HH:mi:ss') and b.CA_ABREVIA = a.MO_ABREVIA and b.CA_FECHFIN IS NULL and b.CA_NUM = a.MO_NUM and c.FFSI is null and c.REFER = b.CA_REFERPA and d.REFER_EST = c.REFER_EST and d.FFSI is null AND \$CONDITIONS" --split-by MO_COD --as-textfile --target-dir /user/poc/input/yyyy/20171026 --driver oracle.jdbc.OracleDriver --m 1

With spark is a lilte bit different, because the execution of the job using the Spark console over Hue work for some jobs but failed with the job that I need to run. Error here is that when I execute the job its failed because "Wrong FS"

 

 

java.lang.IllegalArgumentException: Wrong FS: hdfs://hostname.domain.net:8020/user/oozie/share/lib/lib_20171123121217/spark2/spark-yarn_2.11-2.2.0.cloudera1.jar, expected: hdfs://hostname:8020

I'm not where I need to update, but there is one thing, the FNQN it's correct, I can call the server using both references but not with oozi.

With Spark happend the same thing as sqoop command when I tried to execute a job in a workflow.

 

Sqoop/Oozie Error

 

This is the log I can get from YARN, I don't see anything apart of a warning with ojdc6.jar because I tried to force it in case that the jar was the problem but it looks that is not.

 

2017-11-28 15:43:34,202 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1511521415490_0106_000001
2017-11-28 15:43:34,924 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2017-11-28 15:43:34,924 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@6f43c82)
2017-11-28 15:43:35,331 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 10.4.58.61:8032, Ident: (RM_DELEGATION_TOKEN owner=admin, renewer=oozie mr token, realUser=oozie, issueDate=1511880205594, maxDate=1512485005594, sequenceNumber=200, masterKeyId=6)
2017-11-28 15:43:35,369 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter
2017-11-28 15:43:35,372 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter
2017-11-28 15:43:36,527 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-11-28 15:43:37,047 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2017-11-28 15:43:37,048 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2017-11-28 15:43:37,049 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2017-11-28 15:43:37,050 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2017-11-28 15:43:37,050 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2017-11-28 15:43:37,062 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2017-11-28 15:43:37,062 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2017-11-28 15:43:37,063 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2017-11-28 15:43:37,213 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://tier1.domain.net:8020]
2017-11-28 15:43:37,300 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://tier1.domain.net:8020]
2017-11-28 15:43:37,364 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://tier1.domain.net:8020]
2017-11-28 15:43:37,378 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2017-11-28 15:43:37,479 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2017-11-28 15:43:38,016 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2017-11-28 15:43:38,256 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2017-11-28 15:43:38,256 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2017-11-28 15:43:38,265 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1511521415490_0106 to jobTokenSecretManager
2017-11-28 15:43:38,576 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1511521415490_0106 because: not enabled;
2017-11-28 15:43:38,611 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1511521415490_0106 = 0. Number of splits = 1
2017-11-28 15:43:38,611 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1511521415490_0106 = 0
2017-11-28 15:43:38,611 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1511521415490_0106Job Transitioned from NEW to INITED
2017-11-28 15:43:38,632 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1511521415490_0106.
2017-11-28 15:43:38,666 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
2017-11-28 15:43:38,686 INFO [Socket Reader #1 for port 33130] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 33130
2017-11-28 15:43:38,756 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2017-11-28 15:43:38,802 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at tier2.domain.net/10.4.58.71:33130
2017-11-28 15:43:38,779 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2017-11-28 15:43:38,791 INFO [IPC Server listener on 33130] org.apache.hadoop.ipc.Server: IPC Server listener on 33130: starting
2017-11-28 15:43:38,891 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2017-11-28 15:43:38,923 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2017-11-28 15:43:38,938 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2017-11-28 15:43:38,966 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2017-11-28 15:43:38,970 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2017-11-28 15:43:38,970 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2017-11-28 15:43:38,982 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2017-11-28 15:43:38,983 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2017-11-28 15:43:38,990 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 33459
2017-11-28 15:43:38,990 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2017-11-28 15:43:39,039 INFO [main] org.mortbay.log: Extract jar:file:/data/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/jars/hadoop-yarn-common-2.6.0-cdh5.12.1.jar!/webapps/mapreduce to ./tmp/Jetty_0_0_0_0_33459_mapreduce____4nxhkx/webapp
2017-11-28 15:43:39,994 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:33459
2017-11-28 15:43:39,995 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 33459
2017-11-28 15:43:40,658 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2017-11-28 15:43:40,731 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
2017-11-28 15:43:40,755 INFO [Socket Reader #1 for port 45777] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 45777
2017-11-28 15:43:40,851 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2017-11-28 15:43:40,888 INFO [IPC Server listener on 45777] org.apache.hadoop.ipc.Server: IPC Server listener on 45777: starting
2017-11-28 15:43:41,554 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2017-11-28 15:43:41,554 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2017-11-28 15:43:41,554 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2017-11-28 15:43:41,714 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at tier1.domain.net/10.4.58.61:8030
2017-11-28 15:43:41,890 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:8192, vCores:2>
2017-11-28 15:43:41,890 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.users.admin
2017-11-28 15:43:41,907 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2017-11-28 15:43:41,907 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2017-11-28 15:43:41,954 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1511521415490_0106Job Transitioned from INITED to SETUP
2017-11-28 15:43:42,012 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2017-11-28 15:43:42,044 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1511521415490_0106Job Transitioned from SETUP to RUNNING
2017-11-28 15:43:42,149 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1511521415490_0106_m_000000 Task Transitioned from NEW to SCHEDULED
2017-11-28 15:43:42,152 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1511521415490_0106_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-11-28 15:43:42,177 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>
2017-11-28 15:43:42,184 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1511521415490_0106, File: hdfs://tier1.domain.net:8020/user/admin/.staging/job_1511521415490_0106/job_1511521415490_0106_1.jhist
2017-11-28 15:43:42,920 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2017-11-28 15:43:42,947 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://tier1.domain.net:8020]
2017-11-28 15:43:42,984 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1511521415490_0106: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:7168, vCores:7> knownNMs=1
2017-11-28 15:43:44,044 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2017-11-28 15:43:44,298 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1511521415490_0106_01_000002 to attempt_1511521415490_0106_m_000000_0
2017-11-28 15:43:44,300 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-11-28 15:43:44,419 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Job jar is not present. Not adding any jar to the list of resources.
2017-11-28 15:43:44,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/admin/.staging/job_1511521415490_0106/job.xml
2017-11-28 15:43:44,777 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://tier1.domain.net:8020/user/hue/oozie/workspaces/hue-oozie-1511872125.9/lib/ojdbc6-11.2.0.3.0.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://tier1.domain.net:8020/user/oozie/share/lib/lib_20171123121217/sqoop/ojdbc6-11.2.0.3.0.jar This will be an error in Hadoop 2.0
2017-11-28 15:43:45,180 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #1 tokens and #1 secret keys for NM use for launching container
2017-11-28 15:43:45,180 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 2
2017-11-28 15:43:45,180 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
2017-11-28 15:43:46,764 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
2017-11-28 15:43:46,767 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1511521415490_0106_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2017-11-28 15:43:46,769 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1511521415490_0106: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:6144, vCores:6> knownNMs=1
2017-11-28 15:43:46,779 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1511521415490_0106_01_000002 taskAttempt attempt_1511521415490_0106_m_000000_0
2017-11-28 15:43:46,781 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1511521415490_0106_m_000000_0
2017-11-28 15:43:46,941 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1511521415490_0106_m_000000_0 : 13562
2017-11-28 15:43:46,942 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1511521415490_0106_m_000000_0] using containerId: [container_1511521415490_0106_01_000002 on NM: [tier2.domain.net:8041]
2017-11-28 15:43:46,945 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1511521415490_0106_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2017-11-28 15:43:46,945 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1511521415490_0106_m_000000 Task Transitioned from SCHEDULED to RUNNING
2017-11-28 15:43:52,527 INFO [Socket Reader #1 for port 45777] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1511521415490_0106 (auth:SIMPLE)
2017-11-28 15:43:52,612 INFO [IPC Server handler 0 on 45777] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1511521415490_0106_m_000002 asked for a task
2017-11-28 15:43:52,617 INFO [IPC Server handler 0 on 45777] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1511521415490_0106_m_000002 given task: attempt_1511521415490_0106_m_000000_0
2017-11-28 15:43:57,486 INFO [IPC Server handler 25 on 45777] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1511521415490_0106_m_000000_0 is : 0.0
2017-11-28 15:43:57,718 INFO [IPC Server handler 1 on 45777] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1511521415490_0106_m_000000_0 is : 1.0
2017-11-28 15:43:57,791 INFO [IPC Server handler 2 on 45777] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1511521415490_0106_m_000000_0
2017-11-28 15:43:57,814 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1511521415490_0106_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCESS_FINISHING_CONTAINER
2017-11-28 15:43:57,820 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1511521415490_0106_m_000000_0
2017-11-28 15:43:57,821 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1511521415490_0106_m_000000 Task Transitioned from RUNNING to SUCCEEDED
2017-11-28 15:43:57,823 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
2017-11-28 15:43:57,823 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1511521415490_0106Job Transitioned from RUNNING to COMMITTING
2017-11-28 15:43:57,895 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-11-28 15:43:57,904 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT
2017-11-28 15:43:58,183 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent 
2017-11-28 15:43:58,184 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1511521415490_0106Job Transitioned from COMMITTING to SUCCEEDED
2017-11-28 15:43:58,229 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2017-11-28 15:43:58,229 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2017-11-28 15:43:58,229 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
2017-11-28 15:43:58,229 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2017-11-28 15:43:58,229 INFO [Thread-69] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2017-11-28 15:43:58,229 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2017-11-28 15:43:58,276 INFO [Thread-69] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
2017-11-28 15:43:58,422 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://tier1.domain.net:8020/user/admin/.staging/job_1511521415490_0106/job_1511521415490_0106_1.jhist to hdfs://tier1.domain.net:8020/user/history/done_intermediate/admin/job_1511521415490_0106-1511880207048-admin-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DMy+Workflow%3AA%3Dsqoop%2De4f1%3A-1511880238182-1-0-SUCCEEDED-root.users.admin-1511880221943.jhist_tmp
2017-11-28 15:43:58,656 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://tier1.domain.net:8020/user/history/done_intermediate/admin/job_1511521415490_0106-1511880207048-admin-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DMy+Workflow%3AA%3Dsqoop%2De4f1%3A-1511880238182-1-0-SUCCEEDED-root.users.admin-1511880221943.jhist_tmp
2017-11-28 15:43:58,673 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://tier1.domain.net:8020/user/admin/.staging/job_1511521415490_0106/job_1511521415490_0106_1_conf.xml to hdfs://tier1.domain.net:8020/user/history/done_intermediate/admin/job_1511521415490_0106_conf.xml_tmp
2017-11-28 15:43:58,840 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://tier1.domain.net:8020/user/history/done_intermediate/admin/job_1511521415490_0106_conf.xml_tmp
2017-11-28 15:43:58,862 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://tier1.domain.net:8020/user/history/done_intermediate/admin/job_1511521415490_0106.summary_tmp to hdfs://tier1.domain.net:8020/user/history/done_intermediate/admin/job_1511521415490_0106.summary
2017-11-28 15:43:58,873 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://tier1.domain.net:8020/user/history/done_intermediate/admin/job_1511521415490_0106_conf.xml_tmp to hdfs://tier1.domain.net:8020/user/history/done_intermediate/admin/job_1511521415490_0106_conf.xml
2017-11-28 15:43:58,884 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://tier1.domain.net:8020/user/history/done_intermediate/admin/job_1511521415490_0106-1511880207048-admin-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DMy+Workflow%3AA%3Dsqoop%2De4f1%3A-1511880238182-1-0-SUCCEEDED-root.users.admin-1511880221943.jhist_tmp to hdfs://tier1.domain.net:8020/user/history/done_intermediate/admin/job_1511521415490_0106-1511880207048-admin-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DMy+Workflow%3AA%3Dsqoop%2De4f1%3A-1511880238182-1-0-SUCCEEDED-root.users.admin-1511880221943.jhist
2017-11-28 15:43:58,885 INFO [Thread-69] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2017-11-28 15:43:58,886 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1511521415490_0106_m_000000_0
2017-11-28 15:43:58,907 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1511521415490_0106_01_000002
2017-11-28 15:43:58,920 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-11-28 15:43:58,920 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1511521415490_0106_m_000000_0: 
2017-11-28 15:43:58,920 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1511521415490_0106_m_000000_0 TaskAttempt Transitioned from SUCCESS_FINISHING_CONTAINER to SUCCEEDED
2017-11-28 15:43:58,950 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1511521415490_0106_01_000002 taskAttempt attempt_1511521415490_0106_m_000000_0
2017-11-28 15:43:59,004 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to 
2017-11-28 15:43:59,004 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://tier1.domain.net:19888/jobhistory/job/job_1511521415490_0106
2017-11-28 15:43:59,020 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
2017-11-28 15:44:00,040 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-11-28 15:44:00,041 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://tier1.domain.net:8020 /user/admin/.staging/job_1511521415490_0106
2017-11-28 15:44:00,052 INFO [Thread-69] org.apache.hadoop.ipc.Server: Stopping server on 45777
2017-11-28 15:44:00,079 INFO [IPC Server listener on 45777] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 45777
2017-11-28 15:44:00,079 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2017-11-28 15:44:00,100 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
2017-11-28 15:44:00,101 INFO [Thread-69] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job end notification started for jobID : job_1511521415490_0106
2017-11-28 15:44:00,101 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2017-11-28 15:44:00,113 INFO [Thread-69] org.mortbay.log: Job end notification attempts left 0
2017-11-28 15:44:00,113 INFO [Thread-69] org.mortbay.log: Job end notification trying http://tier1.domain.net:11000/oozie/callback?id=0000002-171128153321817-oozie-oozi-W@sqoop-e4f1&status=SUCCEEDED
2017-11-28 15:44:00,184 INFO [Thread-69] org.mortbay.log: Job end notification to http://tier1.domain.net:11000/oozie/callback?id=0000002-171128153321817-oozie-oozi-W@sqoop-e4f1&status=SUCCEEDED succeeded
2017-11-28 15:44:00,185 INFO [Thread-69] org.mortbay.log: Job end notification succeeded for job_1511521415490_0106
2017-11-28 15:44:05,199 INFO [Thread-69] org.apache.hadoop.ipc.Server: Stopping server on 33130
2017-11-28 15:44:05,206 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2017-11-28 15:44:05,217 INFO [IPC Server listener on 33130] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 33130
2017-11-28 15:44:05,248 INFO [Thread-69] org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0

Any idea?

Announcements