Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created on 01-11-2018 03:07 AM - edited 09-16-2022 05:43 AM
I am tring to schedule a Sqoop action with Oozie to import data from Teradata database, but it fails (the same job completes correctly when using Sqoop in standalone).
This is the job.properties:
nameNode=hdfs://quickstart.cloudera:8020
jobTracker=localhost:8032
oozie.wf.application.path=${nameNode}/user/cloudera/oozie/sqoop-teradata-app
oozie.use.system.libpath=true
This is the workflow.xml:
workflow-app name="OOZIE_SQOOP_WF" xmlns="uri:oozie:workflow:0.4">
<start to="sqoop_action" />
<action name="sqoop_action">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/user/cloudera/sqoop/MyTable"/>
</prepare>
<command>import --connect jdbc:teradata://172.31.7.69/DATABASE=dbname --username admin --password admin --table MyTable --split-by Id --fields-terminated-by '\t' --warehouse-dir sqoop</command>
</sqoop>
<ok to="success"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>JOB FAILED!</message>
</kill>
<end name="success"/>
</workflow-app>
com.cloudera.connector.teradata.TeradataManagerFactory=/var/lib/sqoop/sqoop-connector-teradata-1.6c5.jar
and added the following property configuration in /etc/sqoop/conf.dist/sqoop-site.xml
<property>
<name>sqoop.connection.factories</name>
<value>com.cloudera.connector.teradata.TeradataManagerFactory</value>
</property>
The action fails (FAILED/KILLED1) with error message "Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]". This is the log, where I can't see any error or exception:
2018-01-11 02:35:45,388 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1515662938068_0005_000001 2018-01-11 02:35:46,099 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2018-01-11 02:35:46,099 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@19a3f495) 2018-01-11 02:35:46,748 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 127.0.0.1:8032, Ident: (RM_DELEGATION_TOKEN owner=cloudera, renewer=oozie mr token, realUser=oozie, issueDate=1515666939966, maxDate=1516271739966, sequenceNumber=12, masterKeyId=2) 2018-01-11 02:35:46,800 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter 2018-01-11 02:35:46,804 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter 2018-01-11 02:35:48,037 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2018-01-11 02:35:48,040 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 2018-01-11 02:35:48,042 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 2018-01-11 02:35:48,044 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher 2018-01-11 02:35:48,045 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2018-01-11 02:35:48,050 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 2018-01-11 02:35:48,051 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 2018-01-11 02:35:48,054 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2018-01-11 02:35:48,147 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020] 2018-01-11 02:35:48,214 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020] 2018-01-11 02:35:48,256 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020] 2018-01-11 02:35:48,277 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled 2018-01-11 02:35:48,371 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler 2018-01-11 02:35:49,013 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties 2018-01-11 02:35:49,152 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2018-01-11 02:35:49,152 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started 2018-01-11 02:35:49,175 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1515662938068_0005 to jobTokenSecretManager 2018-01-11 02:35:49,477 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1515662938068_0005 because: not enabled; 2018-01-11 02:35:49,522 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1515662938068_0005 = 0. Number of splits = 1 2018-01-11 02:35:49,522 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1515662938068_0005 = 0 2018-01-11 02:35:49,522 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1515662938068_0005Job Transitioned from NEW to INITED 2018-01-11 02:35:49,524 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1515662938068_0005. 2018-01-11 02:35:49,582 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 2018-01-11 02:35:49,602 INFO [Socket Reader #1 for port 55025] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 55025 2018-01-11 02:35:49,703 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server 2018-01-11 02:35:49,704 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-01-11 02:35:49,707 INFO [IPC Server listener on 55025] org.apache.hadoop.ipc.Server: IPC Server listener on 55025: starting 2018-01-11 02:35:49,713 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at quickstart.cloudera/127.0.0.1:55025 2018-01-11 02:35:49,886 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-01-11 02:35:49,960 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2018-01-11 02:35:49,968 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined 2018-01-11 02:35:49,995 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2018-01-11 02:35:50,007 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce 2018-01-11 02:35:50,007 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2018-01-11 02:35:50,066 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/* 2018-01-11 02:35:50,066 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2018-01-11 02:35:50,085 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 52908 2018-01-11 02:35:50,085 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4 2018-01-11 02:35:50,136 INFO [main] org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.12.0.jar!/webapps/mapreduce to ./tmp/Jetty_0_0_0_0_52908_mapreduce____3vp14l/webapp 2018-01-11 02:35:50,569 INFO [main] org.mortbay.log: Started [email protected]:52908 2018-01-11 02:35:50,569 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 52908 2018-01-11 02:35:51,415 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2018-01-11 02:35:51,423 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 2018-01-11 02:35:51,424 INFO [Socket Reader #1 for port 45459] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 45459 2018-01-11 02:35:51,440 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-01-11 02:35:51,441 INFO [IPC Server listener on 45459] org.apache.hadoop.ipc.Server: IPC Server listener on 45459: starting 2018-01-11 02:35:51,492 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 2018-01-11 02:35:51,492 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 2018-01-11 02:35:51,492 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2018-01-11 02:35:51,557 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030 2018-01-11 02:35:51,702 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:8192, vCores:4> 2018-01-11 02:35:51,702 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.cloudera 2018-01-11 02:35:51,707 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500 2018-01-11 02:35:51,707 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10 2018-01-11 02:35:51,727 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1515662938068_0005Job Transitioned from INITED to SETUP 2018-01-11 02:35:51,736 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP 2018-01-11 02:35:51,738 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1515662938068_0005Job Transitioned from SETUP to RUNNING 2018-01-11 02:35:51,805 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1515662938068_0005_m_000000 Task Transitioned from NEW to SCHEDULED 2018-01-11 02:35:51,811 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1515662938068_0005_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-01-11 02:35:51,813 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1> 2018-01-11 02:35:51,867 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1515662938068_0005, File: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/cloudera/.staging/job_1515662938068_0005/job_1515662938068_0005_1.jhist 2018-01-11 02:35:52,551 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020] 2018-01-11 02:35:52,705 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2018-01-11 02:35:52,776 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1515662938068_0005: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:6144, vCores:7> knownNMs=1 2018-01-11 02:35:54,798 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2018-01-11 02:35:54,802 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1515662938068_0005_01_000002 to attempt_1515662938068_0005_m_000000_0 2018-01-11 02:35:54,803 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0 2018-01-11 02:35:54,886 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Job jar is not present. Not adding any jar to the list of resources. 2018-01-11 02:35:54,913 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /tmp/hadoop-yarn/staging/cloudera/.staging/job_1515662938068_0005/job.xml 2018-01-11 02:35:55,137 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #1 tokens and #1 secret keys for NM use for launching container 2018-01-11 02:35:55,137 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 2 2018-01-11 02:35:55,137 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData 2018-01-11 02:35:55,648 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 2018-01-11 02:35:55,672 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1515662938068_0005_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2018-01-11 02:35:55,682 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1515662938068_0005_01_000002 taskAttempt attempt_1515662938068_0005_m_000000_0 2018-01-11 02:35:55,690 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1515662938068_0005_m_000000_0 2018-01-11 02:35:55,830 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1515662938068_0005: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:5120, vCores:6> knownNMs=1 2018-01-11 02:35:56,004 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1515662938068_0005_m_000000_0 : 13562 2018-01-11 02:35:56,006 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1515662938068_0005_m_000000_0] using containerId: [container_1515662938068_0005_01_000002 on NM: [quickstart.cloudera:34543] 2018-01-11 02:35:56,011 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1515662938068_0005_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2018-01-11 02:35:56,011 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1515662938068_0005_m_000000 Task Transitioned from SCHEDULED to RUNNING 2018-01-11 02:35:59,461 INFO [Socket Reader #1 for port 45459] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1515662938068_0005 (auth:SIMPLE) 2018-01-11 02:35:59,495 INFO [IPC Server handler 1 on 45459] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1515662938068_0005_m_000002 asked for a task 2018-01-11 02:35:59,496 INFO [IPC Server handler 1 on 45459] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1515662938068_0005_m_000002 given task: attempt_1515662938068_0005_m_000000_0 2018-01-11 02:36:03,205 INFO [IPC Server handler 16 on 45459] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1515662938068_0005_m_000000_0 is : 0.0 2018-01-11 02:36:03,341 INFO [IPC Server handler 17 on 45459] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1515662938068_0005_m_000000_0 is : 1.0 2018-01-11 02:36:03,359 INFO [IPC Server handler 13 on 45459] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1515662938068_0005_m_000000_0 2018-01-11 02:36:03,365 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1515662938068_0005_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCESS_FINISHING_CONTAINER 2018-01-11 02:36:03,402 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1515662938068_0005_m_000000_0 2018-01-11 02:36:03,404 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1515662938068_0005_m_000000 Task Transitioned from RUNNING to SUCCEEDED 2018-01-11 02:36:03,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1 2018-01-11 02:36:03,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1515662938068_0005Job Transitioned from RUNNING to COMMITTING 2018-01-11 02:36:03,411 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT 2018-01-11 02:36:03,443 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent 2018-01-11 02:36:03,444 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1515662938068_0005Job Transitioned from COMMITTING to SUCCEEDED 2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry 2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true 2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true 2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true 2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true 2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services 2018-01-11 02:36:03,454 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0 2018-01-11 02:36:03,581 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/cloudera/.staging/job_1515662938068_0005/job_1515662938068_0005_1.jhist to hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005-1515666941120-cloudera-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DOOZIE_SQOOP_WF%3AA%3Dsqoop_ac-1515666963441-1-0-SUCCEEDED-root.cloudera-1515666951717.jhist_tmp 2018-01-11 02:36:03,677 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005-1515666941120-cloudera-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DOOZIE_SQOOP_WF%3AA%3Dsqoop_ac-1515666963441-1-0-SUCCEEDED-root.cloudera-1515666951717.jhist_tmp 2018-01-11 02:36:03,683 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/cloudera/.staging/job_1515662938068_0005/job_1515662938068_0005_1_conf.xml to hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005_conf.xml_tmp 2018-01-11 02:36:03,722 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005_conf.xml_tmp 2018-01-11 02:36:03,732 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005.summary_tmp to hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005.summary 2018-01-11 02:36:03,735 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005_conf.xml_tmp to hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005_conf.xml 2018-01-11 02:36:03,739 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005-1515666941120-cloudera-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DOOZIE_SQOOP_WF%3AA%3Dsqoop_ac-1515666963441-1-0-SUCCEEDED-root.cloudera-1515666951717.jhist_tmp to hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005-1515666941120-cloudera-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DOOZIE_SQOOP_WF%3AA%3Dsqoop_ac-1515666963441-1-0-SUCCEEDED-root.cloudera-1515666951717.jhist 2018-01-11 02:36:03,742 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 2018-01-11 02:36:03,743 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1515662938068_0005_m_000000_0 2018-01-11 02:36:03,773 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1515662938068_0005_m_000000_0 TaskAttempt Transitioned from SUCCESS_FINISHING_CONTAINER to SUCCEEDED 2018-01-11 02:36:03,782 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to 2018-01-11 02:36:03,783 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://quickstart.cloudera:19888/jobhistory/job/job_1515662938068_0005 2018-01-11 02:36:03,797 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered. 2018-01-11 02:36:04,800 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0 2018-01-11 02:36:04,802 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://quickstart.cloudera:8020 /tmp/hadoop-yarn/staging/cloudera/.staging/job_1515662938068_0005 2018-01-11 02:36:04,813 INFO [Thread-68] org.apache.hadoop.ipc.Server: Stopping server on 45459 2018-01-11 02:36:04,819 INFO [IPC Server listener on 45459] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 45459 2018-01-11 02:36:04,820 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2018-01-11 02:36:04,822 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted 2018-01-11 02:36:04,828 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted 2018-01-11 02:36:04,841 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job end notification started for jobID : job_1515662938068_0005 2018-01-11 02:36:04,843 INFO [Thread-68] org.mortbay.log: Job end notification attempts left 0 2018-01-11 02:36:04,843 INFO [Thread-68] org.mortbay.log: Job end notification trying http://quickstart.cloudera:11000/oozie/callback?id=0000002-180111013033300-oozie-oozi-W@sqoop_action&status=SUCCEEDED 2018-01-11 02:36:04,863 INFO [Thread-68] org.mortbay.log: Job end notification to http://quickstart.cloudera:11000/oozie/callback?id=0000002-180111013033300-oozie-oozi-W@sqoop_action&status=SUCCEEDED succeeded 2018-01-11 02:36:04,863 INFO [Thread-68] org.mortbay.log: Job end notification succeeded for job_1515662938068_0005 2018-01-11 02:36:09,868 INFO [Thread-68] org.apache.hadoop.ipc.Server: Stopping server on 55025 2018-01-11 02:36:09,959 INFO [IPC Server listener on 55025] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 55025 2018-01-11 02:36:10,048 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2018-01-11 02:36:10,317 INFO [Thread-68] org.mortbay.log: Stopped [email protected]:0
What am I doing wrong?
Created 01-26-2018 03:41 AM
Finally solved by adding the connection manager parameter and the Teradata jdbc drivers in the path:
sqoop import \ --connection-manager com.teradata.jdbc.TeraDriver \ --connect jdbc:teradata://host/DATABASE=db \ [...]
The following files should be available to Oozie (I've put them in the application lib folder):
I guess the documentation should be updated, since with the informations provided the import from Teradata works from Sqoop standalone but fails always inside Oozie.
Created 01-26-2018 03:41 AM
Finally solved by adding the connection manager parameter and the Teradata jdbc drivers in the path:
sqoop import \ --connection-manager com.teradata.jdbc.TeraDriver \ --connect jdbc:teradata://host/DATABASE=db \ [...]
The following files should be available to Oozie (I've put them in the application lib folder):
I guess the documentation should be updated, since with the informations provided the import from Teradata works from Sqoop standalone but fails always inside Oozie.