Support Questions

Find answers, ask questions, and share your expertise

Oozie Sqoop actions fails when importing data from TERADATA

avatar
Expert Contributor

I am tring to schedule a Sqoop action with Oozie to import data from Teradata database, but it fails (the same job completes correctly when using Sqoop in standalone).

 

This is the job.properties:

 

nameNode=hdfs://quickstart.cloudera:8020
jobTracker=localhost:8032
oozie.wf.application.path=${nameNode}/user/cloudera/oozie/sqoop-teradata-app
oozie.use.system.libpath=true


This is the workflow.xml:

 

workflow-app name="OOZIE_SQOOP_WF" xmlns="uri:oozie:workflow:0.4">
    
	<start to="sqoop_action" />		

	<action name="sqoop_action">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/cloudera/sqoop/MyTable"/>
            </prepare>
            <command>import --connect jdbc:teradata://172.31.7.69/DATABASE=dbname --username admin --password admin --table MyTable --split-by Id --fields-terminated-by '\t' --warehouse-dir sqoop</command>			
        </sqoop>
        <ok to="success"/>
        <error to="fail"/>
    </action>

	<kill name="fail">
		<message>JOB FAILED!</message>
	</kill>

	<end name="success"/>	
    
</workflow-app>

  

  

com.cloudera.connector.teradata.TeradataManagerFactory=/var/lib/sqoop/sqoop-connector-teradata-1.6c5.jar

 
and added the following property configuration in /etc/sqoop/conf.dist/sqoop-site.xml

 

<property>
    <name>sqoop.connection.factories</name>
    <value>com.cloudera.connector.teradata.TeradataManagerFactory</value>
</property>

  

The action fails (FAILED/KILLED1) with error message "Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]". This is the log, where I can't see any error or exception:

  

2018-01-11 02:35:45,388 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1515662938068_0005_000001
2018-01-11 02:35:46,099 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2018-01-11 02:35:46,099 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@19a3f495)
2018-01-11 02:35:46,748 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 127.0.0.1:8032, Ident: (RM_DELEGATION_TOKEN owner=cloudera, renewer=oozie mr token, realUser=oozie, issueDate=1515666939966, maxDate=1516271739966, sequenceNumber=12, masterKeyId=2)
2018-01-11 02:35:46,800 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter
2018-01-11 02:35:46,804 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter
2018-01-11 02:35:48,037 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2018-01-11 02:35:48,040 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2018-01-11 02:35:48,042 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2018-01-11 02:35:48,044 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2018-01-11 02:35:48,045 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2018-01-11 02:35:48,050 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2018-01-11 02:35:48,051 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2018-01-11 02:35:48,054 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2018-01-11 02:35:48,147 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020]
2018-01-11 02:35:48,214 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020]
2018-01-11 02:35:48,256 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020]
2018-01-11 02:35:48,277 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2018-01-11 02:35:48,371 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2018-01-11 02:35:49,013 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties
2018-01-11 02:35:49,152 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2018-01-11 02:35:49,152 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2018-01-11 02:35:49,175 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1515662938068_0005 to jobTokenSecretManager
2018-01-11 02:35:49,477 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1515662938068_0005 because: not enabled;
2018-01-11 02:35:49,522 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1515662938068_0005 = 0. Number of splits = 1
2018-01-11 02:35:49,522 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1515662938068_0005 = 0
2018-01-11 02:35:49,522 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1515662938068_0005Job Transitioned from NEW to INITED
2018-01-11 02:35:49,524 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1515662938068_0005.
2018-01-11 02:35:49,582 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
2018-01-11 02:35:49,602 INFO [Socket Reader #1 for port 55025] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 55025
2018-01-11 02:35:49,703 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2018-01-11 02:35:49,704 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-01-11 02:35:49,707 INFO [IPC Server listener on 55025] org.apache.hadoop.ipc.Server: IPC Server listener on 55025: starting
2018-01-11 02:35:49,713 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at quickstart.cloudera/127.0.0.1:55025
2018-01-11 02:35:49,886 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2018-01-11 02:35:49,960 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2018-01-11 02:35:49,968 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2018-01-11 02:35:49,995 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2018-01-11 02:35:50,007 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2018-01-11 02:35:50,007 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2018-01-11 02:35:50,066 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2018-01-11 02:35:50,066 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2018-01-11 02:35:50,085 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 52908
2018-01-11 02:35:50,085 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2018-01-11 02:35:50,136 INFO [main] org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.12.0.jar!/webapps/mapreduce to ./tmp/Jetty_0_0_0_0_52908_mapreduce____3vp14l/webapp
2018-01-11 02:35:50,569 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:52908
2018-01-11 02:35:50,569 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 52908
2018-01-11 02:35:51,415 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2018-01-11 02:35:51,423 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
2018-01-11 02:35:51,424 INFO [Socket Reader #1 for port 45459] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 45459
2018-01-11 02:35:51,440 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-01-11 02:35:51,441 INFO [IPC Server listener on 45459] org.apache.hadoop.ipc.Server: IPC Server listener on 45459: starting
2018-01-11 02:35:51,492 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2018-01-11 02:35:51,492 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2018-01-11 02:35:51,492 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2018-01-11 02:35:51,557 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
2018-01-11 02:35:51,702 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:8192, vCores:4>
2018-01-11 02:35:51,702 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.cloudera
2018-01-11 02:35:51,707 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2018-01-11 02:35:51,707 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2018-01-11 02:35:51,727 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1515662938068_0005Job Transitioned from INITED to SETUP
2018-01-11 02:35:51,736 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2018-01-11 02:35:51,738 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1515662938068_0005Job Transitioned from SETUP to RUNNING
2018-01-11 02:35:51,805 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1515662938068_0005_m_000000 Task Transitioned from NEW to SCHEDULED
2018-01-11 02:35:51,811 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1515662938068_0005_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-01-11 02:35:51,813 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>
2018-01-11 02:35:51,867 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1515662938068_0005, File: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/cloudera/.staging/job_1515662938068_0005/job_1515662938068_0005_1.jhist
2018-01-11 02:35:52,551 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020]
2018-01-11 02:35:52,705 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2018-01-11 02:35:52,776 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1515662938068_0005: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:6144, vCores:7> knownNMs=1
2018-01-11 02:35:54,798 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2018-01-11 02:35:54,802 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1515662938068_0005_01_000002 to attempt_1515662938068_0005_m_000000_0
2018-01-11 02:35:54,803 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2018-01-11 02:35:54,886 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Job jar is not present. Not adding any jar to the list of resources.
2018-01-11 02:35:54,913 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /tmp/hadoop-yarn/staging/cloudera/.staging/job_1515662938068_0005/job.xml
2018-01-11 02:35:55,137 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #1 tokens and #1 secret keys for NM use for launching container
2018-01-11 02:35:55,137 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 2
2018-01-11 02:35:55,137 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
2018-01-11 02:35:55,648 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
2018-01-11 02:35:55,672 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1515662938068_0005_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2018-01-11 02:35:55,682 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1515662938068_0005_01_000002 taskAttempt attempt_1515662938068_0005_m_000000_0
2018-01-11 02:35:55,690 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1515662938068_0005_m_000000_0
2018-01-11 02:35:55,830 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1515662938068_0005: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:5120, vCores:6> knownNMs=1
2018-01-11 02:35:56,004 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1515662938068_0005_m_000000_0 : 13562
2018-01-11 02:35:56,006 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1515662938068_0005_m_000000_0] using containerId: [container_1515662938068_0005_01_000002 on NM: [quickstart.cloudera:34543]
2018-01-11 02:35:56,011 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1515662938068_0005_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2018-01-11 02:35:56,011 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1515662938068_0005_m_000000 Task Transitioned from SCHEDULED to RUNNING
2018-01-11 02:35:59,461 INFO [Socket Reader #1 for port 45459] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1515662938068_0005 (auth:SIMPLE)
2018-01-11 02:35:59,495 INFO [IPC Server handler 1 on 45459] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1515662938068_0005_m_000002 asked for a task
2018-01-11 02:35:59,496 INFO [IPC Server handler 1 on 45459] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1515662938068_0005_m_000002 given task: attempt_1515662938068_0005_m_000000_0
2018-01-11 02:36:03,205 INFO [IPC Server handler 16 on 45459] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1515662938068_0005_m_000000_0 is : 0.0
2018-01-11 02:36:03,341 INFO [IPC Server handler 17 on 45459] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1515662938068_0005_m_000000_0 is : 1.0
2018-01-11 02:36:03,359 INFO [IPC Server handler 13 on 45459] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1515662938068_0005_m_000000_0
2018-01-11 02:36:03,365 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1515662938068_0005_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCESS_FINISHING_CONTAINER
2018-01-11 02:36:03,402 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1515662938068_0005_m_000000_0
2018-01-11 02:36:03,404 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1515662938068_0005_m_000000 Task Transitioned from RUNNING to SUCCEEDED
2018-01-11 02:36:03,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
2018-01-11 02:36:03,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1515662938068_0005Job Transitioned from RUNNING to COMMITTING
2018-01-11 02:36:03,411 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT
2018-01-11 02:36:03,443 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent 
2018-01-11 02:36:03,444 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1515662938068_0005Job Transitioned from COMMITTING to SUCCEEDED
2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2018-01-11 02:36:03,452 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2018-01-11 02:36:03,454 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
2018-01-11 02:36:03,581 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/cloudera/.staging/job_1515662938068_0005/job_1515662938068_0005_1.jhist to hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005-1515666941120-cloudera-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DOOZIE_SQOOP_WF%3AA%3Dsqoop_ac-1515666963441-1-0-SUCCEEDED-root.cloudera-1515666951717.jhist_tmp
2018-01-11 02:36:03,677 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005-1515666941120-cloudera-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DOOZIE_SQOOP_WF%3AA%3Dsqoop_ac-1515666963441-1-0-SUCCEEDED-root.cloudera-1515666951717.jhist_tmp
2018-01-11 02:36:03,683 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/cloudera/.staging/job_1515662938068_0005/job_1515662938068_0005_1_conf.xml to hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005_conf.xml_tmp
2018-01-11 02:36:03,722 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005_conf.xml_tmp
2018-01-11 02:36:03,732 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005.summary_tmp to hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005.summary
2018-01-11 02:36:03,735 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005_conf.xml_tmp to hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005_conf.xml
2018-01-11 02:36:03,739 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005-1515666941120-cloudera-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DOOZIE_SQOOP_WF%3AA%3Dsqoop_ac-1515666963441-1-0-SUCCEEDED-root.cloudera-1515666951717.jhist_tmp to hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/history/done_intermediate/cloudera/job_1515662938068_0005-1515666941120-cloudera-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DOOZIE_SQOOP_WF%3AA%3Dsqoop_ac-1515666963441-1-0-SUCCEEDED-root.cloudera-1515666951717.jhist
2018-01-11 02:36:03,742 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2018-01-11 02:36:03,743 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1515662938068_0005_m_000000_0
2018-01-11 02:36:03,773 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1515662938068_0005_m_000000_0 TaskAttempt Transitioned from SUCCESS_FINISHING_CONTAINER to SUCCEEDED
2018-01-11 02:36:03,782 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to 
2018-01-11 02:36:03,783 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://quickstart.cloudera:19888/jobhistory/job/job_1515662938068_0005
2018-01-11 02:36:03,797 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
2018-01-11 02:36:04,800 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2018-01-11 02:36:04,802 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://quickstart.cloudera:8020 /tmp/hadoop-yarn/staging/cloudera/.staging/job_1515662938068_0005
2018-01-11 02:36:04,813 INFO [Thread-68] org.apache.hadoop.ipc.Server: Stopping server on 45459
2018-01-11 02:36:04,819 INFO [IPC Server listener on 45459] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 45459
2018-01-11 02:36:04,820 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2018-01-11 02:36:04,822 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2018-01-11 02:36:04,828 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
2018-01-11 02:36:04,841 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job end notification started for jobID : job_1515662938068_0005
2018-01-11 02:36:04,843 INFO [Thread-68] org.mortbay.log: Job end notification attempts left 0
2018-01-11 02:36:04,843 INFO [Thread-68] org.mortbay.log: Job end notification trying http://quickstart.cloudera:11000/oozie/callback?id=0000002-180111013033300-oozie-oozi-W@sqoop_action&status=SUCCEEDED
2018-01-11 02:36:04,863 INFO [Thread-68] org.mortbay.log: Job end notification to http://quickstart.cloudera:11000/oozie/callback?id=0000002-180111013033300-oozie-oozi-W@sqoop_action&status=SUCCEEDED succeeded
2018-01-11 02:36:04,863 INFO [Thread-68] org.mortbay.log: Job end notification succeeded for job_1515662938068_0005
2018-01-11 02:36:09,868 INFO [Thread-68] org.apache.hadoop.ipc.Server: Stopping server on 55025
2018-01-11 02:36:09,959 INFO [IPC Server listener on 55025] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 55025
2018-01-11 02:36:10,048 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2018-01-11 02:36:10,317 INFO [Thread-68] org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0

 
What am I doing wrong?

1 ACCEPTED SOLUTION

avatar
Expert Contributor

 

 

Finally solved by adding the connection manager parameter and the Teradata jdbc drivers in the path:

 

sqoop import \
--connection-manager com.teradata.jdbc.TeraDriver \
--connect jdbc:teradata://host/DATABASE=db \
[...]


The following files should be available to Oozie (I've put them in the application lib folder):

  • sqoop-connector-teradata-1.6c5.jar
  • tdgssconfig.jar
  • terajdbc4.jar

 

I guess the documentation should be updated, since with the informations provided the import from Teradata works from Sqoop standalone but fails always inside Oozie.

 

 

View solution in original post

1 REPLY 1

avatar
Expert Contributor

 

 

Finally solved by adding the connection manager parameter and the Teradata jdbc drivers in the path:

 

sqoop import \
--connection-manager com.teradata.jdbc.TeraDriver \
--connect jdbc:teradata://host/DATABASE=db \
[...]


The following files should be available to Oozie (I've put them in the application lib folder):

  • sqoop-connector-teradata-1.6c5.jar
  • tdgssconfig.jar
  • terajdbc4.jar

 

I guess the documentation should be updated, since with the informations provided the import from Teradata works from Sqoop standalone but fails always inside Oozie.