Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

sqoop2 import from oracle to HDFS doesnt work - CDH5.0, RHEL6

sqoop2 import from oracle to HDFS doesnt work - CDH5.0, RHEL6

New Contributor

 

I was trying to import a simple table from Oracle database to HDFS using sqoop2. I have setup the database connection through Hue and setup a job. The job was getting submitted and the job status was showing "Booting" in hue. When I go back into Hue logs or Sqoop2 logs, I see the below messages. It was getting the table structure from oracle. So I believe it's succeeding in fetching the oracle database table structure and was able to communicate to the oracle database without any issues. No errors messages are showing in sqoop or hue logs, so it was getting really hard to troubleshoot. All I see was "Attempting transaction commit". Please help.

 

Also is there any debugging option that I can enable to trace down the issue?

 

 

Boundaries: min=15, max=607702, columnType=2
Using dataSql: select * from sqoop.articles where ${CONDITIONS}
Field names: ARTICLE_NUMBER,PUBLICATION_NUMBER,IS_NUMBER,IEE_TYPE,DATE_OF_LAST_UPDATE,DOCUMENT_TYPE,INSPEC_ACCESSION_NUMBER,DATE_OF_INSERTION,TAPE_SERVICE_ISSUE_YEAR,RECORD_TYPE,TAPE_SERVICE_ISSUE_ISS,SOURCE_OF_RECORD,COPYRIGHT_NAME,STATUS_OF_RECORD,COPYRIGHT_YEAR,NEW_RECORD_TYPE,RECORD_TYPE_NAME,AMENDMENT_START_DATE,AMENDMENT_ORIGINAL_DATE,JOURNAL_ID_NO,ITEM_TITLE,PUBLICATION_TITLE,PUBLICATION_ABBREVIATED_TITLE,PUBLICATION_START_DATE,PUBLICATION_END_DATE,PUBLICATION_ORIGINAL_DATE,PATENT_NUMBER,CODEN,PATENT_COUNTRY,ISSN,PATENT_START_DATE,PATENT_ORIGINAL_DATE,PATENT_ASSIGNEE,PUBLISHERS_NAME,PUBLISHERS_COUNTRY,PUBLISHERS_LOCATION,DOCUMENT_NUMBER,ISBN,SICI_NUMBER,COPYRIGHT_CCC,AVAILABILITY,PRICE,CONFERENCE_START_DATE,UNCONVENTIONAL_MEDIUM,CONFERENCE_END_DATE,URL,CONFERENCE_ORIGINAL_DATE,DOCUMENT_COLLECTION_URL,CONFERENCE_LOCATION,CONFERENCE_COUNTRY,LANGUAGE,TRANSLATOR,MATERIAL_IDENT_NO,MULTIMEDIA_FLAG,HTML_FLAG,PDF_FILE_LOCATION,BIOMEDICAL_ENG,ASPP_FLAG,ASPP_UPP_FLAG,SPECTRUM_FLAG,SPECTRUM_UPP_FLAG,PROCEEDINGS_FLAG,POP_FLAG,POP_ALL_FLAG,POP_ALL_UPP_FLAG,IEL_FLAG,PDF_FILE_SIZE,YEAR,CHRON,END_PAGE,START_PAGE,ABSTRACT,NUMBER_OF_REFERENCES,Y2K_MIN,ISSUE_VOLUME,ISSUE_NUMBER,NUMBER_OF_PAGES,PART,DOI,CROSS_REF,PAGE_NUMBERS,CATALOG_NUMBER,TRACE_DATE,CSMDL_FLAG,PES_FLAG,LEOS_FLAG,ITEL_FLAG,EMBS_FLAG,SSCS_FLAG,PUBLICATION_DATE,CONFERENCE_DATE,NON_IEEE,SEQUENCE_NUMBER,PUBITYPE,INDEX_CONTENT,RSCDL_FLAG,CIP_FLAG,CSDL_FLAG,CPES_FLAG,EDMCDL_FLAG,PESCDL_FLAG,CSCDL_FLAG,RATP_FLAG,PESDL_FLAG,REFERENCES,FORWARD_REFERENCES,NOTES,STDS_PRODUCT_NUMBER,EISBN,IACDL_FLAG,UFFCCDL_FLAG,IPEL_FLAG,LETTER,GRSDL_FLAG,ITSCDL_FLAG,APSDL_FLAG,SMCSDL_FLAG,EMCSDL_FLAG,CESDL_FLAG,OESDL_FLAG,NPSSDL_FLAG,EMBSC_FLAG,MTTSDL_FLAG,CL_FLAG,PUBMED_ID,ONLINE_DATE,BTSDL_FLAG,ENTERPRISECABINET_FLAG,STANDARDCABINET_FLAG,OPENACCESS_FLAG,DATE_OF_PREPRINT,DATE_OF_RAPID_POST,ARTICLE_STAGE,MDLCABINET_FLAG,ASPP_2005,POP_2005,POP_ALL_2005,PBD_FORMAT,RIGHTSLINK_FLAG,GL_ACCOUNT,HTML_FLAG_UPDATE_DATE,ML_HTML_FLAG,EXTERNAL_ID,SCOPE,PURPOSE,TOKEN_VALUE
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
Attempting transaction commit
3 REPLIES 3
Highlighted

Re: sqoop2 import from oracle to HDFS doesnt work - CDH5.0, RHEL6

Expert Contributor
Hey there, what do your task logs say?

Re: sqoop2 import from oracle to HDFS doesnt work - CDH5.0, RHEL6

New Contributor

Thanks for the quick response.

Both Resource manager and node manager aren't showing any errors, logs as below:
However, in the nodemanager UI, it shows that the job has failed:

Job Name:User Name:Queue:State:Uberized:Submitted:Started:Finished:Elapsed:Diagnostics:
Sqoop: DXPL2HDFS
sqoop2
root.sqoop2
FAILED
false
Thu May 07 13:22:52 EDT 2015
Thu May 07 13:22:57 EDT 2015
Thu May 07 13:23:18 EDT 2015
21sec
Task task_1430248435867_0029_r_000000 failed 0 times

Logs:

Resource Manager:
====================
2015-05-07 13:23:19,980 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=sqoop2 OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1430248435867_0029 CONTAINERID=container_1430248435867_0029_01_000005
2015-05-07 13:23:19,980 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode: Released container container_1430248435867_0029_01_000005 of capacity <memory:1024, vCores:1> on host dbaddv02:8041, which currently has 1 containers, <memory:1024, vCores:1> used and <memory:5038, vCores:1> available, release resources=true
2015-05-07 13:23:19,980 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application attempt appattempt_1430248435867_0029_000001 released container container_1430248435867_0029_01_000005 on node: host: dbaddv02:8041 #containers=1 available=<memory:5038, vCores:1> used=<memory:1024, vCores:1> with event: FINISHED
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1430248435867_0029_01_000001 Container Transitioned from RUNNING to COMPLETED
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1430248435867_0029_000001
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerApp: Completed container: container_1430248435867_0029_01_000001 in state: COMPLETED event:FINISHED
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1430248435867_0029_000001 State change from FINISHING to FINISHED
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=sqoop2 OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1430248435867_0029 CONTAINERID=container_1430248435867_0029_01_000001
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1430248435867_0029 State change from FINISHING to FINISHED
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=sqoop2 OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1430248435867_0029 CONTAINERID=container_1430248435867_0029_01_000001
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1430248435867_0029 State change from FINISHING to FINISHED
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode: Released container container_1430248435867_0029_01_000001 of capacity <memory:1024, vCores:1> on host dbaddv02:8041, which currently has 0 containers, <memory:0, vCores:0> used and <memory:6062, vCores:2> available, release resources=true
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=sqoop2 OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1430248435867_0029
2015-05-07 13:23:25,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application attempt appattempt_1430248435867_0029_000001 released container container_1430248435867_0029_01_000001 on node: host: dbaddv02:8041 #containers=0 available=<memory:6062, vCores:2> used=<memory:0, vCores:0> with event: FINISHED
2015-05-07 13:23:25,986 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1430248435867_0029,name=Sqoop: DXPL2HDFS_WORKSHOP,user=sqoop2,queue=root.sqoop2,state=FINISHED,trackingUrl=http://dbaddv02:8088/proxy/application_1430248435867_0029/jobhistory/job/job_1430248435867_0029,appMasterHost=dbaddv02,startTime=1431019372970,finishTime=1431019399175,finalStatus=FAILED
2015-05-07 13:23:25,986 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application appattempt_1430248435867_0029_000001 is done. finalState=FINISHED
2015-05-07 13:23:25,986 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1430248435867_0029_000001
2015-05-07 13:23:25,986 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1430248435867_0029 requests cleared

node manager
==============
2015-05-07 13:23:25,745 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=sqoop2 OPERATION=Container Finished - Succeeded TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1430248435867_0029 CONTAINERID=container_1430248435867_0029_01_000001
2015-05-07 13:23:25,745 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1430248435867_0029_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE
2015-05-07 13:23:25,745 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1430248435867_0029_01_000001 from application application_1430248435867_0029
2015-05-07 13:23:25,745 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_1430248435867_0029_01_000001 for log-aggregation
2015-05-07 13:23:25,745 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1430248435867_0029
2015-05-07 13:23:25,984 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed container container_1430248435867_0029_01_000001
2015-05-07 13:23:25,992 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1430248435867_0029_01_000001
2015-05-07 13:23:26,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1430248435867_0029 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2015-05-07 13:23:26,986 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /infra1/yarn/nm/usercache/sqoop2/appcache/application_1430248435867_0029
2015-05-07 13:23:26,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1430248435867_0029
2015-05-07 13:23:26,987 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1430248435867_0029 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2015-05-07 13:23:26,987 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1430248435867_0029
2015-05-07 13:23:26,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1430248435867_0029
2015-05-07 13:23:26,987 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1430248435867_0029 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2015-05-07 13:23:26,987 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1430248435867_0029
2015-05-07 13:23:26,987 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Starting aggregate log-file for app application_1430248435867_0029 at /tmp/logs/sqoop2/logs/application_1430248435867_0029/dbaddv02_8041.tmp
2015-05-07 13:23:26,987 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /mysqldata/yarn/nm/usercache/sqoop2/appcache/application_1430248435867_0029
2015-05-07 13:23:26,988 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /oh1/yarn/nm/usercache/sqoop2/appcache/application_1430248435867_0029
2015-05-07 13:23:27,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_1430248435867_0029_01_000002. Current good log dirs are /var/log/hadoop-yarn/container
2015-05-07 13:23:27,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_1430248435867_0029_01_000005. Current good log dirs are /var/log/hadoop-yarn/container
2015-05-07 13:23:27,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_1430248435867_0029_01_000001. Current good log dirs are /var/log/hadoop-yarn/container
2015-05-07 13:23:27,042 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/log/hadoop-yarn/container/application_1430248435867_0029
2015-05-07 13:23:27,061 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Finished aggregate log-file for app application_1430248435867_0029
2015-05-07 13:23:28,071 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1430248435867_0029_01_000001

Re: sqoop2 import from oracle to HDFS doesnt work - CDH5.0, RHEL6

New Contributor

This is what I see in the task log through UI in all the three servers of the cluster ( same message being repeated for all node managers in the cluster):

 

Error: org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs during extractor run at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0013:Cannot write to the data writer at org.apache.sqoop.job.mr.SqoopMapper$MapDataWriter.writeContent(SqoopMapper.java:149) at org.apache.sqoop.job.mr.SqoopMapper$MapDataWriter.writeArrayRecord(SqoopMapper.java:131) at org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:61) at org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:31) at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:96) ... 7 more Caused by: java.io.IOException: org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0012:The type is not supported - java.math.BigDecimal at org.apache.sqoop.job.io.Data.writeArray(Data.java:309) at org.apache.sqoop.job.io.Data.write(Data.java:171) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:98) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:82) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1126) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.sqoop.job.mr.SqoopMapper$MapDataWriter.writeContent(SqoopMapper.java:147) ... 11 more Caused by: org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0012:The type is not supported - java.math.BigDecimal ... 20 more