sqoop import issue

andreas — Fri, 16 Sep 2022 14:27:27 GMT

I am trying to import a single table from sqoop and i get this error:

Warning: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/06/18 16:25:25 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.0
19/06/18 16:25:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/06/18 16:25:26 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
19/06/18 16:25:26 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
19/06/18 16:25:26 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop will not process this sqoop connection, as an insufficient number of mappers are being used.
19/06/18 16:25:26 INFO manager.SqlManager: Using default fetchSize of 1000
19/06/18 16:25:26 INFO tool.CodeGenTool: Beginning code generation
19/06/18 16:25:26 INFO tool.CodeGenTool: Will generate java class as codegen_WORKFLOW
19/06/18 16:25:27 INFO manager.OracleManager: Time zone has been set to GMT
19/06/18 16:25:27 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM WORKFLOW t WHERE 1=0
19/06/18 16:25:27 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
19/06/18 16:25:29 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-cloudera/compile/e8c2761367830b3f0e903699f598700b/codegen_WORKFLOW.java to /home/cloudera/./codegen_WORKFLOW.java. Error: Destination '/home/cloudera/./codegen_WORKFLOW.java' already exists
19/06/18 16:25:29 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/e8c2761367830b3f0e903699f598700b/codegen_WORKFLOW.jar
19/06/18 16:25:29 INFO manager.OracleManager: Time zone has been set to GMT
19/06/18 16:25:29 WARN manager.OracleManager: The table WORKFLOW contains a multi-column primary key. Sqoop will default to the column IDWORKFLOW only for this job.
19/06/18 16:25:29 INFO manager.OracleManager: Time zone has been set to GMT
19/06/18 16:25:29 WARN manager.OracleManager: The table WORKFLOW contains a multi-column primary key. Sqoop will default to the column IDWORKFLOW only for this job.
19/06/18 16:25:29 INFO mapreduce.ImportJobBase: Beginning import of WORKFLOW
19/06/18 16:25:29 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/06/18 16:25:29 INFO manager.OracleManager: Time zone has been set to GMT
19/06/18 16:25:30 INFO manager.OracleManager: Time zone has been set to GMT
19/06/18 16:25:30 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM WORKFLOW t WHERE 1=0
19/06/18 16:25:30 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM WORKFLOW t WHERE 1=0
19/06/18 16:25:30 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/06/18 16:25:30 INFO client.RMProxy: Connecting to ResourceManager at clouderasrv/172.23.16.226:8032
19/06/18 16:25:31 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/cloudera/.staging/job_1560863992639_0002
19/06/18 16:26:23 INFO db.DBInputFormat: Using read commited transaction isolation
19/06/18 16:26:24 INFO mapreduce.JobSubmitter: number of splits:1
19/06/18 16:26:24 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
19/06/18 16:26:25 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1560863992639_0002
19/06/18 16:26:25 INFO mapreduce.JobSubmitter: Executing with tokens: []
19/06/18 16:26:25 INFO conf.Configuration: resource-types.xml not found
19/06/18 16:26:25 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
19/06/18 16:26:26 INFO impl.YarnClientImpl: Submitted application application_1560863992639_0002
19/06/18 16:26:26 INFO mapreduce.Job: The url to track the job: http://clouderasrv:8088/proxy/application_1560863992639_0002/
19/06/18 16:26:26 INFO mapreduce.Job: Running job: job_1560863992639_0002
19/06/18 16:26:36 INFO mapreduce.Job: Job job_1560863992639_0002 running in uber mode : false
19/06/18 16:26:36 INFO mapreduce.Job: map 100% reduce 0%
19/06/18 16:26:37 INFO mapreduce.Job: Job job_1560863992639_0002 failed with state KILLED due to: The required MAP capability is more than the supported max container capability in the cluster. Killing the Job. mapResourceRequest: <memory:2560, vCores:1> maxContainerCapability:<memory:2048, vCores:4>
Job received Kill while in RUNNING state.

19/06/18 16:26:37 INFO mapreduce.Job: Counters: 3
Job Counters
Killed map tasks=1
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=0
19/06/18 16:26:37 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
19/06/18 16:26:37 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 66.9005 seconds (0 bytes/sec)
19/06/18 16:26:37 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
19/06/18 16:26:37 INFO mapreduce.ImportJobBase: Retrieved 0 records.
19/06/18 16:26:37 ERROR tool.ImportTool: Import failed: Import job failed!

Any idea why this error is coming up?

Re: sqoop import issue

Harsh J — Tue, 18 Jun 2019 14:36:59 GMT

Please share your full Sqoop CLI.

The error you are receiving suggests that the configuration passed to this
specific Sqoop job carried a parameter asking for Map memory to be higher
than what the administrator has configured as a limit a Map task may
request. As a result, the container request is rejected. Lowering the
request memory size of map tasks will let it pass through this check.

Re: sqoop import issue

andreas — Tue, 18 Jun 2019 15:06:59 GMT

Can you please advice how to do this?Is this a parameter in cloudera manager
or parameter in sqoop import command?
Thanks

Re: sqoop import issue

Harsh J — Wed, 19 Jun 2019 06:28:59 GMT

It could be passed by either modes, hence the request for the CLI used.

The property to modify on the client configuration (via CM properties or
via -D early CLI args) is called 'mapreduce.map.memory.mb', and the
administrative limit is defined in the Resource Manager daemon
configuration via 'yarn.scheduler.maximum-allocation-mb'

Re: sqoop import issue

andreas — Wed, 19 Jun 2019 08:00:47 GMT

Thank you Harsh

question Re: sqoop import issue in Support Questions

sqoop import issue

Re: sqoop import issue

Re: sqoop import issue

Re: sqoop import issue

Re: sqoop import issue