Support Questions

Find answers, ask questions, and share your expertise

sqoop import issue

avatar
Explorer

Hi 

I am trying to import a single table from sqoop and i get this error:

 

Warning: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/06/18 16:25:25 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.0
19/06/18 16:25:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/06/18 16:25:26 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
19/06/18 16:25:26 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
19/06/18 16:25:26 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop will not process this sqoop connection, as an insufficient number of mappers are being used.
19/06/18 16:25:26 INFO manager.SqlManager: Using default fetchSize of 1000
19/06/18 16:25:26 INFO tool.CodeGenTool: Beginning code generation
19/06/18 16:25:26 INFO tool.CodeGenTool: Will generate java class as codegen_WORKFLOW
19/06/18 16:25:27 INFO manager.OracleManager: Time zone has been set to GMT
19/06/18 16:25:27 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM WORKFLOW t WHERE 1=0
19/06/18 16:25:27 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
19/06/18 16:25:29 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-cloudera/compile/e8c2761367830b3f0e903699f598700b/codegen_WORKFLOW.java to /home/cloudera/./codegen_WORKFLOW.java. Error: Destination '/home/cloudera/./codegen_WORKFLOW.java' already exists
19/06/18 16:25:29 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/e8c2761367830b3f0e903699f598700b/codegen_WORKFLOW.jar
19/06/18 16:25:29 INFO manager.OracleManager: Time zone has been set to GMT
19/06/18 16:25:29 WARN manager.OracleManager: The table WORKFLOW contains a multi-column primary key. Sqoop will default to the column IDWORKFLOW only for this job.
19/06/18 16:25:29 INFO manager.OracleManager: Time zone has been set to GMT
19/06/18 16:25:29 WARN manager.OracleManager: The table WORKFLOW contains a multi-column primary key. Sqoop will default to the column IDWORKFLOW only for this job.
19/06/18 16:25:29 INFO mapreduce.ImportJobBase: Beginning import of WORKFLOW
19/06/18 16:25:29 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/06/18 16:25:29 INFO manager.OracleManager: Time zone has been set to GMT
19/06/18 16:25:30 INFO manager.OracleManager: Time zone has been set to GMT
19/06/18 16:25:30 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM WORKFLOW t WHERE 1=0
19/06/18 16:25:30 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM WORKFLOW t WHERE 1=0
19/06/18 16:25:30 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/06/18 16:25:30 INFO client.RMProxy: Connecting to ResourceManager at clouderasrv/172.23.16.226:8032
19/06/18 16:25:31 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/cloudera/.staging/job_1560863992639_0002
19/06/18 16:26:23 INFO db.DBInputFormat: Using read commited transaction isolation
19/06/18 16:26:24 INFO mapreduce.JobSubmitter: number of splits:1
19/06/18 16:26:24 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
19/06/18 16:26:25 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1560863992639_0002
19/06/18 16:26:25 INFO mapreduce.JobSubmitter: Executing with tokens: []
19/06/18 16:26:25 INFO conf.Configuration: resource-types.xml not found
19/06/18 16:26:25 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
19/06/18 16:26:26 INFO impl.YarnClientImpl: Submitted application application_1560863992639_0002
19/06/18 16:26:26 INFO mapreduce.Job: The url to track the job: http://clouderasrv:8088/proxy/application_1560863992639_0002/
19/06/18 16:26:26 INFO mapreduce.Job: Running job: job_1560863992639_0002
19/06/18 16:26:36 INFO mapreduce.Job: Job job_1560863992639_0002 running in uber mode : false
19/06/18 16:26:36 INFO mapreduce.Job: map 100% reduce 0%
19/06/18 16:26:37 INFO mapreduce.Job: Job job_1560863992639_0002 failed with state KILLED due to: The required MAP capability is more than the supported max container capability in the cluster. Killing the Job. mapResourceRequest: <memory:2560, vCores:1> maxContainerCapability:<memory:2048, vCores:4>
Job received Kill while in RUNNING state.

19/06/18 16:26:37 INFO mapreduce.Job: Counters: 3
Job Counters
Killed map tasks=1
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=0
19/06/18 16:26:37 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
19/06/18 16:26:37 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 66.9005 seconds (0 bytes/sec)
19/06/18 16:26:37 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
19/06/18 16:26:37 INFO mapreduce.ImportJobBase: Retrieved 0 records.
19/06/18 16:26:37 ERROR tool.ImportTool: Import failed: Import job failed!

 

Any idea why this error is coming up?

1 ACCEPTED SOLUTION

avatar
Mentor
It could be passed by either modes, hence the request for the CLI used.

The property to modify on the client configuration (via CM properties or
via -D early CLI args) is called 'mapreduce.map.memory.mb', and the
administrative limit is defined in the Resource Manager daemon
configuration via 'yarn.scheduler.maximum-allocation-mb'

View solution in original post

4 REPLIES 4

avatar
Mentor
Please share your full Sqoop CLI.

The error you are receiving suggests that the configuration passed to this
specific Sqoop job carried a parameter asking for Map memory to be higher
than what the administrator has configured as a limit a Map task may
request. As a result, the container request is rejected. Lowering the
request memory size of map tasks will let it pass through this check.

avatar
Explorer
Can you please advice how to do this?Is this a parameter in cloudera manager
or parameter in sqoop import command?
Thanks

avatar
Mentor
It could be passed by either modes, hence the request for the CLI used.

The property to modify on the client configuration (via CM properties or
via -D early CLI args) is called 'mapreduce.map.memory.mb', and the
administrative limit is defined in the Resource Manager daemon
configuration via 'yarn.scheduler.maximum-allocation-mb'

avatar
Explorer
Thank you Harsh