Support Questions
Find answers, ask questions, and share your expertise

sqoop import AM container job failed

Contributor

HI Team,

I am getting below an error while importing tables from oracle database.

Job job_1459150653243_0002 failed with state FAILED due to: Application application_1459150653243_0002 failed 2 times due to AM Container for appattempt_1459150653243_0002_000002 exited with exitCode: -1000

Please see attached screenshots of an error.

sqoop-job-failed.jpg

11 REPLIES 11

Expert Contributor

Hello, if you run the application again, check the specific application log for more information. You can post that file as well.

@Nilesh

See this and also run yarn log -applicationid application_1459150653243_0002 and post the output

3049-screen-shot-2016-03-28-at-21856-pm.png

Contributor

@Neeraj Sabharwal

showing below error

[hdfs@nn1 ~]$ yarn log -applicationid application_1459150653243_0004

Error: Could not find or load main class log

Expert Contributor

@Nilesh

typo

yarn logs -applicationId <app_id>

Contributor

@Mats Johansson

Below is the output I got after issuing command

/app-logs/hdfs/logs/application_1459253164991_0004 does not have any log files.

Expert Contributor

@Nilesh

Did just see your ExitCode: -1000 (just answered the yarn logs typo first). -1000 indicates a permission problem on your local usercache.

* First find your local cache. From the error above it looks like /hadoop/yarn/local/filecache?

* Parameters involved: If you don't have the yarn.nodemanager.local-dirs configured anywhere, look for it in the default location ${hadoop.tmp.dir}/nm-local-dir. If the hadoop.tmp.dir is not configured in the core-site.xml, it will be under /tmp/hadoop-${user.name}. The user.name is the UNIX user that you are using to run the current Hadoop process.

* For the next step please be careful to pick the correct file/dir if it's a cluster with several jobs runnings.

* Then delete the files/directories under the local cache related to your user/job at all the node manager nodes. Also make sure you have appropriate space available for the local cache. Please observe that you have to do this on all your nodes.

Contributor
@Mats Johansson

No I can`t see Exitcode: -1000 please find below output for the same

[hdfs@nn1 ~]$ yarn logs -applicationId application_1459253164991_0004

16/03/30 11:43:47 INFO impl.TimelineClientImpl: Timeline service address: http://nn1.hdp.local:8188/ws/v1/timeline/ 16/03/30 11:43:47 INFO client.RMProxy: Connecting to ResourceManager at nn1.hdp.local/192.168.3.47:8050 /app-logs/hdfs/logs/application_1459253164991_0004 does not have any log files.

Along with this please see the output where I can not find 14_tmp folder in /hadoop/yarn/local/filcache .

3085-sqoop-mrjob-failed.jpg

Expert Contributor

@Nilesh - There are ExitCode -1000 in your first post and also in your latest screen dump. -1000 are related to permissions on the local cache directories on every node.

If there is no /hadoop/yarn/... directory structure, check the yarn.nodemanager.local-dirs and related parameters in core-site.xml.

If dir exists - check those directories for permissions and free space. Also check which OS user/group is running the failing job and make sure it has the correct permissions.

If the job runs for a while before failing please check the size of the files under /hadoop/yarn/local so it does not reach any OS or filesystem limits.

Contributor

@Mats Johansson

I changed "yarn.nodemanager.local-dirs" parameter from /hadoop/yarn/local to /tmp/yarn/local from ambari and recursively changed permission to 777 for /yarn/local folder. Also there is sufficient space available in /tmp but still got same error