question Re: Sqoop Error, Sqoop 2 not working through hue in Support Questions

Sqoop Error, Sqoop 2 not working through hue

Jais — Fri, 16 Sep 2022 09:20:41 GMT

Hello,

We have installed CDH 5.3 on 20 nodes, which comes with both sqoop 1 and sqoop2.

I am trying to run a sqoop 1 job through command line interactive shell, but unable to as it throws execption. As I change the jar, the error differs.

DB: postgresql

I have kept the jar file for postgresql inside /var/lib/sqoop/

I tried to give all required permissions on the sqoop, but still unable to run it.

It says no db jar selected or read.

When I try to use sqoop 2 through Hue, It doesnot allow me to create sqoop job, it is like when you click on create new job, it does not go to next page.

Re: Sqoop Error, Sqoop 2 not working through hue

Jais — Sat, 31 Jan 2015 16:53:09 GMT

Following is the error I am getting.....

Re: Sqoop Error, Sqoop 2 not working through hue

Jais — Sat, 31 Jan 2015 17:24:06 GMT

If I try same command from different node, the error differs....

sqoop import --connect jdbc:postgresql://10.4.2.68:6453/snap318 --username ae -P --table lead_dmn --target-dir /user/njaiswal/cidb/prime
Warning: /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/01/31 17:22:18 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.0
Enter password:
15/01/31 17:22:22 INFO manager.SqlManager: Using default fetchSize of 1000
15/01/31 17:22:22 INFO tool.CodeGenTool: Beginning code generation
15/01/31 17:22:22 ERROR manager.SqlManager: Error executing statement: org.postgresql.util.PSQLException: ERROR: Encountered "AS" at line 1, column 29.
Was expecting one of:
<EOF>
";" ...

org.postgresql.util.PSQLException: ERROR: Encountered "AS" at line 1, column 29.
Was expecting one of:
<EOF>
";" ...

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
at org.postgresql.jdbc2.AbstractJdbc2Connection.execSQLUpdate(AbstractJdbc2Connection.java:263)
at org.postgresql.jdbc2.AbstractJdbc2Connection.setTransactionIsolation(AbstractJdbc2Connection.java:829)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:883)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
15/01/31 17:22:22 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Re: Sqoop Error, Sqoop 2 not working through hue

abe — Wed, 04 Feb 2015 22:07:58 GMT

Sqoop2 is going throgh a lot of changes and isn't supported in Hue in CDH 5.3.0. It should be in CDH 5.3.2 AFAIK.

The Sqoop1 command you are running have two different problems:

The postgresql jdbc driver jar isn't available on one of your nodes. This can be fixed by downloading it and putting it in /var/lib/sqoop.
The problem seems to be some kind of SQL error. Could you add --verbose to your sqoop command? sqoop import --verbose ...

Re: Sqoop Error, Sqoop 2 not working through hue

Jais — Wed, 04 Feb 2015 22:53:58 GMT

Hi abe,

I am getting new issue on hdfs,

sqoop is running fine and it creates an output directory on the given --target-dir path /xxx.........but the issue is I can see the directory on hdfs, but cannot access, when given hadoop fs -ls /xxx , it says no such file or directory.....Pls help me to resolve this issue

Re: Sqoop Error, Sqoop 2 not working through hue

Jais — Wed, 04 Feb 2015 23:03:32 GMT

Also when I tried to access it through Hue.....It says following

You are a Hue admin but not a HDFS superuser (which is "hdfs").

Re: Sqoop Error, Sqoop 2 not working through hue

abe — Wed, 04 Feb 2015 23:38:37 GMT

It seems like the job didn't run. Are you sure it finished? There should be task logs associated with your job. Could you check them for errors? Also, make sure --verbose is in your command for the best possible debug information.

Re: Sqoop Error, Sqoop 2 not working through hue

Jais — Thu, 05 Feb 2015 01:07:42 GMT

Yes, the Job run successfully, but still I am unable to view the file as the directory which has it, doesnot open.......

sqoop ran successfully and retrieved 4 records, and also created the directory by itself when I specify explecitly from command line,

It is not allowing me to look into direcctory.......it says no such file or directory, but I can notice that it is present......something to do with permissions for hdfs??

Re: Sqoop Error, Sqoop 2 not working through hue

abe — Thu, 05 Feb 2015 01:27:10 GMT

From the CLI, Hadoop normally shows: "ls: Permission denied: user=admin..." if you don't have permission. What is your --target-dir set to?

Re: Sqoop Error, Sqoop 2 not working through hue

Jais — Thu, 05 Feb 2015 01:33:45 GMT

You are right it should say permission denied, but strangely it says "No file or directory found" , but we can see that directory is present.....

Re: Sqoop Error, Sqoop 2 not working through hue

Jais — Thu, 05 Feb 2015 01:37:35 GMT

Hey abe ,

I am running the sqoop command on node 1 and the cluster comprises of 20 nodes..... 1nn , 19 dn...

I can access hdfs from all the nodes, also able to access directories that was created using hadoop fs -mkdir /user/xyz/abc......., The only issue is unable to access directory created by sqoop import, though the sqoop job was successful

Re: Sqoop Error, Sqoop 2 not working through hue

Jais — Thu, 05 Feb 2015 01:45:49 GMT

Hello abe,

following is the sqoop command:

sqoop import --connect jdbc:mysql://ip-address/db --username abc --password cde --query "select * from hfpoverty WHERE hindus_females > 10 AND \$CONDITIONS" --target-dir /user/jais/test3 -m 1

Re: Sqoop Error, Sqoop 2 not working through hue

Jais — Thu, 05 Feb 2015 01:53:15 GMT

So, this directory has been created test3........but unable to access it as it says no such file or directory........

Re: Sqoop Error, Sqoop 2 not working through hue

Jais — Tue, 10 Feb 2015 05:53:01 GMT

Hello abe,

Thanks for helping me, the issue has been resolved bby itself, do not know what happened, I felt there some kind of special characters auto generated on top of the directory name, hence it was saying no such file or directory, as soon as I deleted the directory and created the new one, it works fine.

Re: Sqoop Error, Sqoop 2 not working through hue

nara — Wed, 18 Feb 2015 22:17:17 GMT

I have somewhat similar issue but my error is little different.

Command line -

sudo -u hdfs sqoop import --connect jdbc:oracle:thin:@//BIDEVDC01.gmpvt.net:1521/BIDEV --table CM_GIS_OUTAGE_EXTRACT --fields-terminated-by '\t' --username DWADM --password dwadm --target-dir /user/etl/sqoop/init/CM_GIS_OUTAGE_EXTRACT --verbose --connection-manager org.apache.sqoop.manager.OracleManager --num-mappers 4 --mapreduce-job-name CM_GIS_OUTAGE_EXTRACT --direct --split-by DEVICE_OID --hive-import &> sqoop-CM_GIS_OUTAGE_EXTRACT.log

ERROR -

15/02/18 17:16:14 INFO mapreduce.Job: map 0% reduce 0%
15/02/18 17:16:14 INFO mapreduce.Job: Job job_1423869234667_0007 failed with state FAILED due to: Application application_1423869234667_0007 failed 2 times due to AM Container for appattempt_1423869234667_0007_000002 exited with exitCode: -1000 due to: Application application_1423869234667_0007 initialization failed (exitCode=255) with output: main : command provided 0
main : user is nobody
main : requested yarn user is hdfs
Can't create directory /data1/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data10/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data11/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data12/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data2/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data3/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data4/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data5/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data6/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data7/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data8/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Can't create directory /data9/yarn/nm/usercache/hdfs/appcache/application_1423869234667_0007 - Permission denied
Did not create any app directories

Re: Sqoop Error, Sqoop 2 not working through hue

perk — Wed, 18 Mar 2015 10:49:01 GMT

Nara,

you could try deleting the /data1/yarn/nm/usercache/hdfs directory from your datanodes - as suggested in here:

http://community.cloudera.com/t5/Batch-Processing-and-Workflow/Can-t-create-directory-yarn-nm-usercache-urika-appcache/td-p/24891

That worked for me.