Support Questions
Find answers, ask questions, and share your expertise

sqoop stuck at No connection paramenters specified. Using regular API for making connection.

Contributor

Hi,

 

 

I am trying to run this import command:

 

sqoop import --connect jdbc:postgresql://[host]:[port]/[db] --username [user] --password-file [path_to_pass] -table [table] --hive-import -hive-drop-import-delims --as-parquetfile --parquet-configurator-implementation hadoop --hive-table district_startdates_test --target-dir [target_with_db] -m 1 --hs2-url="jdbc:hive2://[cdp]/;ssl=true;transportMode=http;httpPath=[path_to_proxy];user=[user];password=[cluster_passw]" --verbose

 

but it gets stuck at  

 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.

 

...any idea?

thanks

1 ACCEPTED SOLUTION

Contributor

Hi @mszurap , 

 

 

thanks for the tool

So, somehow the connection is now working , namely the command we tried to cloudera support, like:

`sqoop eval --connect ...`

or

 ` sqoop list-databases --connect` 

 

before running any sqoop command, we set up a ssh tunnel, with 

ssh -o ServerAliveInterval=60 -M -S mktemp -fN -L local_port:psql_fqdn:remote_p  user@host_IP 

 

(which I was doing also before, so not sure why now it works... 

I will mark it now as solved

 

thanks anyway,

Gr

View solution in original post

10 REPLIES 10

Expert Contributor

Hi @grlzz ,

Have you verified the DB connection URL "jdbc:postgresql://[host]:[port]/[db] " - is that working outside of sqoop? (with any external JDBC based tool)

Have you used the same PostgreSQL driver which is supposed to be present under /var/lib/sqoop?

Also if the sqoop command is really "stuck", please check in another terminal window where is it stuck with jstack:

1. Get the process id of the sqoop command

ps -ef | grep sqoop

 

2. Collect the jstack output - with the same user as the sqoop import is running:

/usr/java/latest/bin/jstack $PID

 

This can help to understand what it is trying to do (for example it tries to connect to the database - but maybe the database is ssl enabled?)

Contributor

Hi,

 

thanks for your reply.

I haven't checked other JDBC tools yet, but I attach hereby the jstack output 

(by the way, it actually proceeded to another stage:

 

Heap
 PSYoungGen      total 147456K, used 76137K [0x000000071bb00000, 0x000000072db00000, 0x00000007c0000000)
  eden space 126464K, 60% used [0x000000071bb00000,0x000000072055a788,0x0000000723680000)
  from space 20992K, 0% used [0x0000000724b00000,0x0000000724b00000,0x0000000725f80000)
  to   space 20992K, 0% used [0x0000000723680000,0x0000000723680000,0x0000000724b00000)
 ParOldGen       total 192000K, used 9642K [0x00000005d3000000, 0x00000005deb80000, 0x000000071bb00000)
  object space 192000K, 5% used [0x00000005d3000000,0x00000005d396ab90,0x00000005deb80000)
 Metaspace       used 27157K, capacity 27412K, committed 27776K, reserved 1073152K
  class space    used 2984K, capacity 3100K, committed 3200K, reserved 1048576K

best,

Gr 

Contributor

...where it seems 'stuck' ... (the table is quite small, so I do not expect to take a lot of time -- I stopped after ~ 40 min)

Expert Contributor

Thanks for collecting and attaching the jstack. Yes, it confirms that Sqoop is trying to connect to the database, however that does not respond (the postgre driver is reading from the socket):

Thread 7658: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @BCI=0 (Interpreted frame)
...
 - org.postgresql.core.VisibleBufferedInputStream.readMore(int) @BCI=86, line=143 (Interpreted frame)
...
 - org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(org.postgresql.core.PGStream, java.lang.String, java.lang.String, java.util.Properties, org.postgresql.core.Logger) @BCI=10, line=376 (Interpreted frame)
 - org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(org.postgresql.util.HostSpec[], java.lang.String, java.lang.String, java.util.Properties, org.postgresql.core.Logger) @BCI=675, line=173 (Interpreted frame)
...
 - org.postgresql.Driver.makeConnection(java.lang.String, java.util.Properties) @BCI=18, line=393 (Interpreted frame)
 - org.postgresql.Driver.connect(java.lang.String, java.util.Properties) @BCI=165, line=267 (Interpreted frame)
 - java.sql.DriverManager.getConnection(java.lang.String, java.util.Properties, java.lang.Class) @BCI=171, line=664 (Interpreted frame)
 - java.sql.DriverManager.getConnection(java.lang.String, java.lang.String, java.lang.String) @BCI=37, line=247 (Interpreted frame)
 - org.apache.sqoop.manager.SqlManager.makeConnection() @BCI=182, line=888 (Interpreted frame)
 - org.apache.sqoop.manager.GenericJdbcManager.getConnection() @BCI=10, line=59 (Interpreted frame)
...
 - org.apache.sqoop.Sqoop.runSqoop(org.apache.sqoop.Sqoop, java.lang.String[]) @BCI=12, line=187 (Interpreted frame)
 

Please verify if your connection string is correct with some 3rd party tool - or ask your DBA what is a correct JDBC connection string.

Moderator

@grlzz Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks!


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

Contributor
Hi Diana,

no, this issue is not solved yet

best,
Gr

Moderator

Hi @grlzz 

Sorry that this has not being resolved,

Have you tried the fix above that @mszurap  provided?

Please verify if your connection string is correct with some 3rd party tool - or ask your DBA what is a correct JDBC connection string.

If you are still experiencing the issue, can you provide the information requested or additional details?

 

Thanks


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

Contributor
Hi,


yes, we are trying that, on 'our' side -- the point is that the team is basically me, and I have to ask around for help, with parteners related to our organization

best,
Gr

Expert Contributor

Hello @grlzz ,

As mentioned, to test the database connectivity you can use 3rd party tools, like SQuirreL:

http://squirrel-sql.sourceforge.net/

The following page describes how to connect to Postgres with Squirrel SQL Client:

https://www.cdata.com/kb/tech/postgresql-jdbc-squirrel-sql.rst

Please check what's the Postgres DB version and check the corresponding documentation, this can be a good start how to assemble the JDBC connection string:

https://jdbc.postgresql.org/documentation/80/connect.html 

If the connection string works in Squirrel, then use the same connection string in Sqoop too.

Hope this helps.

Best regards, Miklos

Contributor

Hi @mszurap , 

 

 

thanks for the tool

So, somehow the connection is now working , namely the command we tried to cloudera support, like:

`sqoop eval --connect ...`

or

 ` sqoop list-databases --connect` 

 

before running any sqoop command, we set up a ssh tunnel, with 

ssh -o ServerAliveInterval=60 -M -S mktemp -fN -L local_port:psql_fqdn:remote_p  user@host_IP 

 

(which I was doing also before, so not sure why now it works... 

I will mark it now as solved

 

thanks anyway,

Gr

; ;