sqoop stuck at No connection paramenters specified. Using regular API for making connection.

[#1]

Hi,

I am trying to run this import command:

sqoop import --connect jdbc:postgresql://[host]:[port]/[db] --username [user] --password-file [path_to_pass] -table [table] --hive-import -hive-drop-import-delims --as-parquetfile --parquet-configurator-implementation hadoop --hive-table district_startdates_test --target-dir [target_with_db] -m 1 --hs2-url="jdbc:hive2://[cdp]/;ssl=true;transportMode=http;httpPath=[path_to_proxy];user=[user];password=[cluster_passw]" --verbose

but it gets stuck at  

 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.

...any idea?

thanks

[#2]

Hi @grlzz ,

Have you verified the DB connection URL "jdbc:postgresql://[host]:[port]/[db] " - is that working outside of sqoop? (with any external JDBC based tool)

Have you used the same PostgreSQL driver which is supposed to be present under /var/lib/sqoop?

Also if the sqoop command is really "stuck", please check in another terminal window where is it stuck with jstack:

1. Get the process id of the sqoop command

ps -ef | grep sqoop

2. Collect the jstack output - with the same user as the sqoop import is running:

/usr/java/latest/bin/jstack $PID

This can help to understand what it is trying to do (for example it tries to connect to the database - but maybe the database is ssl enabled?)

[#3]

Hi,

thanks for your reply.

I haven't checked other JDBC tools yet, but I attach hereby the jstack output 

(by the way, it actually proceeded to another stage:

Heap
 PSYoungGen      total 147456K, used 76137K [0x000000071bb00000, 0x000000072db00000, 0x00000007c0000000)
  eden space 126464K, 60% used [0x000000071bb00000,0x000000072055a788,0x0000000723680000)
  from space 20992K, 0% used [0x0000000724b00000,0x0000000724b00000,0x0000000725f80000)
  to   space 20992K, 0% used [0x0000000723680000,0x0000000723680000,0x0000000724b00000)
 ParOldGen       total 192000K, used 9642K [0x00000005d3000000, 0x00000005deb80000, 0x000000071bb00000)
  object space 192000K, 5% used [0x00000005d3000000,0x00000005d396ab90,0x00000005deb80000)
 Metaspace       used 27157K, capacity 27412K, committed 27776K, reserved 1073152K
  class space    used 2984K, capacity 3100K, committed 3200K, reserved 1048576K

best,

Gr 

[#4]

...where it seems 'stuck' ... (the table is quite small, so I do not expect to take a lot of time -- I stopped after ~ 40 min)

[#5]

Thanks for collecting and attaching the jstack. Yes, it confirms that Sqoop is trying to connect to the database, however that does not respond (the postgre driver is reading from the socket):

Thread 7658: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @BCI=0 (Interpreted frame)
...
 - org.postgresql.core.VisibleBufferedInputStream.readMore(int) @BCI=86, line=143 (Interpreted frame)
...
 - org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(org.postgresql.core.PGStream, java.lang.String, java.lang.String, java.util.Properties, org.postgresql.core.Logger) @BCI=10, line=376 (Interpreted frame)
 - org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(org.postgresql.util.HostSpec[], java.lang.String, java.lang.String, java.util.Properties, org.postgresql.core.Logger) @BCI=675, line=173 (Interpreted frame)
...
 - org.postgresql.Driver.makeConnection(java.lang.String, java.util.Properties) @BCI=18, line=393 (Interpreted frame)
 - org.postgresql.Driver.connect(java.lang.String, java.util.Properties) @BCI=165, line=267 (Interpreted frame)
 - java.sql.DriverManager.getConnection(java.lang.String, java.util.Properties, java.lang.Class) @BCI=171, line=664 (Interpreted frame)
 - java.sql.DriverManager.getConnection(java.lang.String, java.lang.String, java.lang.String) @BCI=37, line=247 (Interpreted frame)
 - org.apache.sqoop.manager.SqlManager.makeConnection() @BCI=182, line=888 (Interpreted frame)
 - org.apache.sqoop.manager.GenericJdbcManager.getConnection() @BCI=10, line=59 (Interpreted frame)
...
 - org.apache.sqoop.Sqoop.runSqoop(org.apache.sqoop.Sqoop, java.lang.String[]) @BCI=12, line=187 (Interpreted frame)
 

Please verify if your connection string is correct with some 3rd party tool - or ask your DBA what is a correct JDBC connection string.

[#6]

@grlzz Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks!

[#7]

Hi Diana,

no, this issue is not solved yet

best,
Gr

[#8]

Hi @grlzz 

Sorry that this has not being resolved,

Have you tried the fix above that @mszurap  provided?

Please verify if your connection string is correct with some 3rd party tool - or ask your DBA what is a correct JDBC connection string.

If you are still experiencing the issue, can you provide the information requested or additional details?

Thanks

[#9]

Hi,


yes, we are trying that, on 'our' side -- the point is that the team is basically me, and I have to ask around for help, with parteners related to our organization

best,
Gr

[#10]

Hello @grlzz ,

As mentioned, to test the database connectivity you can use 3rd party tools, like SQuirreL:

http://squirrel-sql.sourceforge.net/

The following page describes how to connect to Postgres with Squirrel SQL Client:

https://www.cdata.com/kb/tech/postgresql-jdbc-squirrel-sql.rst

Please check what's the Postgres DB version and check the corresponding documentation, this can be a good start how to assemble the JDBC connection string:

https://jdbc.postgresql.org/documentation/80/connect.html 

If the connection string works in Squirrel, then use the same connection string in Sqoop too.

Hope this helps.

Best regards, Miklos