question Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection. in Support Questions

sqoop stuck at No connection paramenters specified. Using regular API for making connection.

grlzz — Wed, 26 Jan 2022 12:34:10 GMT

Hi,

I am trying to run this import command:

sqoop import --connect jdbc:postgresql://[host]:[port]/[db] --username [user] --password-file [path_to_pass] -table [table] --hive-import -hive-drop-import-delims --as-parquetfile --parquet-configurator-implementation hadoop --hive-table district_startdates_test --target-dir [target_with_db] -m 1 --hs2-url="jdbc:hive2://[cdp]/;ssl=true;transportMode=http;httpPath=[path_to_proxy];user=[user];password=[cluster_passw]" --verbose

but it gets stuck at

DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.

...any idea?

thanks

Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection.

mszurap — Thu, 27 Jan 2022 13:04:13 GMT

Hi @grlzz ,

Have you verified the DB connection URL "jdbc:postgresql://[host]:[port]/[db] " - is that working outside of sqoop? (with any external JDBC based tool)

Have you used the same PostgreSQL driver which is supposed to be present under /var/lib/sqoop?

Also if the sqoop command is really "stuck", please check in another terminal window where is it stuck with jstack:

1. Get the process id of the sqoop command

ps -ef | grep sqoop

2. Collect the jstack output - with the same user as the sqoop import is running:

/usr/java/latest/bin/jstack $PID

This can help to understand what it is trying to do (for example it tries to connect to the database - but maybe the database is ssl enabled?)

Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection.

grlzz — Thu, 27 Jan 2022 15:19:15 GMT

Hi,

thanks for your reply.

I haven't checked other JDBC tools yet, but I attach hereby the jstack output

(by the way, it actually proceeded to another stage:

Heap PSYoungGen total 147456K, used 76137K [0x000000071bb00000, 0x000000072db00000, 0x00000007c0000000) eden space 126464K, 60% used [0x000000071bb00000,0x000000072055a788,0x0000000723680000) from space 20992K, 0% used [0x0000000724b00000,0x0000000724b00000,0x0000000725f80000) to space 20992K, 0% used [0x0000000723680000,0x0000000723680000,0x0000000724b00000) ParOldGen total 192000K, used 9642K [0x00000005d3000000, 0x00000005deb80000, 0x000000071bb00000) object space 192000K, 5% used [0x00000005d3000000,0x00000005d396ab90,0x00000005deb80000) Metaspace used 27157K, capacity 27412K, committed 27776K, reserved 1073152K class space used 2984K, capacity 3100K, committed 3200K, reserved 1048576K

best,

Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection.

grlzz — Thu, 27 Jan 2022 15:21:06 GMT

...where it seems 'stuck' ... (the table is quite small, so I do not expect to take a lot of time -- I stopped after ~ 40 min)

Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection.

mszurap — Thu, 27 Jan 2022 16:37:45 GMT

Thanks for collecting and attaching the jstack. Yes, it confirms that Sqoop is trying to connect to the database, however that does not respond (the postgre driver is reading from the socket):

Thread 7658: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @BCI=0 (Interpreted frame) ... - org.postgresql.core.VisibleBufferedInputStream.readMore(int) @BCI=86, line=143 (Interpreted frame) ... - org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(org.postgresql.core.PGStream, java.lang.String, java.lang.String, java.util.Properties, org.postgresql.core.Logger) @BCI=10, line=376 (Interpreted frame) - org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(org.postgresql.util.HostSpec[], java.lang.String, java.lang.String, java.util.Properties, org.postgresql.core.Logger) @BCI=675, line=173 (Interpreted frame) ... - org.postgresql.Driver.makeConnection(java.lang.String, java.util.Properties) @BCI=18, line=393 (Interpreted frame) - org.postgresql.Driver.connect(java.lang.String, java.util.Properties) @BCI=165, line=267 (Interpreted frame) - java.sql.DriverManager.getConnection(java.lang.String, java.util.Properties, java.lang.Class) @BCI=171, line=664 (Interpreted frame) - java.sql.DriverManager.getConnection(java.lang.String, java.lang.String, java.lang.String) @BCI=37, line=247 (Interpreted frame) - org.apache.sqoop.manager.SqlManager.makeConnection() @BCI=182, line=888 (Interpreted frame) - org.apache.sqoop.manager.GenericJdbcManager.getConnection() @BCI=10, line=59 (Interpreted frame) ... - org.apache.sqoop.Sqoop.runSqoop(org.apache.sqoop.Sqoop, java.lang.String[]) @BCI=12, line=187 (Interpreted frame)

Please verify if your connection string is correct with some 3rd party tool - or ask your DBA what is a correct JDBC connection string.

Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection.

DianaTorres — Mon, 31 Jan 2022 19:40:55 GMT

@grlzz Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks!

Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection.

grlzz — Tue, 01 Feb 2022 10:36:22 GMT

Hi Diana,

no, this issue is not solved yet

best,
Gr

Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection.

DianaTorres — Wed, 02 Feb 2022 00:30:05 GMT

Hi @grlzz

Sorry that this has not being resolved,

Have you tried the fix above that @mszurap provided?

Please verify if your connection string is correct with some 3rd party tool - or ask your DBA what is a correct JDBC connection string.

If you are still experiencing the issue, can you provide the information requested or additional details?

Thanks

Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection.

grlzz — Wed, 02 Feb 2022 08:28:22 GMT

Hi,

yes, we are trying that, on 'our' side -- the point is that the team is basically me, and I have to ask around for help, with parteners related to our organization

best,
Gr

Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection.

mszurap — Thu, 03 Feb 2022 09:25:03 GMT

Hello @grlzz ,

As mentioned, to test the database connectivity you can use 3rd party tools, like SQuirreL:

http://squirrel-sql.sourceforge.net/

The following page describes how to connect to Postgres with Squirrel SQL Client:

https://www.cdata.com/kb/tech/postgresql-jdbc-squirrel-sql.rst

Please check what's the Postgres DB version and check the corresponding documentation, this can be a good start how to assemble the JDBC connection string:

https://jdbc.postgresql.org/documentation/80/connect.html

If the connection string works in Squirrel, then use the same connection string in Sqoop too.

Hope this helps.

Best regards, Miklos

Re: sqoop stuck at No connection paramenters specified. Using regular API for making connection.

grlzz — Thu, 03 Feb 2022 09:43:35 GMT

Hi @mszurap ,

thanks for the tool

So, somehow the connection is now working , namely the command we tried to cloudera support, like:

`sqoop eval --connect ...`

` sqoop list-databases --connect`

before running any sqoop command, we set up a ssh tunnel, with

ssh -o ServerAliveInterval=60 -M -S mktemp -fN -L local_port:psql_fqdn:remote_p user@host_IP

(which I was doing also before, so not sure why now it works...

I will mark it now as solved

thanks anyway,