About mszurap

mszurap · ‎01-27-2022

Hi @grlzz , Have you verified the DB connection URL "jdbc:postgresql://[host]:[port]/[db] " - is that working outside of sqoop? (with any external JDBC based tool) Have you used the same PostgreSQL driver which is supposed to be present under /var/lib/sqoop? Also if the sqoop command is really "stuck", please check in another terminal window where is it stuck with jstack: 1. Get the process id of the sqoop command ps -ef | grep sqoop 2. Collect the jstack output - with the same user as the sqoop import is running: /usr/java/latest/bin/jstack $PID This can help to understand what it is trying to do (for example it tries to connect to the database - but maybe the database is ssl enabled?)

mszurap · ‎01-27-2022

Hi @DataMo , I'm not an expert on this area, however the "Kerberos authentication" through a browser (SPNEGO) is a bit more complex than sending a username/password pair (HTTP Basic authentication). As I see https://issues.apache.org/jira/browse/NIFI-6250 - "Add Kerberos authentication support to InvokeHTTP Processor" is still open.

mszurap · ‎01-11-2022

Hi @Kamil_Valko , The error message suggests that you are hitting IMPALA-9136 which is fixed in CDH 6.3.4 and in CDP 7.1.0 and newer. The regular Catalogd restart is likely due to some other problems, please file a support case to be able to review it in more detail. You can also review the role or the stdout/stderr logs of the Catalog daemon, maybe it shows an OutOfMemoryError - which can be common if the HMS / catalog metadata has increased recently - and the catalog heap is not adjusted. Best regards Miklos

mszurap · ‎01-06-2022

Hi @Donal_RC , Please check if the Spark History Server UI is TLS/SSL enabled or not. I suppose it is not, usually SHS WebUI is available only on http. (the log snippets suggest that it is plain http) Chrome is automatically switching to https for some company domains, this feature is called HSTS - HTTP Strict Transport Security. Please check the addresss bar in Chrome, you should see that the URL starts with https. If you have Internet Explorer, that doesn't do this, you can try there. Alternatively you can: - modify the URL and access the SHS with the host's IP address - or enable TLS/SSL for the SHS too. Hope this helps, Miklos

mszurap · ‎01-04-2022

Hi @cardozogp , With sqoop import (DB -> HDFS) sqoop will submit just a couple of (or in range of max 100s) of queries to the database: - 1 query to get the table schema - a couple of queries to determine the split ranges (see "split-by") - and every mapper starts 1 select statement with their own split key range. The mappers from then on just call "getNext" on the result sets to fetch the next batch of results from the DB. With sqoop export (HDFS->DB) however the mappers are calling separate insert statements for every "batch" (~100 s of records), so there may be millions of insert statements submitted, in that case this indeed has an impact. As there are really just a few queries in play with sqoop import, the usage of prepared statements does not have a significant benefit. Let us know if your DBA team has some other insights which we may not be aware of. Best regards Miklos

mszurap · ‎01-03-2022

Great, thanks for sharing your solution!

mszurap · ‎12-30-2021

Hi Saurabh, The "Establishing SSL connection..." message already suggests that the MySQL client (in this case Cloudera Manager code "DbCommandExecutor") is using TLS/SSL coonection to the MySQL database. The last "caused by" tells the reason for the failure: Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) This happens when - The MySQL client does not allow a specific cipher suite (it has some restrictions) - and the MySQL server does not support any of the cipher suites which the client allows. Please check what settings are on the MySQL server side: mysql --ssl-ca=<path_to/truststore.pem> -uroot -p -e "SHOW GLOBAL VARIABLES LIKE '%ssl%';STATUS;" and check the CM server side JDK configuration settings in: /usr/java/latest/jre/lib/security/java.security Look for ProtocolsWhiteList, CipherWhiteList and CipherBlackList settings.

mszurap · ‎11-25-2021

Hi, This use case seems to be similar to queries which have a filter "WHERE 1=0" - those also do not return any data. See for example such a discussion why that is needed/used in some places: https://stackoverflow.com/questions/9140606/why-would-you-use-where-1-0-statement-in-sql My best guess would be that the application calling such queries needs the schema of the result set in advance - for example to create and render the layout (columns) or autofilters or to prepare another subsequent query properly (for example it has to do some decisions - which filters the next query needs to contain, maybe they are different for a string and for a date column, etc) Best regards Miklos Miklos Szurap Customer Operations Engineer, Cloudera

mszurap · ‎11-22-2021

Hi, the complex MAP data type is supported, however it may not be as usable as other simple types. Please see: https://docs.cloudera.com/runtime/7.2.10/impala-sql-reference/topics/impala-map.html and https://docs.cloudera.com/runtime/7.2.10/impala-sql-reference/topics/impala-complex-types.html#complex_types/complex_sample_schema There is a similar "GET_JSON_OBJECT" built-in UDF, which may help you - depending what you need to achieve, please see: https://docs.cloudera.com/runtime/7.2.10/impala-sql-reference/topics/impala-misc-functions.html#misc_functions__get_json_object

mszurap · ‎11-15-2021

You've mentioned that even if you intentionally "break" the workflow with changing the class name it still does not throw an error message. Can you check what's in the workflow.xml when you change the workflow? This would help to decide is Hue not saving the workflow properly or is the logging broken only. Is the workflow successful even with a wrong classname?

Online	Offline
Last Visited	‎04-24-2026 02:08 AM

Member Since	‎11-04-2015 11:53 PM
Last Visited	‎04-24-2026 02:08 AM
Posts	261
Kudos received	44

Cloudera Community

Re: Hive fails to start with "Caused by: java.lang...

Re: The heap memory usage of NameNode is much high...

Re: Hue and Sqoop white spaces in query

Re: straight SELECT and SELECT via CTE produce dif...

Re: Best practices for partition tables in Impala ...

Re: sqoop stuck at No connection paramenters speci...

Re: Kudu Metrics - SPENGO Authentication (kerberiz...

Re: Impala Catalog Server restarted with IllegalSt...

Re: Unable to connect to Spark History Server Web ...

Re: Bind in Sqoop querys on import command

Re: scm_prepare_database.sh, ERROR 4

Re: scm_prepare_database.sh, ERROR 4

Re: Impala query with limit 0

Re: Is there any way to cast string to map?

Re: Oozie Java Action doesnt do anything