Hi @Jmeks , I assume the partitioning column for this table is literally named as "date". Please note that "date" is a reserved word in Hive (see the docs for reference), so it is not recommended to use it in table names and column identifiers. If you really need to use it as an identifier, then encapsulate it within backticks. For example: ALTER TABLE customer_transaction_extract DROP IF EXISTS PARTITION (`date`="2019-11-13"); Hope this helps.  Best regards, Miklos ... View more

Hi, Do you have any general information about the types of tasks that the driver performs after a job completes? I do not have a comprehensive list of such tasks, this is just what we usually observe through cases and slowness reports. Of course there may be completely different tasks what the driver does - any custom spark code which does not involve parallel execution / data processing might run on the Spark driver side only, for example connecting to an external system through JDBC, or doing some computation (not with RDDs or DataFrames). ... View more

Hello @grlzz , As mentioned, to test the database connectivity you can use 3rd party tools, like SQuirreL: http://squirrel-sql.sourceforge.net/ The following page describes how to connect to Postgres with Squirrel SQL Client: https://www.cdata.com/kb/tech/postgresql-jdbc-squirrel-sql.rst Please check what's the Postgres DB version and check the corresponding documentation, this can be a good start how to assemble the JDBC connection string: https://jdbc.postgresql.org/documentation/80/connect.html  If the connection string works in Squirrel, then use the same connection string in Sqoop too. Hope this helps. Best regards, Miklos ... View more

Hi @nagacg ,  Usually this happens (emphasis also on the "intermittent" nature) when the BI tools like SAS connects to the Impala service (impala coordinator deamons) through a load balancer. With the Load-balancer the different requests are routed to different impala coordinator deamons and likely one of the impala coordinator deamon is in bad health. In this case not all the operations are failing - just some of them, as you've described. It is sometimes not obvious (from Cloudera Manager UI) that an impalad is unhealthy, need to verify all of them by connecting directly to them 1 by 1 with another tool, like impala-shell. (need to verify only the coordinators this way). I would suggest to involve Cloudera Support to assist you with this. ... View more

Thanks for collecting and attaching the jstack. Yes, it confirms that Sqoop is trying to connect to the database, however that does not respond (the postgre driver is reading from the socket): Thread 7658: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @BCI=0 (Interpreted frame) ... - org.postgresql.core.VisibleBufferedInputStream.readMore(int) @BCI=86, line=143 (Interpreted frame) ... - org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(org.postgresql.core.PGStream, java.lang.String, java.lang.String, java.util.Properties, org.postgresql.core.Logger) @BCI=10, line=376 (Interpreted frame) - org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(org.postgresql.util.HostSpec[], java.lang.String, java.lang.String, java.util.Properties, org.postgresql.core.Logger) @BCI=675, line=173 (Interpreted frame) ... - org.postgresql.Driver.makeConnection(java.lang.String, java.util.Properties) @BCI=18, line=393 (Interpreted frame) - org.postgresql.Driver.connect(java.lang.String, java.util.Properties) @BCI=165, line=267 (Interpreted frame) - java.sql.DriverManager.getConnection(java.lang.String, java.util.Properties, java.lang.Class) @BCI=171, line=664 (Interpreted frame) - java.sql.DriverManager.getConnection(java.lang.String, java.lang.String, java.lang.String) @BCI=37, line=247 (Interpreted frame) - org.apache.sqoop.manager.SqlManager.makeConnection() @BCI=182, line=888 (Interpreted frame) - org.apache.sqoop.manager.GenericJdbcManager.getConnection() @BCI=10, line=59 (Interpreted frame) ... - org.apache.sqoop.Sqoop.runSqoop(org.apache.sqoop.Sqoop, java.lang.String[]) @BCI=12, line=187 (Interpreted frame) Please verify if your connection string is correct with some 3rd party tool - or ask your DBA what is a correct JDBC connection string. ... View more

Hi @grlzz , Have you verified the DB connection URL "jdbc:postgresql://[host]:[port]/[db] " - is that working outside of sqoop? (with any external JDBC based tool) Have you used the same PostgreSQL driver which is supposed to be present under /var/lib/sqoop? Also if the sqoop command is really "stuck", please check in another terminal window where is it stuck with jstack: 1. Get the process id of the sqoop command ps -ef | grep sqoop   2. Collect the jstack output - with the same user as the sqoop import is running: /usr/java/latest/bin/jstack $PID   This can help to understand what it is trying to do (for example it tries to connect to the database - but maybe the database is ssl enabled?) ... View more

Hi @Kamil_Valko , The error message suggests that you are hitting IMPALA-9136 which is fixed in CDH 6.3.4 and in CDP 7.1.0 and newer. The regular Catalogd restart is likely due to some other problems, please file a support case to be able to review it in more detail. You can also review the role or the stdout/stderr logs of the Catalog daemon, maybe it shows an OutOfMemoryError - which can be common if the HMS / catalog metadata has increased recently - and the catalog heap is not adjusted. Best regards  Miklos ... View more