Re: Apache Flume required to be run in CDP environ...

Yosieam · ‎06-24-2022

Can you tell me where I should check that the process data in single JVM? the purpose of my spark job is writing the result to hive table and the oom comes when the job try to write all data into hive table

mszurap · ‎06-22-2022

HI @shivam0408 , Can you clarify what CDH/HDP/CDP version are you using and what is the datatype of the "DATETIME" column? What is the desired end result of this command? To drop all the partitions?

PCP2 · ‎06-17-2022

We dont have much information in logs. we refer stderr logs...PFA logs for the failed instance.

Uday_Singh2022 · ‎06-17-2022

Thanks Guys.

DataMan-HJ · ‎06-13-2022

Thanks @mszurap for your quick response. 🙂

luckes · ‎06-10-2022

I'm sure my source code file has UTF-8 encoding and I compiling the class by maven. When my SQL is set to ： upsert into user_info(id, name, address, email, insert_time) values (?,cast(? as string),cast(? as string),?,?) it's works, but the url parameter “UseNativeQuery=1” does't works!

mszurap · ‎06-08-2022

Hi Andrea, Great to see that it has been found now and thanks for marking the post as answered. All the best, Miklos

mszurap · ‎06-08-2022

Hi @tallamohan The direct usage of the Hive classes (CliSessionState, SessionState, Driver) in the provided code falls under the "Hive CLI" or "Hcat CLI" access, which is not supported in CDP: https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade/topics/hive-unsupported.html Please open a case on MyCloudera Support Portal to get that clarified. The recommended approach would be to use beeline and access the Hive service through HiveServer2. Best regards Miklos

mszurap · ‎06-08-2022

Please check which CDH version the cluster has. Cloudera ODBC Driver version 2.5.x in general is compatible with CDH 5.x, for CDH 6.x please use the latest 2.6.x version, please check our website / Downloads section. To further triage this: - Try to connect directly to a specific Impala coordinator host instead of the load-balancer - if load-balancer is used - Enable the driver side logging (check the driver's "Installation guide" how to enable it) which can give further clues. - Cross check that the Impala service is indeed SSL enabled, use different "openssl" commands to verify the certificate presented by the service, including the truststore used on the client side Hope this helps, Miklos Szurap, Customer Operations Engineer, Cloudera

Amn_468 · ‎06-05-2022

@mszurap Thanks for your detailed explanation this certainly helps.

mszurap · ‎06-03-2022

That is great, thank you for sharing the solution! Best regards Miklos

VidyaSargur · ‎06-02-2022

@roshanbi, Following up to check if the reply helped resolve your issue? If you are still experiencing the issue, can you provide the information @mszurap has requested?

mszurap · ‎05-19-2022

Hi! Sorry, but this seems some R specific usage problem in which I cannot help. What you can do is to enable DEBUG/TRACE level logging on the ODBC driver side (please check the ODBC Driver documentation how to do it), maybe there you can find further clues.

DianaTorres · ‎05-03-2022

@jarededrake Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks!

cjervis · ‎04-15-2022

@yagoaparecidoti If your issue is resolved, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Seaport · ‎04-14-2022

@mszurap Thanks for the response. I actually took the 2nd option you mentioned - ingesting it into a table which has only a single (string) column. But I am not sure whether it is the right approach. I appreciate the confirmation. Regards,

jarededrake · ‎04-11-2022

Sorry for the late reply thank you for you help with this, I will try to run it with Oozie instead. So it is not recommended at all to run spark jobs created with Java from Hue > Query > Editor > Java?

VidyaSargur · ‎03-29-2022

@Sayed016, Thanks for providing the details on the issue and credits to @mszurap! I have accepted your response as a solution, which should surely help others in the future.

mszurap · ‎03-25-2022

Hi Rama, yes, you can configure that in the " Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini" . OPSAPS-41615 is still open, in the future you can ask the status from any of your account team contacts. If you don't know who are those contacts, please ask/clarify that through the already open support case. Best regards, Miklos

VidyaSargur · ‎03-24-2022

@mhsyed Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

mszurap · ‎03-10-2022

Hi @M129 , the error message is not too descriptive. Can you please check the HiveMetaStore logs what is the complete error message - and reason for the failure? Thanks Miklos

mszurap · ‎02-24-2022

One more item to add to have a complete picture. SparkSQL does not support directly the usage of Hive ACID tables. For that in CDP you can use the Hive Warehouse Connector (HWC), please see: https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/integrating-hive-and-bi/topics/hive_hivewarehouseconnector_for_handling_apache_spark_data.html

Jmeks · ‎02-24-2022

Many Thanks. I finally managed to delete the partitions using your code

mszurap · ‎02-08-2022

Hi, Do you have any general information about the types of tasks that the driver performs after a job completes? I do not have a comprehensive list of such tasks, this is just what we usually observe through cases and slowness reports. Of course there may be completely different tasks what the driver does - any custom spark code which does not involve parallel execution / data processing might run on the Spark driver side only, for example connecting to an external system through JDBC, or doing some computation (not with RDDs or DataFrames).

grlzz · ‎02-03-2022

Hi @mszurap , thanks for the tool So, somehow the connection is now working , namely the command we tried to cloudera support, like: `sqoop eval --connect ...` or ` sqoop list-databases --connect` before running any sqoop command, we set up a ssh tunnel, with ssh -o ServerAliveInterval=60 -M -S mktemp -fN -L local_port:psql_fqdn:remote_p user@host_IP (which I was doing also before, so not sure why now it works... I will mark it now as solved thanks anyway, Gr

nagacg · ‎01-28-2022

Hi Mszurap, Thank you for your reply. yes I am worked with Cloudera team on this. Thank you .

mszurap · ‎01-27-2022

Hi @DataMo , I'm not an expert on this area, however the " Kerberos authentication" through a browser (SPNEGO) is a bit more complex than sending a username/password pair (HTTP Basic authentication). As I see https://issues.apache.org/jira/browse/NIFI-6250 - "Add Kerberos authentication support to InvokeHTTP Processor" is still open.

Kamil_Valko · ‎01-16-2022

Hello, problem was with OOO memory, after add some, it looks ok. Thank for advice. Best regards Kamil

DianaTorres · ‎01-07-2022

@cardozogp Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks

Donal_RC · ‎01-06-2022

This appears to be the cause. Switching to MS Edge avoided the HSTS feature in Chrome. Will look at enabling HTTPS on all UIs though since that is the correct solution in the longterm (spark.ssl.historyServer.enabled option is currently set to false ).

Online	Offline
Last Visited	‎06-24-2022 04:16 AM

Date Registered	‎11-04-2015 11:53 PM
Last Visited	‎06-24-2022 04:16 AM
Total Messages Posted	168
Total Kudos Received	21

Re: Apache Flume required to be run in CDP environ...

Re: Case-insensitive comparison Hive

Re: HDFS block count does not decrease after delet...

Re: Impala Lock contention error while running Ref...

Re: Creating an Impala External Table from fixed w...

Re: Error spark job input is too large to fit in a...

Re: error in dropping partition in hive

Re: Oozie job failed due to below error.

Re: Apache Flume required to be run in CDP environ...

Re: Case-insensitive comparison Hive

Re: impala jdbc doesn't work for preparestatement ...

Re: HDFS block count does not decrease after delet...

Re: hive load data query is failing with following...

Re: [Cloudera][ThriftExtension] (6) Error occurred...

Re: Impala Lock contention error while running Ref...

Re: NodeManager fails to start

Re: update with alias not working

Re: R client connections fails with CDP Impala

Re: Oozie work flow not finding main class

Re: nifi manages the supervisor.conf file?

Re: Creating an Impala External Table from fixed w...

Re: Getting "Caused by: java.lang.NoClassDefFoundE...

Re: Spark job through Oozie is failing with - sche...

Re: Preserve Source client IP address when use Loa...

Re: 404 when Adding MKL wrapper parcel

Re: Error while Creating an external table in hive...

Re: Spark unsupported fearutes in CDP

Re: Delete partition Hive error

Re: Spark-Streaming stalls at regular intervals

Re: sqoop stuck at No connection paramenters speci...

Re: SAS -Impala connection error : Impala Thrift ...

Re: Kudu Metrics - SPENGO Authentication (kerberiz...

Re: Impala Catalog Server restarted with IllegalSt...

Re: Bind in Sqoop querys on import command

Re: Unable to connect to Spark History Server Web ...