Member since
09-16-2021
357
Posts
53
Kudos Received
28
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
86 | 02-04-2025 05:55 AM | |
502 | 11-22-2024 05:29 AM | |
247 | 11-15-2024 06:38 AM | |
527 | 11-13-2024 07:12 AM | |
548 | 11-10-2024 11:19 PM |
10-10-2024
04:28 PM
1 Kudo
@IanWilloughby If you are still experiencing the issue, can you provide the information @ggandharan has requested? Thanks.
... View more
10-07-2024
12:29 AM
1 Kudo
We recommend utilizing CDW for Kubernetes on Hive. Based on the description, it seems that you are currently using the apache-hive library. In the upstream (Apache), images have already been pushed to Docker Hub, so you can utilize the same. I have attached the relevant documents for your reference. https://hive.apache.org/development/quickstart/ https://docs.cloudera.com/data-warehouse/cloud/overview/topics/dw-service-architecture.html
... View more
09-18-2024
09:19 PM
1 Kudo
This solution worked for eliminating error , but data is not being fetched from table. empty data frame showing.
... View more
09-18-2024
01:22 AM
1 Kudo
@zhuodongLi, Did the responses help resolve your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.
... View more
09-11-2024
08:47 AM
@ggangadharan thanks for your reply. Yes, as soon spark sees NUMBER data type in oralce it convert the df datatype to decimal(38,10) then when precision value in oracle column contains >30 spark cant accommodate it as it only allows 28 max digits if decimal(38,10) hence getting this issue. yeah as you said the probable solution is to cast it as string Type.
... View more
09-05-2024
04:53 AM
1 Kudo
@Lorenzo The issue seems to be related to HIVE-27191 where some mhl_txnids do not exist in TXNS,completed_txn_components txn_components table but they are still present in min_history_level table, as a result, the cleaner gets blocked and many entries are stuck in the ready-for-cleaning state. To confirm that collect the output of below query SELECT MHL_TXNID FROM HIVE.MIN_HISTORY_LEVEL WHERE MHL_MIN_OPEN_TXNID = (SELECT MIN(MHL_MIN_OPEN_TXNID) FROM HIVE.MIN_HISTORY_LEVEL); Once we get the output of the above query check if those txn ids are there in TXNS,completed_txn_components txn_components tables using below commands. select * from txn_components where tc_txnid IN (MHL_TXNID ); select * from completed_txn_components where ctc_txnid IN (MHL_TXNID); select * from TXNS where ctc_txnid IN (MHL_TXNID); If we got 0 results from the above queries this confirms that the MHL_TXNIDs we got above are orphans and we need to remove them in order to unblock the cleaner. delete from MIN_HISTORY_LEVEL where MHL_TXNID=13422; --(repeat for all) Hope this helps you in resolving the issue
... View more
09-05-2024
01:36 AM
1 Kudo
Are you using the same user account to connect via ODBC which you used to log in to Hue? Please verify that.
... View more
09-04-2024
05:35 AM
If setting the proper queue name resolves the problem, it is possible that the query may have been submitted in the default queue, where it competes for resources with other queries and fails due to a timeout error
... View more
08-28-2024
01:40 AM
Unfortunately, it is not possible to change the Application-Name of an already started Application Master in Apache Hadoop YARN. The Application-Name is set when the application is submitted and cannot be modified during runtime. The Application-Name is typically specified as a parameter when submitting the application using the spark-submit command or the YARN REST API. Once the application is started, the Application-Name is fixed and cannot be changed. If you need to change the Application-Name, you will need to stop the existing application and submit a new one with the desired name.
... View more
08-28-2024
12:40 AM
1 Kudo
When writing to a statically partitioned table using HWC, the following query is internally fired to Hive through JDBC after writing data to a temporary location: Spark write statement: df.write.format(HIVE_WAREHOUSE_CONNECTOR).mode("append").option("partition", "c1='val1',c2='val2'").option("table", "t1").save(); HWC internal query: LOAD DATA INPATH '<spark.datasource.hive.warehouse.load.staging.dir>' [OVERWRITE] INTO TABLE db.t1 PARTITION (c1='val1',c2='val2'); During static partitioning, the partition information is known during compile time, resulting in the creation of a staging directory in the partition directory. On the other hand, when writing to a dynamically partitioned table using HWC, the following query is internally fired to Hive through JDBC after writing data to a temporary location: Spark write statement: df.write.format(HIVE_WAREHOUSE_CONNECTOR).mode("append").option("partition", "c1='val1',c2").option("table", "t1").save(); HWC internal query: CREATE TEMPORARY EXTERNAL TABLE db.job_id_table(cols....) STORED AS ORC LOCATION '<spark.datasource.hive.warehouse.load.staging.dir>';
INSERT INTO TABLE t1 PARTITION (c1='val1',c2) SELECT <cols> FROM db.job_id_table; During dynamic partitioning, the partition information is known during runtime, hence the staging directory is created at the table level. Once the DAG is completed, the MOVE TASK will move the files to the respective partitions.
... View more