Member since
06-02-2020
331
Posts
64
Kudos Received
49
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1028 | 07-11-2024 01:55 AM | |
2951 | 07-09-2024 11:18 PM | |
2538 | 07-09-2024 04:26 AM | |
1926 | 07-09-2024 03:38 AM | |
2195 | 06-05-2024 02:03 AM |
07-16-2024
01:48 AM
Hi @ayukus0705 The nf_hh0 column data appears to be stored in a format other than string. When you try to read this data using a string data type, it may lead to above issue. To resolve this issue, you can either change the data type of the column to match the actual data format, or convert the data to a string format.
... View more
07-11-2024
05:23 AM
Hi @saifikhan Just by providing ArithmeticException, we cant provide any solution. This can be occurred due to your code or apache spark code. Check the exception stack-trace and fix the issue if issue is from your code.
... View more
07-11-2024
01:55 AM
Hi @MoatazNader Yes, you can create/update/delete the iceberg table data using Impala in CDP 7.1.9 Creating table: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/iceberg-how-to/topics/iceberg-table-creation.html Insert Data: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/iceberg-how-to/topics/iceberg-insert-table-data.html Update/Delete: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/iceberg-how-to/topics/iceberg-best-practice-row-modifications.html Reference: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/iceberg-how-to/topics/iceberg-table-creation.html
... View more
07-09-2024
11:49 PM
1 Kudo
Introduction
The Spark and Iceberg Supportability Matrix provides comprehensive information regarding the compatibility and supportability of Spark and Iceberg versions with various operating systems, frameworks, and dependencies.
Apache Iceberg History
The development of Iceberg was started in 2017 by Netflix. The project was open-sourced and donated to the Apache Software Foundation in November 2018. In May 2020, the Iceberg project graduated to become a top-level Apache project.
Apache Iceberg 0.7.0 was released on Oct 26, 2019 (Incubating)
Apache Iceberg 0.8.0 was released on May 7, 2020 (Incubating).
Apache Iceberg 0.9.0 was released on Jul 14, 2020.
Apache Iceberg 0.9.1 was released on Aug 11, 2020.
Apache Iceberg 0.10.0 was released on Nov 12, 2020.
Apache Iceberg 0.11.0 was released on Jan 27, 2021.
Apache Iceberg 0.11.1 was released on Apr 3, 2021.
Apache Iceberg 0.12.0 was released on August 15, 2021.
Apache Iceberg 0.12.1 was released on November 8th, 2021.
Apache Iceberg 0.13.0 was released on February 4th, 2022.
Apache Iceberg 0.13.1 was released on February 14th, 2022.
Apache Iceberg 0.13.2 was released on June 15th, 2022.
Apache Iceberg 0.14.0 was released on July 16 2022.
Apache Iceberg 0.14.1 was released on Sep 12, 2022.
Apache Iceberg 1.0.0 was released on Nov 3, 2022.
Apache Iceberg 1.1.0 was released on November 28th, 2022.
Apache Iceberg 1.2.0 was released on March 20th, 2023.
Apache Iceberg 1.2.1 was released on April 11th, 2023.
Apache Iceberg 1.3.0 was released on May 30th, 2023.
Apache Iceberg 1.3.1 was released on July 25, 2023.
Apache Iceberg 1.4.0 was released on October 4, 2023.
Apache Iceberg 1.4.1 was released on October 23, 2023.
Apache Iceberg 1.4.2 was released on November 2, 2023.
Apache Iceberg 1.4.3 was released on December 27, 2023.
Apache Iceberg 1.5.0 was released on March 11, 2024.
Apache Iceberg 1.5.1 was released on April 25, 2024.
Apache Iceberg 1.5.2 was released on May 9, 2024.
Apache Spark and Iceberg Supportability Matrix Table
The following table explains the Iceberg Version Release Date Status Default Spark Version Supported Spark Version(s):
Iceberg Version
Release Date
Status
Default Spark Version
Supported Spark Version(s)
0.7.0
Oct 26, 2019
Incubating
2.4
2.4
0.8.0
May 07, 2020
Incubating
2.4
2.4
0.9.0
Jul 14, 2020
2.4,3.0
0.9.1
Aug 11, 2020
2.4,3.0
0.10.0
Nov 12, 2020
2.4,3.0
0.11.0
Jan 27, 2021
2.4,3.0
0.11.1
Apr 03, 2021
2.4,3.0
0.12.0
Aug 15, 2021
2.4,3.0,3.1
0.12.1
Nov 08, 2021
2.4,3.0,3.1
0.13.0
Feb 04, 2022
3.2
2.4,3.0,3.1,3.2
0.13.1
Feb 14, 2022
3.2
2.4,3.0,3.1,3.2
0.13.2
Jun 15, 2022
3.2
2.4,3.0,3.1,3.2
0.14.0
Jul 17, 2022
3.3
2.4,3.0,3.1,3.2,3.3
0.14.1
Sep 12, 2022
3.3
2.4,3.0,3.1,3.2,3.3
1.0.0
Nov 03, 2022
3.3
2.4,3.0,3.1,3.2,3.3
1.1.0
Nov 28, 2022
3.3
2.4,3.1,3.2,3.3
1.2.0
Mar 20, 2023
3.3
2.4,3.1,3.2,3.3
1.2.1
Apr 11, 2023
3.3
2.4,3.1,3.2,3.3
1.3.0
May 30, 2023
3.4
3.1,3.2,3.3,3.4
1.3.1
Jul 25, 2023
3.4
3.1,3.2,3.3,3.4
1.4.0
Oct 04, 2023
3.5
3.2,3.3,3.4,3.5
1.4.1
Oct 23, 2023
3.5
3.2,3.3,3.4,3.5
1.4.2
Nov 02, 2023
3.5
3.2,3.3,3.4,3.5
1.4.3
Dec 27, 2023
3.5
3.2,3.3,3.4,3.5
1.5.0
Mar 11, 2024
3.5
3.3,3.4,3.5
1.5.1
Apr 25, 2024
3.5
3.3,3.4,3.5
1.5.2
May 09, 2024
3.5
3.3,3.4,3.5
References
Iceberg Releases
Github Iceberg
Thank you for taking the time to read this article. We hope you found it informative and helpful in enhancing your understanding of the topic. If you have any questions or feedback, please feel free to contact me. Remember, your support motivates us to continue creating valuable content. If this article helped you, please consider giving it a like and providing a kudos. We appreciate your support!
... View more
07-09-2024
11:35 PM
1 Kudo
Apache Spark 3.5.1 will support Java 8/11/17 and Scala Binary Version 2.12/2.13. If you want to use Scala Binary Version 2.12 then recommended Scala version is 2.12.18 Coming to your questions: 1. With out providing the Exception stack trace details difficult to provide a solution. 2. Reason could be in your application code while creating spark session maybe you have hard coded client mode. 3. To use AWS, you need to download hadoop-aws jars files and pass it in spark submit command. References: 1. https://spark.apache.org/docs/3.5.1/index.html 2. https://github.com/apache/spark/tree/v3.5.1
... View more
07-09-2024
11:26 PM
1 Kudo
Based on event log files, you need to adjust Spark History Server settings. Could you please check SHS cleanup is enabled or not. If you enable spark automatically it clean the old event log files. To load larger event log files, you need to adjust the DAEMON_MEMORY_SIZE. You can refer the following article to adjust the SHS parameters: https://spark.apache.org/docs/latest/monitoring.html#spark-history-server-configuration-options
... View more
07-09-2024
11:18 PM
1 Kudo
Issue looks like you are hitting CDPD-66940 jira and try to upgrade your CDS3 parcel to latest and it will resolve your issue.
... View more
07-09-2024
06:03 AM
Hi @ren97 We don't have any setup to test your scenario. We can access iceberg table using impala, so i asked you to test. We have internal tools like Hue, Zeppelin by using these you can access Iceberg table and visualise your data.
... View more
07-09-2024
04:26 AM
1 Kudo
Hi @ren97 Hive LLAP mainly used for accessing the Hive tables (Managed and External) and not for Iceberg tables. I don't think we are supported to access the Iceberg tables using PowerBI directly. One thing you can try configure Impala and from Impala you create iceberg table and try to access it.
... View more
07-09-2024
03:38 AM
1 Kudo
Hi @Ismail_A Don't add Spark libraries and its dependent jars as fat jar to your Spark application (MyApplication.jar). If you are using maven build tool you can specify the scope as provided. <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.12</artifactId> <version>3.4.0</version> <scope>provided</scope> </dependency> If you are using gradle you can try similar like below dependencies {
compileOnly group: 'org.apache.spark', name: 'spark-sql_2.12', version: '3.4.0'
} With the above steps it is not resolved your issue, then run your code by launching spark-shell/pyspark and see it is working or not.
... View more