Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

unable to view the content of acid table in spark and hive

SOLVED Go to solution
Highlighted

unable to view the content of acid table in spark and hive

New Contributor

i have created acid orc partition table in hive. By using spark i insert the data to that table . in table pointing location its showing data like

/warehouse/tablespace/managed/hive/testdb.db/table__2/full_date=2019-07-17/delta_0000000_0000000_0000/ 0000_0 .


when i query the table it showing only column names can any one give me a solution

1 ACCEPTED SOLUTION

Accepted Solutions

Re: unable to view the content of acid table in spark and hive

Super Guru

@Duraisankar S

You can run Major compaction on the partition in Hive, after the major compaction is done base-**** directory will be created.

Then spark able to read the specific partition which have base-*** directories in it.

But spark not able to read delta directories as there is an open Jira [SPARK-15348] about spark is not able to read acid table.


I think starting from HDP-3.X HiveWareHouseConnector is able to support to read HiveAcid tables.
5 REPLIES 5

Re: unable to view the content of acid table in spark and hive

Super Guru

@Duraisankar S

You can run Major compaction on the partition in Hive, after the major compaction is done base-**** directory will be created.

Then spark able to read the specific partition which have base-*** directories in it.

But spark not able to read delta directories as there is an open Jira [SPARK-15348] about spark is not able to read acid table.


I think starting from HDP-3.X HiveWareHouseConnector is able to support to read HiveAcid tables.

Re: unable to view the content of acid table in spark and hive

New Contributor

thank u 4 the info

Re: unable to view the content of acid table in spark and hive

Super Guru

@Duraisankar S

If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly :-)

Re: unable to view the content of acid table in spark and hive

New Contributor

@shu


hive warehouse connector support for acid table is available in HDP-3.0 r do i need to upgrade HDP-3.0 to HDP-3.1



Re: unable to view the content of acid table in spark and hive

New Contributor

@Shu hive warehouse connector support for acid table is available in HDP-3.0 r do i need to upgrade HDP-3.0 to HDP-3.1