Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

unable to view the content of acid table in spark and hive

avatar

i have created acid orc partition table in hive. By using spark i insert the data to that table . in table pointing location its showing data like

/warehouse/tablespace/managed/hive/testdb.db/table__2/full_date=2019-07-17/delta_0000000_0000000_0000/ 0000_0 .


when i query the table it showing only column names can any one give me a solution

1 ACCEPTED SOLUTION

avatar
Master Guru

@Duraisankar S

You can run Major compaction on the partition in Hive, after the major compaction is done base-**** directory will be created.

Then spark able to read the specific partition which have base-*** directories in it.

But spark not able to read delta directories as there is an open Jira [SPARK-15348] about spark is not able to read acid table.


I think starting from HDP-3.X HiveWareHouseConnector is able to support to read HiveAcid tables.

View solution in original post

5 REPLIES 5

avatar
Master Guru

@Duraisankar S

You can run Major compaction on the partition in Hive, after the major compaction is done base-**** directory will be created.

Then spark able to read the specific partition which have base-*** directories in it.

But spark not able to read delta directories as there is an open Jira [SPARK-15348] about spark is not able to read acid table.


I think starting from HDP-3.X HiveWareHouseConnector is able to support to read HiveAcid tables.

avatar

thank u 4 the info

avatar
Master Guru

@Duraisankar S

If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂

avatar

@shu


hive warehouse connector support for acid table is available in HDP-3.0 r do i need to upgrade HDP-3.0 to HDP-3.1



avatar

@Shu hive warehouse connector support for acid table is available in HDP-3.0 r do i need to upgrade HDP-3.0 to HDP-3.1