question Re: unable to view the content of acid table in spark and hive in Support Questions

unable to view the content of acid table in spark and hive

sduraisankar93 — Wed, 17 Jul 2019 19:34:42 GMT

i have created acid orc partition table in hive. By using spark i insert the data to that table . in table pointing location its showing data like

/warehouse/tablespace/managed/hive/testdb.db/table__2/full_date=2019-07-17/delta_0000000_0000000_0000/ 0000_0 .

when i query the table it showing only column names can any one give me a solution

Re: unable to view the content of acid table in spark and hive

Shu_ashu — Thu, 18 Jul 2019 10:25:39 GMT

@Duraisankar S

You can run Major compaction on the partition in Hive, after the major compaction is done base-**** directory will be created.

Then spark able to read the specific partition which have base-*** directories in it.

But spark not able to read delta directories as there is an open Jira [SPARK-15348] about spark is not able to read acid table.

I think starting from HDP-3.X HiveWareHouseConnector is able to support to read HiveAcid tables.

Re: unable to view the content of acid table in spark and hive

sduraisankar93 — Thu, 18 Jul 2019 13:13:58 GMT

thank u 4 the info

Re: unable to view the content of acid table in spark and hive

Shu_ashu — Thu, 18 Jul 2019 23:15:17 GMT

@Duraisankar S

If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂

Re: unable to view the content of acid table in spark and hive

sduraisankar93 — Thu, 18 Jul 2019 23:38:51 GMT

@shu

hive warehouse connector support for acid table is available in HDP-3.0 r do i need to upgrade HDP-3.0 to HDP-3.1

Re: unable to view the content of acid table in spark and hive

sduraisankar93 — Thu, 18 Jul 2019 23:39:00 GMT

@Shu hive warehouse connector support for acid table is available in HDP-3.0 r do i need to upgrade HDP-3.0 to HDP-3.1