question Re: Query hive acid with spark in Archives of Support Questions (Read Only)

Query hive acid with spark

itsssudheer — Fri, 22 Sep 2017 22:24:04 GMT

Unable to query hive acid with spark getting following exception

Caused by: java.util.concurrent.ExecutionException: java.lang.NumberFormatException: For input string: "0012050_0000"

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:998)

... 118 more

Caused by: java.lang.NumberFormatException: For input string: "0012050_0000"

at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

at java.lang.Long.parseLong(Long.java:589)

at java.lang.Long.parseLong(Long.java:631)

at org.apache.hadoop.hive.ql.io.AcidUtils.parseDelta(AcidUtils.java:310)

at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:379)

at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:634)

at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:620)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

Re: Query hive acid with spark

deepesh1 — Fri, 22 Sep 2017 23:48:47 GMT

Reading Hive ACID ORC data has some issues, it seems the data is not visible unless you compact the table atleast once, see SPARK-16996. Hive ACID tables ORC data operations are currently not supported from Spark, see SPARK-15348.

Re: Query hive acid with spark

njayakumar — Sat, 23 Sep 2017 04:10:01 GMT

@sudheer

Could you please run the major compact after the ETL ingestion.

please find below alter statement for the reference 

alter table <<table_name>> compact 'MAJOR';

Re: Query hive acid with spark

maciek1 — Mon, 20 Nov 2017 15:41:53 GMT

@deepesh
I think there is also problem with hive modification made by hortonworks.
Could you please look at my comments in Spark Jira ? SPARK-16996

Re: Query hive acid with spark

thirun66 — Mon, 30 Jul 2018 01:26:43 GMT

alter table <<table_name>> compact 'MAJOR';

Hi, Is this way worked for any one ?

If worked for you, can you help me out in for what table we need to this compact action.

In may case I have a master table mt1 and lookup tables t2,t3,t4,t5. And I am inserting this select result to target table t6. These all are orc type.

Thanks

Thirupathi Nerella

Re: Query hive acid with spark

thirun66 — Mon, 30 Jul 2018 01:27:31 GMT

Is this worked ?