Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Query hive acid with spark

Solved Go to solution
Highlighted

Query hive acid with spark

Contributor

Unable to query hive acid with spark getting following exception

Caused by: java.util.concurrent.ExecutionException: java.lang.NumberFormatException: For input string: "0012050_0000"

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:998)

... 118 more

Caused by: java.lang.NumberFormatException: For input string: "0012050_0000"

at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

at java.lang.Long.parseLong(Long.java:589)

at java.lang.Long.parseLong(Long.java:631)

at org.apache.hadoop.hive.ql.io.AcidUtils.parseDelta(AcidUtils.java:310)

at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:379)

at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:634)

at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:620)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Query hive acid with spark

Contributor

@sudheer

Could you please run the major compact after the ETL ingestion.

please find below alter statement for the reference 

alter table <<table_name>> compact 'MAJOR';

View solution in original post

5 REPLIES 5

Re: Query hive acid with spark

Master Collaborator

Reading Hive ACID ORC data has some issues, it seems the data is not visible unless you compact the table atleast once, see SPARK-16996. Hive ACID tables ORC data operations are currently not supported from Spark, see SPARK-15348.

Highlighted

Re: Query hive acid with spark

Contributor

@sudheer

Could you please run the major compact after the ETL ingestion.

please find below alter statement for the reference 

alter table <<table_name>> compact 'MAJOR';

View solution in original post

Highlighted

Re: Query hive acid with spark

Explorer

Is this worked ?

Highlighted

Re: Query hive acid with spark

Explorer

@deepesh
I think there is also problem with hive modification made by hortonworks.
Could you please look at my comments in Spark Jira ? SPARK-16996

Highlighted

Re: Query hive acid with spark

Explorer

alter table <<table_name>> compact 'MAJOR';

Hi, Is this way worked for any one ?

If worked for you, can you help me out in for what table we need to this compact action.

In may case I have a master table mt1 and lookup tables t2,t3,t4,t5. And I am inserting this select result to target table t6. These all are orc type.

Thanks

Thirupathi Nerella

Don't have an account?
Coming from Hortonworks? Activate your account here