Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

Solved Go to solution
Highlighted

SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

New Contributor

G'Day all,

 

We've moved from Impala 1.1 to 1.1.1 and are evaluating the use of Parquet.  Unfortunately after creating the table it is unable to be used with the following error:

org.apache.hadoop.hive.serde2.SerDeException SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

when attempting to access (or even drop) the table using impala-shell

 

Details of installation:

Clean install to Impala 1.1

Package upgrade to 1.1.1

 

parquet-hive-1.0.jar exists in 

/opt/cloudera/parcels/IMPALA-1.1.1-1.p0.17/lib/impala/lib/

/opt/cloudera/parcels/CDH-4.3.1-1.cdh4.3.1.p0.110/lib/hive/lib

 

Has anyone else encountered this issue?

 

Derek

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

New Contributor

I solved my problem.

 

My problem was two fold... I had a lingering old install of hive which was what my shell was defaulting to. I removed the second bad install of hive. The second problem is that the latest version of cdh seems to be broken for parquet. There's a bunch of missing jars. I followed the following guide and I am now able to use parquet files.

 

http://analog99.wordpress.com/2013/01/07/setting-up-stats-db-in-hive/

View solution in original post

8 REPLIES 8

Re: SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

Explorer

Hi Derek -

 

In order to allow compatibilty between Impala and Hive, Parquet backed tables after 1.1.1 contain additional metadata about the file format. Run the following commands (in Hive) to update your metadata. 

 

ALTER TABLE table_name SET SERDE 'parquet.hive.serde.ParquetHiveSerDe';
ALTER TABLE table_name SET FILEFORMAT
  INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
  OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat";

 

Highlighted

Re: SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

New Contributor

I am getting the following error when I try to update the hive metadata:

 

Exception in thread "main" java.lang.NoClassDefFoundError: parquet/Log
at parquet.hive.DeprecatedParquetInputFormat.<clinit>(DeprecatedParquetInputFormat.java:63)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:299)
at org.apache.hadoop.hive.ql.metadata.Table.<init>(Table.java:96)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:966)
at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.addInputsOutputsAlterTable(DDLSemanticAnalyzer.java:1105)
at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTableSerde(DDLSemanticAnalyzer.java:1033)
at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:209)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:929)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:893)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: parquet.Log
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Highlighted

Re: SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

Explorer

Hey there -

 

I am really sorry for not replying back, I forgot to subscribe to thread. Have you got this issue worked out already? If so, it would be nice to get the fix so we can close the loop. 

 

If not, I suspect its because you need more of the Parquet JARs in that CDH directory. Is this exception being thrown from the Hive shell or the Impala shell? Run the following command for me, so I can see where the JARs currently are. 

 

$ find /opt/cloudera/parcels/CDH/lib/hive/ -name "parquet*.jar"

 Also, is this using the Hive Metastore server? Make sure to restart that, as well as Impala and try again. 

Highlighted

Re: SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

New Contributor

I’m facing the same problem. I installed Impala 1.1.1 and CDH4.1.2.

I created a table in the impala-shell using the Parquet file format but I am not able to insert data into into.

impala> create table data_parquet like data stored as parquetfile;
impala> insert into data_parquet select * from data;
Query: insert into data_parquet select * from data
ERROR: AnalysisException: Target table 'default.data_parquet' is incompatible with SELECT / PARTITION expressions.
Expression 'data.payload_user_uid' (type: BIGINT) is not compatible with column 'payload_user_type' (type: STRING)

I tried the suggestion on this topic.

hive> alter table data_parquet set serde 'parquet.hive.serde.ParquetHiveSerDe';
FAILED: RuntimeException java.lang.ClassNotFoundException: parquet.hive.DeprecatedParquetInputFormat

I have the parquet jar in the hive lib directory.

$ find /opt/cloudera/parcels/CDH/lib/hive/lib -name "parquet*.jar"
/opt/cloudera/parcels/CDH/lib/hive/lib/parquet-hive-1.0.jar
Highlighted

Re: SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

New Contributor

I solved my problem.

 

My problem was two fold... I had a lingering old install of hive which was what my shell was defaulting to. I removed the second bad install of hive. The second problem is that the latest version of cdh seems to be broken for parquet. There's a bunch of missing jars. I followed the following guide and I am now able to use parquet files.

 

http://analog99.wordpress.com/2013/01/07/setting-up-stats-db-in-hive/

View solution in original post

Highlighted

Re: SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

New Contributor

Hi,

 

Can you publish the list of jars you used to solve this issue. I am facing the same issue in version 5.4.0 and read it occure in 5.4.2 as-well.

 

-Sreesankar

Highlighted

Re: SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

Master Collaborator

I believe you may be hitting https://issues.cloudera.org/browse/IMPALA-2048 or a variant thereof. Please have a look at that JIRA, it includes a workaround that may be acceptable to you.

Highlighted

Re: SerDe parquet.hive.serde.ParquetHiveSerDe does not exist

New Contributor

This seems to be a bug and hope we get the fix included in cdh 5.4.3.

 

one of the work arounds observed was to

 

Have 2 hive table pointing to the same locationg nad have onle of the table accessed through Impala while thye other is accessed through Hive.

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here