Reply
Highlighted
New Contributor
Posts: 6
Registered: ‎11-18-2016

java.lang.NoSuchMethodError when merge small files is triggered after copying data

Hi,

 

When I change the setting of "Small File Average Size Merge Threshold" to be large enough to trigger a merge, I get the NoSuchMethodError Exception shown below.

 

I am moving data from an external tsv table to a parquet table using Hive 1.1.0-cdh5.7.0

(INSERT INTO TABLE parquet_table SELECT * FROM tsv_data)

 

Moving the data into the parquet table is fine, everything is successful if no merge is attempted.

 

Any suggestions on how to address this would be appreciated.

 

 

Diagnostic Messages for this Task:
Error: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:265)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:212)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:332)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:721)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:251)
... 11 more
Caused by: java.lang.NoSuchMethodError: parquet.schema.Types$MessageTypeBuilder.addFields([Lparquet/schema/Type;)Lparquet/schema/Types$BaseGroupBuilder;
at org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.getSchemaByName(DataWritableReadSupport.java:158)
at org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.init(DataWritableReadSupport.java:221)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:256)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:81)
at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:66)
... 16 more

 

Champion
Posts: 760
Registered: ‎05-16-2016

Re: java.lang.NoSuchMethodError when merge small files is triggered after copying data

did you change the  hive.merge.mapfiles or hive.merge.mapredfiles is set to true ? 

New Contributor
Posts: 6
Registered: ‎11-18-2016

Re: java.lang.NoSuchMethodError when merge small files is triggered after copying data

Yes, I believe hive.merge.mapfiles is the applicable flag in this case, but both were set to true when the problem was last observed.

Cloudera Employee
Posts: 41
Registered: ‎11-04-2015

Re: java.lang.NoSuchMethodError when merge small files is triggered after copying data

Hi,

 

The NoSuchMethodError suggests that there are two incompatible version of java classes in your classpath (these are usually found in "hive-exec.jar"). Maybe one node in your cluster is misconfigured and contains old hive libraries.

 

This usually can happen after an upgrade when you migrate from packages to parcel based installation. Please check if the /usr/lib/hive and /usr/lib/hive-hcatalog directories are deleted on the NodeManager hosts (at least rename them first).

 

Regards

 Miklos Szurap

Customer Operations Engineer

New Contributor
Posts: 6
Registered: ‎11-18-2016

Re: java.lang.NoSuchMethodError when merge small files is triggered after copying data

We have two nodes in the small test cluster that I am working with, in both cases, I see:

 

$ find /usr/lib -name hive-exec.jar | xargs ls -l
lrwxrwxrwx 1 root root 28 May 25 04:23 /usr/lib/hive/lib/hive-exec.jar -> hive-exec-1.1.0-cdh5.7.0.jar
lrwxrwxrwx 1 root root 31 May 25 04:24 /usr/lib/impala/lib/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:26 /usr/lib/oozie/libserver/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:26 /usr/lib/oozie/libtools/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:26 /usr/lib/oozie/oozie-sharelib-mr1/lib/hcatalog/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:26 /usr/lib/oozie/oozie-sharelib-mr1/lib/hive/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:26 /usr/lib/oozie/oozie-sharelib-mr1/lib/hive2/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:26 /usr/lib/oozie/oozie-sharelib-mr1/lib/spark/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:27 /usr/lib/oozie/oozie-sharelib-mr1/lib/sqoop/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:27 /usr/lib/oozie/oozie-sharelib-yarn/lib/hcatalog/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:27 /usr/lib/oozie/oozie-sharelib-yarn/lib/hive/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:27 /usr/lib/oozie/oozie-sharelib-yarn/lib/hive2/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:27 /usr/lib/oozie/oozie-sharelib-yarn/lib/spark/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:27 /usr/lib/oozie/oozie-sharelib-yarn/lib/sqoop/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar
lrwxrwxrwx 1 root root 31 May 25 04:24 /usr/lib/sentry/lib/hive-exec.jar -> /usr/lib/hive/lib/hive-exec.jar

 

So, these at least, would appear to be the same.

Is there someplace else I should be checking?

Cloudera Employee
Posts: 41
Registered: ‎11-04-2015

Re: java.lang.NoSuchMethodError when merge small files is triggered after copying data

Hi,

 

You can search for the class with a command like:

 

for i in `find /usr/lib -name '*.jar'` ; do /usr/java/jdk1.7.0_67-cloudera/bin/jar tf $i | grep "parquet/schema/Type.class" && echo $i ; done

You haven't wrote, is it a fresh install?

New Contributor
Posts: 6
Registered: ‎11-18-2016

Re: java.lang.NoSuchMethodError when merge small files is triggered after copying data

I'm not sure about the details of the installation history. I believe it was installed in one pass and has not been upgraded. In trying to run the suggested find command, I see that we have only jdk1.8.0_60 installed; we do not have jdk1.7.0_67-cloudera.

Removing redundant "parquet/schema/Type.class" lines, the results from the find are:

/usr/lib/crunch/lib/parquet-column.jar
/usr/lib/flume-ng/lib/parquet-column.jar
/usr/lib/hadoop/parquet-column.jar
/usr/lib/hadoop/parquet-hadoop-bundle.jar
/usr/lib/hadoop/parquet-pig-bundle.jar
/usr/lib/hbase-solr/lib/parquet-column.jar
/usr/lib/hive/lib/hive-exec-1.1.0-cdh5.7.0.jar
/usr/lib/hive/lib/hive-exec.jar
/usr/lib/hive/lib/hive-jdbc-1.1.0-cdh5.7.0-standalone.jar
/usr/lib/hive/lib/hive-jdbc-standalone.jar
/usr/lib/hive/lib/parquet-hadoop-bundle.jar
/usr/lib/impala/lib/hive-exec.jar
/usr/lib/impala/lib/parquet-hadoop-bundle.jar
/usr/lib/kite/kite-tools-1.0.0-cdh5.7.0-runtime.jar
/usr/lib/kite/kite-tools-runtime.jar
/usr/lib/kite/lib/kite-tools-1.0.0-cdh5.7.0-runtime.jar
/usr/lib/kite/lib/parquet-column.jar
/usr/lib/oozie/libserver/hive-exec.jar
/usr/lib/oozie/libserver/parquet-hadoop-bundle.jar
/usr/lib/oozie/libtools/hive-exec.jar
/usr/lib/oozie/libtools/parquet-hadoop-bundle.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/hcatalog/hive-exec.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/hcatalog/parquet-hadoop-bundle.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/hive/hive-exec.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/hive/parquet-hadoop-bundle.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/hive2/hive-exec.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/pig/parquet-pig-bundle.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/spark/hive-exec.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/spark/parquet-column.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/sqoop/hive-exec.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/sqoop/parquet-column.jar
/usr/lib/oozie/oozie-sharelib-mr1/lib/sqoop/parquet-hadoop-bundle.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/hcatalog/hive-exec.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/hcatalog/parquet-hadoop-bundle.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/hive/hive-exec.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/hive/parquet-hadoop-bundle.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/hive2/hive-exec.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/pig/parquet-pig-bundle.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/spark/hive-exec.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/spark/parquet-column.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/sqoop/hive-exec.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/sqoop/parquet-column.jar
/usr/lib/oozie/oozie-sharelib-yarn/lib/sqoop/parquet-hadoop-bundle.jar
/usr/lib/parquet/lib/parquet-column-1.5.0-cdh5.7.0.jar
/usr/lib/parquet/lib/parquet-hadoop-bundle-1.5.0-cdh5.7.0.jar
/usr/lib/parquet/lib/parquet-pig-bundle-1.5.0-cdh5.7.0.jar
/usr/lib/parquet/parquet-column.jar
/usr/lib/parquet/parquet-hadoop-bundle.jar
/usr/lib/parquet/parquet-pig-bundle.jar
/usr/lib/search/lib/parquet-column.jar
/usr/lib/search/lib/search-crunch/parquet-column.jar
/usr/lib/sentry/lib/hive-exec.jar
/usr/lib/sentry/lib/parquet-hadoop-bundle.jar
/usr/lib/sqoop/lib/parquet-column.jar
/usr/lib/sqoop2/webapps/sqoop/WEB-INF/lib/parquet-column.jar

 

Cloudera Employee
Posts: 41
Registered: ‎11-04-2015

Re: java.lang.NoSuchMethodError when merge small files is triggered after copying data

It's not obvious from that output. Let's try from the other end. As you have package based installation can you run the following?

 

rpm -qa | egrep "cdh|parquet"
New Contributor
Posts: 6
Registered: ‎11-18-2016

Re: java.lang.NoSuchMethodError when merge small files is triggered after copying data

Here is the result:

 

$ rpm -qa | egrep "cdh|parquet"
zookeeper-3.4.5+cdh5.7.0+94-1.cdh5.7.0.p0.80.el6.x86_64
parquet-format-2.1.0+cdh5.7.0+14-1.cdh5.7.0.p0.79.el6.noarch
hadoop-libhdfs-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-hdfs-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-mapreduce-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hive-hcatalog-1.1.0+cdh5.7.0+522-1.cdh5.7.0.p0.88.el6.noarch
flume-ng-1.6.0+cdh5.7.0+37-1.cdh5.7.0.p0.79.el6.noarch
search-1.0.0+cdh5.7.0+0-1.cdh5.7.0.p0.78.el6.noarch
sqoop2-1.99.5+cdh5.7.0+38-1.cdh5.7.0.p0.79.el6.noarch
spark-python-1.6.0+cdh5.7.0+180-1.cdh5.7.0.p0.84.el6.noarch
hadoop-kms-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-hdfs-nfs3-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
whirr-0.9.0+cdh5.7.0+19-1.cdh5.7.0.p0.78.el6.noarch
hue-pig-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
hue-search-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
cloudera-cdh-5-0.x86_64
avro-libs-1.7.6+cdh5.7.0+113-1.cdh5.7.0.p0.78.el6.noarch
hue-common-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
bigtop-jsvc-0.6.0+cdh5.7.0+804-1.cdh5.7.0.p0.78.el6.x86_64
hive-jdbc-1.1.0+cdh5.7.0+522-1.cdh5.7.0.p0.88.el6.noarch
hbase-1.2.0+cdh5.7.0+129-1.cdh5.7.0.p0.88.el6.x86_64
hive-1.1.0+cdh5.7.0+522-1.cdh5.7.0.p0.88.el6.noarch
hadoop-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-yarn-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64
hadoop-0.20-mapreduce-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
kite-1.0.0+cdh5.7.0+130-1.cdh5.7.0.p0.77.el6.noarch
pig-0.12.0+cdh5.7.0+84-1.cdh5.7.0.p0.77.el6.noarch
sqoop-1.4.6+cdh5.7.0+56-1.cdh5.7.0.p0.78.el6.noarch
spark-core-1.6.0+cdh5.7.0+180-1.cdh5.7.0.p0.84.el6.noarch
crunch-0.11.0+cdh5.7.0+93-1.cdh5.7.0.p0.78.el6.noarch
sqoop2-client-1.99.5+cdh5.7.0+38-1.cdh5.7.0.p0.79.el6.noarch
oozie-4.1.0+cdh5.7.0+267-1.cdh5.7.0.p0.78.el6.noarch
hbase-solr-1.5+cdh5.7.0+64-1.cdh5.7.0.p0.78.el6.noarch
spark-worker-1.6.0+cdh5.7.0+180-1.cdh5.7.0.p0.84.el6.noarch
spark-history-server-1.6.0+cdh5.7.0+180-1.cdh5.7.0.p0.84.el6.noarch
hadoop-hdfs-fuse-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
pig-udf-datafu-1.1.0+cdh5.7.0+20-1.cdh5.7.0.p0.77.el6.noarch
hadoop-httpfs-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hue-plugins-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
hue-beeswax-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
hue-sqoop-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
hue-spark-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
hue-hbase-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
hue-rdbms-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
avro-tools-1.7.6+cdh5.7.0+113-1.cdh5.7.0.p0.78.el6.noarch
bigtop-utils-0.7.0+cdh5.7.0+0-1.cdh5.7.0.p0.78.el6.noarch
bigtop-tomcat-0.7.0+cdh5.7.0+0-1.cdh5.7.0.p0.78.el6.noarch
solr-4.10.3+cdh5.7.0+389-1.cdh5.7.0.p0.85.el6.noarch
parquet-1.5.0+cdh5.7.0+176-1.cdh5.7.0.p0.78.el6.noarch
sentry-1.5.1+cdh5.7.0+184-1.cdh5.7.0.p0.86.el6.noarch
hadoop-client-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hive-webhcat-1.1.0+cdh5.7.0+522-1.cdh5.7.0.p0.88.el6.noarch
solr-mapreduce-1.0.0+cdh5.7.0+0-1.cdh5.7.0.p0.78.el6.noarch
oozie-client-4.1.0+cdh5.7.0+267-1.cdh5.7.0.p0.78.el6.noarch
spark-master-1.6.0+cdh5.7.0+180-1.cdh5.7.0.p0.84.el6.noarch
mahout-0.9+cdh5.7.0+29-1.cdh5.7.0.p0.79.el6.noarch
llama-1.0.0+cdh5.7.0+0-1.cdh5.7.0.p0.78.el6.noarch
hive-hbase-1.1.0+cdh5.7.0+522-1.cdh5.7.0.p0.88.el6.noarch
hue-impala-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
hue-zookeeper-3.9.0+cdh5.7.0+1759-1.cdh5.7.0.p0.86.el6.x86_64
impala-shell-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64

 

Everything matches "cdh5.7.0", except for cloudera-cdh-5-0.x86_64

 

Cloudera Employee
Posts: 41
Registered: ‎11-04-2015

Re: java.lang.NoSuchMethodError when merge small files is triggered after copying data

Hi,

As far as I see on https://www.cloudera.com/documentation/enterprise/5-7-x/topics/cdh_qs_mrv1_pseudo.html the cloudera-cdh-5-0.x86_64.rpm is related to MRv1. Do you need that? Also do you see the same packages on all the hosts?

Announcements