Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDFS canary test fails NoClassDefFoundError: SubnetUtils

Highlighted

HDFS canary test fails NoClassDefFoundError: SubnetUtils

New Contributor

Hi,

 

the HDFS canaray test fails on our cluster:

2016-04-17 07:55:34,272 WARN com.cloudera.cmon.firehose.polling.CdhTask: (14 skipped) Exception in doWork for task: hdfs1_HDFS_CANARY
java.lang.NoClassDefFoundError: org/apache/commons/net/util/SubnetUtils
        at org.apache.hadoop.net.NetUtils.isValidSubnet(NetUtils.java:837)
        at org.apache.hadoop.hdfs.DFSClient.getLocalInterfaceAddrs(DFSClient.java:770)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:715)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:650)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2643)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2680)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2662)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:379)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:178)
        at com.cloudera.cmf.cdh5client.hdfs.FileSystemImpl.<init>(FileSystemImpl.java:41)
        at com.cloudera.cmf.cdh5client.CDH5ObjectFactoryImpl.getFileSystem(CDH5ObjectFactoryImpl.java:115)
        at com.cloudera.cmon.firehose.polling.AbstractFileSystemClientTask.doWorkWithClientConfig(AbstractFileSystemClientTask.java:52)
        at com.cloudera.cmon.firehose.polling.AbstractCdhWorkUsingClientConfigs.doWork(AbstractCdhWorkUsingClientConfigs.java:45)
        at com.cloudera.cmon.firehose.polling.CdhTask$InstrumentedWork.doWork(CdhTask.java:230)
        at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.runTask(ImpersonatingTaskWrapper.java:72)
        at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.access$000(ImpersonatingTaskWrapper.java:21)
        at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper$1.run(ImpersonatingTaskWrapper.java:107)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at com.cloudera.cmf.cdh5client.security.UserGroupInformationImpl.doAs(UserGroupInformationImpl.java:41)
        at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.doWork(ImpersonatingTaskWrapper.java:103)
        at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

This occurs already since version 4. But I had just disabled the test.

Currently we are running Cloudera Express 5.7.0 on Centos 6.

Any ideas how to solve this?

 

2 REPLIES 2

Re: HDFS canary test fails NoClassDefFoundError: SubnetUtils

Super Collaborator

Can you let me know the version of CDH and whether this is packages or parcel based install and where you prompted with the exact (org/apache/commons/net/util/SubnetUtils) error in version 4?

 

Just reviewing the stack trace the following line DFSClient.java:770, suggests that java.lang.NoClassDefFoundError in the NetUtils cannot locate the library org.apache.commons.net.util.SubnetUtils .

 

java.lang.NoClassDefFoundError: org/apache/commons/net/util/SubnetUtils
        at org.apache.hadoop.net.NetUtils.isValidSubnet(NetUtils.java:837)
        at org.apache.hadoop.hdfs.DFSClient.getLocalInterfaceAddrs(DFSClient.java:770)

 

Highlighted

Re: HDFS canary test fails NoClassDefFoundError: SubnetUtils

New Contributor

We are using parcel based install (5.7.0-1.cdh5.7.0.p0.45).

 


@Michalis wrote:

 where you prompted with the exact (org/apache/commons/net/util/SubnetUtils) error in version 4?


Good question. I looked through the logs and the NoClassDefFoundError occured only after we upgraded to version 5. Before the canary test failed often due to network problems (in the beginning our Infiniband network was pretty unstable).

So the NoClassDefFoundError might be related to the upgrade. As far as I remember we followed strictly the upgrade documentation and did not run into any bigger problems.

 

The commons-net jar is linked in many folders. Is it required somewhere else or could the parcel-cache/xxx/CDH-4.4 directory cause problems?

 

 

# pwd
/opt/cloudera
# find . -name "commons-net*"
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop-0.20-mapreduce/contrib/gridmix/lib/commons-net-1.4.1.jar
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/oozie/libtools/commons-net-3.1.jar
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hbase/lib/commons-net-3.1.jar
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hcatalog/share/webhcat/svr/lib/commons-net-3.1.jar
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/client-0.20/commons-net-3.1.jar
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/client-0.20/commons-net.jar
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/lib/commons-net-3.1.jar
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/client/commons-net-3.1.jar
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/client/commons-net.jar
./parcel-cache/tmppWq8n4/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/commons-net-1.4.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/sqoop2/client-lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop-0.20-mapreduce/contrib/gridmix/lib/commons-net-1.4.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/oozie/libtools/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/oozie/oozie-sharelib-mr1/lib/hive2/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/oozie/oozie-sharelib-mr1/lib/hcatalog/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/oozie/oozie-sharelib-mr1/lib/hive/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/oozie/oozie-sharelib-mr1/lib/spark/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/oozie/oozie-sharelib-yarn/lib/hive2/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/oozie/oozie-sharelib-yarn/lib/hcatalog/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/oozie/oozie-sharelib-yarn/lib/hive/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/oozie/oozie-sharelib-yarn/lib/spark/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/oozie/libserver/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hbase/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hbase-solr/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/impala/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/mahout/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop/client-0.20/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop/client-0.20/commons-net.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop/client/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop/client/commons-net.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop-kms/webapps/kms/WEB-INF/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/sentry/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/parquet/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop-mapreduce/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/crunch/lib/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/search/lib/search-crunch/commons-net-3.1.jar
./parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/search/lib/commons-net-3.1.jar

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here