Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

com.google.protobuf.InvalidProtocolBufferException when accessing CDH4 hdfs from CDH5 cluser nodes.

com.google.protobuf.InvalidProtocolBufferException when accessing CDH4 hdfs from CDH5 cluser nodes.

New Contributor

Hi ,

I am trying to distcp between CDH4(4.7.1) and CDH5(5.5.1) clusters. distcp is running into issues, please see below log on distcp command run.  Alternatively i tried to access CDH4 hdfs from CDH4 client it throwing below error.  Any help is appreciated.

 

simple hdfs access:

---------------------------

$ hadoop version
Hadoop 2.6.0-cdh5.5.1
Subversion http://github.com/cloudera/hadoop -r e1581abbb6ab62b0a41b7ce6141d7280bf0c53da
Compiled by jenkins on 2015-12-02T18:38Z
Compiled with protoc 2.5.0
From source with checksum 98e07176d1787150a6a9c087627562c
This command was run using /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/hadoop-common-2.6.0-cdh5.5.1.jar

 

$ hadoop fs -ls hdfs://cdh4_cluster.abc.com:9000/hadoop_ops/
ls: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).; Host Details : local host is: "dm001.dev2.abc.com/10.116.206.47"; destination host is: "cdh4_cluster.abc.com":9000;

 

Below is distcp failure which run from CDH4 environment to push data to CDH5:

----------------------------------------------------------------------------------------------------------

$ hadoop version
Hadoop 2.0.0-mr1-cdh4.7.1
Subversion file:///data/jenkins/workspace/generic-package-centos64-5-5/topdir/BUILD/hadoop-2.0.0-cdh4.7.1/src/hadoop-mapreduce1-project -r Unknown
Compiled by jenkins on Tue Nov 18 08:20:05 PST 2014
From source with checksum 9ff165dd3cdf37f78b5950cfcef45687
This command was run using /server/turn/deploy/build-160203/turn/lib/hadoop-common-2.0.0-cdh4.7.1.jar

 

 

$ hadoop distcp hftp://cdh4_cluster.abc.com:50070/hadoop_ops hdfs://cdh5_cluser.abc.com:8020/hadoop_ops/
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]


SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status; Host Details : local host is: "devdm004.abc.com/10.16.8.9"; destination host is: "cdh5_cluser.abc.com":8020;
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
    at org.apache.hadoop.ipc.Client.call(Client.java:1242)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
    at com.sun.proxy.$Proxy8.getFileInfo(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
    at com.sun.proxy.$Proxy8.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1380)
    at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1051)
    at org.apache.hadoop.tools.DistCp.copy(DistCp.java:668)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:883)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:910)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status
    at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81)
    at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto$Builder.buildParsed(RpcPayloadHeaderProtos.java:1094)
    at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto$Builder.access$1300(RpcPayloadHeaderProtos.java:1028)
    at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:986)
    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:949)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:847)

 

1 REPLY 1

Re: com.google.protobuf.InvalidProtocolBufferException when accessing CDH4 hdfs from CDH5 cluser nod

Master Guru
Please visit
http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_admin_distcp_data_cluster_migrate...
to
see what the access matrix and point of running command is for CDH4 <->
CDH5 transfers via DistCp.