question Re: Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit. in Archives of Support Questions (Read Only)

Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

sachinsga_com — Wed, 16 Mar 2016 14:57:47 GMT

My hive query is failing even after setting the property ipc.maximum.data.length to the end limit of int which is 2147483647 with the following stacktrace:

Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
	at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
	at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
	at com.google.protobuf.CodedInputStream.readRawByte(CodedInputStream.java:769)
	at com.google.protobuf.CodedInputStream.readRawVarint64(CodedInputStream.java:462)
	at com.google.protobuf.CodedInputStream.readUInt64(CodedInputStream.java:188)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$ColumnStatistics.<init>(OrcProto.java:4330)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$ColumnStatistics.<init>(OrcProto.java:4280)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$ColumnStatistics$1.parsePartialFrom(OrcProto.java:4454)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$ColumnStatistics$1.parsePartialFrom(OrcProto.java:4449)
	at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeStatistics.<init>(OrcProto.java:12224)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeStatistics.<init>(OrcProto.java:12171)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeStatistics$1.parsePartialFrom(OrcProto.java:12260)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeStatistics$1.parsePartialFrom(OrcProto.java:12255)
	at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$Metadata.<init>(OrcProto.java:12898)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$Metadata.<init>(OrcProto.java:12845)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$Metadata$1.parsePartialFrom(OrcProto.java:12934)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$Metadata$1.parsePartialFrom(OrcProto.java:12929)
	at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
	at org.apache.hadoop.hive.ql.io.orc.OrcProto$Metadata.parseFrom(OrcProto.java:13045)
	at org.apache.hadoop.hive.ql.io.orc.ReaderImpl$MetaInfoObjExtractor.<init>(ReaderImpl.java:426)
	at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:295)
	at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:197)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:999)
	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
	... 16 more

Re: Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

jyadav — Wed, 16 Mar 2016 21:36:35 GMT

Looks like your table has ORC format so can you please try to set below properties and try it again?

set orc.compress.size=4096

set hive.exec.orc.default.stripe.size=268435456

Re: Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

sachinsga_com — Thu, 17 Mar 2016 01:29:45 GMT

actually its a select query which is creating this issue, so do you want me to set these properties while storing the data in the table??

Re: Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

jyadav — Thu, 17 Mar 2016 12:02:27 GMT

Yes, Please follow these steps and let me know if you still face same issue. Also kindly mention your HDP version.

hive> set orc.compress.size=4096;

hive> set hive.exec.orc.default.stripe.size=268435456;

hive> your create table DDL;

hive> load data query in orc table;

hive> you select query;

Re: Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

sachinsga_com — Fri, 18 Mar 2016 00:56:55 GMT

hdp version is = HDP 2.1.2.0-402

I will also follow the steps given by you.

Re: Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

sachinsga_com — Mon, 21 Mar 2016 20:19:09 GMT

it worked for some days data BTW can you please elaborate the meaning of these properties?

Re: Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

edi_bice — Wed, 23 Mar 2016 03:27:23 GMT

It's a Hive bug - see Hive JIRA for details about the bug

https://issues.apache.org/jira/browse/HIVE-11592

Re: Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

naga_kolli — Fri, 30 Sep 2016 03:45:42 GMT

even I have the same issue.

My hdp version is 2.3.4.23-3

Please advise

Re: Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

ksh22sidartha — Mon, 05 Aug 2019 19:08:48 GMT

@Jitendra Yadav I am facing the same issue in HDP 2.6.5 . I follow the steps you have mentioned above. It works until some data size. I am processing ORC hive table with spark,where a file is store as one of the column in this table. Is there any permanent solution for this?

Re: Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

ksh22sidartha — Tue, 06 Aug 2019 12:57:33 GMT

I am facing the same issue in HDP 2.6.5 . I follow the steps you have mentioned above. It works until some data size. I am processing ORC hive table with spark,where a file is store as one of the column in this table. Is there any permanent solution for this?