Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Getting 'Truncated Message' Error while executing ExecuteScript processor to parse protobuf coming from ConsumeKafka processor

Getting 'Truncated Message' Error while executing ExecuteScript processor to parse protobuf coming from ConsumeKafka processor

New Contributor

Hello, I have the following setup: a ConsumeKafka processor that receives a protobuf parsed file, then sends to a ExecuteScript in python that parses the protobuf and finally sends to a putfile processor.

My problem is that I'm receiving the 'Truncated Message' error when the message have a nested Json, I don't receive any error when parsing a flat json.

The error occurs at the command: register.ParseFromString(buf)

Thanks in advance

def parser(buf) :
  import register_pb2
  try :       
    register = register_pb2.Register()
    register.ParseFromString(buf)
...

class PyStreamCallback(StreamCallback):
  def __init__(self):
        pass

  def process(self, inputStream, outputStream):
    text = IOUtils.toByteArray(inputStream)
    newObj = parser(inputStream)
    outputStream.write(bytearray(json.dumps(newObj, indent=4).encode('utf-8'))) 

flowFile = session.get()
if (flowFile != None):
  flowFile = session.write(flowFile,PyStreamCallback())
  session.transfer(flowFile, REL_SUCCESS)
2 REPLIES 2
Highlighted

Re: Getting 'Truncated Message' Error while executing ExecuteScript processor to parse protobuf coming from ConsumeKafka processor

Super Guru

I'm not familiar with the register_pb2 module, is that a custom module? You are calling ParseFromString but passing in a byte array, so maybe there's an encoding issue? In any case, that error is coming from protobuf code so it is likely not getting something it is expecting.

Highlighted

Re: Getting 'Truncated Message' Error while executing ExecuteScript processor to parse protobuf coming from ConsumeKafka processor

New Contributor

Hello @Matt Burgess, first I'd like to thank you for all your posts on your blog. They are really useful.

Yes, register_pb2 module is a python compiled custom proto file, I've tried using IOUtils.toString too but it not works too.

Don't have an account?
Coming from Hortonworks? Activate your account here