Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

update TCP stream with batchsize 10000 at once using update record

Solved Go to solution
Highlighted

update TCP stream with batchsize 10000 at once using update record

Expert Contributor

Hi,

I am getting a plain json stream with '\n' delimiter through TCP.

I am listening to TCP using listenTCP, set batchsize to 10000.

My json is with variable values, ex:

{"a":"20180831","b":"b"}
{"a":"20180831","b":"b","c":"c"}

I want to add a partition_value attribute to every line in json stream at once, which should look like

The attribute a is always present in json, so I want to use partition_value in a

{"a":"20180831","b":"b","partition_value":"20180831"}
{"a":"20180831","b":"b","c":"c","partition_value":"20180831"}

I have used "UpdateRecord" processor

below are the configuration

UpdateRecord

JsonTreeReader


88404-jsontreereader.jpg


AvroSchemaRegistry


AvroRecordSetWriter



I used UpdateRecord -> jsontreereader ->avroschemaregistry

|_________ -> avrorecordsetwriter

Then I have used avrotojson

I am getting only one line as output, can you please suggest where it is happening wrong or let me know if there is a better way to do it

Thank you

{"a":"20180831","b":"b","c":null,"partition_value":"20180831"}

88407-avrotojson.jpg

88406-avrosetwriter.jpg

88405-avroschemaregistry.jpg

88403-updaterecord.jpg

1 ACCEPTED SOLUTION

Accepted Solutions

Re: update TCP stream with batchsize 10000 at once using update record

Expert Contributor

I find an alternate way of doing

Thank you

2 REPLIES 2

Re: update TCP stream with batchsize 10000 at once using update record

Expert Contributor

Looks like the jpgs are not aligned as expected, but the names of jpgs are listed below in order, Thank you

Re: update TCP stream with batchsize 10000 at once using update record

Expert Contributor

I find an alternate way of doing

Thank you