update TCP stream with batchsize 10000 at once using update record

mark_hadoop — Sun, 18 Aug 2019 09:35:43 GMT

Hi,

I am getting a plain json stream with '\n' delimiter through TCP.

I am listening to TCP using listenTCP, set batchsize to 10000.

My json is with variable values, ex:

{"a":"20180831","b":"b"}
{"a":"20180831","b":"b","c":"c"}

I want to add a partition_value attribute to every line in json stream at once, which should look like

The attribute a is always present in json, so I want to use partition_value in a

{"a":"20180831","b":"b","partition_value":"20180831"}
{"a":"20180831","b":"b","c":"c","partition_value":"20180831"}

I have used "UpdateRecord" processor

below are the configuration

UpdateRecord

JsonTreeReader

AvroSchemaRegistry

AvroRecordSetWriter

I used UpdateRecord -> jsontreereader ->avroschemaregistry

|_________ -> avrorecordsetwriter

Then I have used avrotojson

I am getting only one line as output, can you please suggest where it is happening wrong or let me know if there is a better way to do it

Thank you

{"a":"20180831","b":"b","c":null,"partition_value":"20180831"}

Re: update TCP stream with batchsize 10000 at once using update record

mark_hadoop — Fri, 31 Aug 2018 11:18:01 GMT

Looks like the jpgs are not aligned as expected, but the names of jpgs are listed below in order, Thank you

mark_hadoop — Wed, 05 Sep 2018 16:58:27 GMT

I find an alternate way of doing

Thank you