Creating a new Record from input record

[#1]

I have a usecase which involves creating a new Record(actually several records) from a record in input flowfile, and writing it in output flowfile.

PS: I would use avro reader to read input and avro writer to write into output.

What is idiomatic way of doing it in nifi?

[#2]

@manishg, what exactly are you trying to achieve? Can you be a little bit more specific?
In NiFi you have plenty of processors which will split your data but it depends on what you are trying to split and how.

For example you could use an SplitRecord Processors which basically does exactly what you need, it splits an input FlowFilw into multiple smaller FlowFiles. All you have to do is configure a Reader and a Writer, so that NiFi knows how to read and how to write the data. Afterwards, you set how many record you want on each new flowfile .... and that is all.

You can also you SplitAvro, SplitJSON, SplitContent, SplitText and even SplitXML, depending on what you are actually trying to achieve with your entire Flow.

Another option would be to use a Scripting processor like ExecuteStreamCommand or ExecuteScript, in which you define and configure a script to perform the logic you are looking for. And there are many other options which you could also try.

So, what I am trying to say is that if you want a specific answer to your question, you need to say exactly what you are trying to do. Otherwise, the above lines should be sufficient to understand that what you are trying to achieve in NiFi is possible 🙂

PS: don't try and split a CSV File with million of lines into individual flow files containing 1 line, because you will regret doing that 🙂

[#3]

So my requirement is that I have to take some selected fields from input record, do some computations with them, create a new Record with new fields which are populated with these computed values, and write this new record into output flow file.

I have to do it all in a custom processor I am writing.

[#4]

My attempt so far:

List<RecordField> fields = new ArrayList<>();
RecordField field1= new RecordField("field1", RecordFieldType.STRING.getDataType());
fields.add(field1);
....
....

SimpleRecordSchema schema = new SimpleRecordSchema(fields);
final Record mapRecord = new MapRecord(schema, new HashMap<>());
final List<FieldValue> selectedFields = new ArrayList<>();
//TODO how to create corresponding FieldValue instances from raw values
for (final FieldValue selectedField : selectedFields) {
    mapRecord.setValue(selectedField.getField(), selectedField.getValue());
}

[#5 · Accepted solution]

Well the only advise I can give you is to write your processor and see what errors you have and come back with them. Nobody can write your processor if only you know your requirements.

What I can suggest you though, is to have a look at the following examples, as they might assist you with what you are trying to achieve:
https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-1/ta-p/248922
https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-2/ta-p/249018
https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-3/ta-p/249148