Thanks that worked ... View more

@Shu I used couple of replace text processor to fix it. All good. Thanks ... View more

Thanks very much @Shu ... View more

Thanks for your input. As my question was related to the previous ones,I put it there, but I think you're right. I started a new one. https://community.hortonworks.com/questions/210117/elasticsearch-dont-show-all-of-the-snort-s-index-1.html ... View more

@Faisal Durrani Use UpdateRecord processor before PutHBaseRecord Processor and create a new field i.e concatenated with PK's then in PutHBaseRecord processor Record Reader add the newly created field in the Avro Schema so that you can use the concatenated field as row identifier. row_id //newly created field name concat(/pk1,/pk2) //processor gets pk1,pk2 field values from record and concatenates them and keep as row_id. By using UpdateRecord processor we are going to work on chunks of data and very efficient way of updating the contents of flowfile. For more reference regarding update record processor follow this link. ... View more

@Sam LL You can do this in NiFi by using Record oriented processors (ConvertRecord). Using Convert Record processor: input json: { NAME: "xxx", CITY: "yyy", AGE:"00", ZIPCODE: "12345", ADDRESS: " "} Use ConvertRecord processor with Json Tree reader as controller service and give your matching avro schema and enable controller service. Avro Schema: { "namespace": "nifi", "name": "balances", "type": "record", "fields": [ {"name": "NAME", "type": ["null", "string"]}, {"name": "CITY", "type": ["null", "string"]}, {"name": "AGE", "type": ["null", "string"]}, {"name": "ZIPCODE", "type": ["null", "string"]}, {"name": "ADDRESS", "type": ["null", "string"]} ] } By using the configured reader convertRecord processor is able to read your incoming data. To write in json format configure/enable JsonRecordSetWriter controller service and give the matching avro schema(change the order of the fields while writing) AvroSchema: { "namespace": "nifi", "name": "balances", "type": "record", "fields": [ {"name": "NAME", "type": ["null", "string"]}, {"name": "AGE", "type": ["null", "string"]}, {"name": "ADDRESS", "type": ["null", "string"]}, {"name": "CITY", "type": ["null", "string"]}, {"name": "ZIPCODE", "type": ["null", "string"]} ] } Now while writing the output data processor writes with the matching avro schema with ordering the fields in same way Output: [{"NAME":"xxx","AGE":"00","ADDRESS":" ","CITY":"yyy","ZIPCODE":"12345"}] Refer to this link for configure/enabling ConvertRecord processor. - If the Answer helped to resolve your issue, Click on Accept button below to accept the answer. ... View more

@Suhas Reddy Use Replace Text processor to change the contents of flowfile. Search Value ("_id":.*?,).*("eyeColor":.*") Replacement Value $1$2 Character Set UTF-8 Maximum Buffer Size 1 MB //change the value according to flowfile size Replacement Strategy Regex Replace Evaluation Mode Entire text Input: {"_id": "5b42fe8f7f663540330b3bdc","index": 0,"guid": "60358c95-e50c-4f5e-ad48-00d9e1f9a849","isActive": true,"balance": "$2,483.56","picture": "http://placehold.it/32x32","age": 32,"eyeColor": "green"} Output: {"_id": "5b42fe8f7f663540330b3bdc","eyeColor": "green"} If your required output flowfile is {"eyeColor": "green","_id": "5b42fe8f7f663540330b3bdc"} then change the replace text configs to Search Value ("_id":.*?),.*("eyeColor":.*") Replacement Value $2,$1 In addition to achieve the same case by using record oriented processors(Convert Record (or) Update Record) if you know the schema of your json message then use ConvertRecord/UpdateRecord processor to read your incoming data with jsontree reader controller service and configure the only required fields(eyecolor,_id) in JsonRecordSetwriter so that your output flowfile will going to have only the configured fields and this processor works with array of json messages Refer to these links regarding convert record and update record processors. - If the Answer helped to resolve your issue, Click on Accept button below to accept the answer. ... View more

Thanks Shu.. ... View more