@ankurkapoor_wor Hi, Even I am facing the same issue as @mqureshi. I am trying to fetch data from SQL server in Avro fomat through NiFi and load it to Redshift through copy command. But the generated Avro file is converting the date and timestamp datatypes to string because of which copy command is loading all NULL values in the target table. So I tried to follow your approach, In my case I'm using ExecuteSQLRecord processor to fetch the data from SQL server and writing it to json format and then trying to convert it to Avro format using ConvertJsonToAvro processor but then I am unable to parse Record Schema. Could you please help me also to resolve this issue.  Thanks in advance! Anusha ... View more

Conceptually the flow might look something like this: Thank you, Matt - When an "Answer" addresses/solves your question, please select "Accept" beneath that answer. This encourages user participation in this forum. ... View more

@rajat puchnanda Method 1: You can use Query record processor and add new dynamic property with value as sql query like select id,count(*) from flowfile group by id and the processor will run the sql query on the flowfile content then gives the result as output flowfile. Refer to this link to get more details regarding configuration of Query Record processor. Method2: If you are thinking to Group by all like records then you can use Partition Record processor and specify the record path that you want to group by then the processor will groups all the like records into inidividual groups. Refer to this link to get more details regarding configuration of Partition Record processor. ... View more

@rajat puchnanda Here is screenshot showing the menus and icons that go along with the above explanation: Thanks, Matt ... View more

@rajat puchnanda Merge a group of flowfile (or) records is possible with MergeContent/MergeRecord processors. Example: if flowfile(ff1) having 123 records then ff2 having 345 by using mergecontent/record processors we can merge these flowfiles in to one like 123345. Merge means combining the group of records/flowfiles(union all) ,if you want to remove duplicates(i.e 3 is duplicate record) from the combined record flowfile content then you can use QueryRecord Processor with row_number window function to eliminate duplicates. This scenario is possible with NiFi without using lookup record processors. But as you mentioned in one of the answer Scenario2: InputFile 1 deptid firstname lastname 1 Aman Sharma 2 Raman Verma InputFile 2 deptid salary email 1 20000 abc@gmail.com 2 30000 bgf@gmail.com OutputFile(By merging file1 and file2):- deptid firstname lastname salary email 1 Aman Sharma 20000 abc@gmail.com 2 Raman Verma 30000 bgf@gmail.com This is not possible with MergeContent/Record but you can try with QueryRecord processor by implementing group and collect as set (or) some sort of sql logic in queryrecord processor to transpose the data into your desired format.This query Would be intensive if you are doing on larger number of records. ... View more

Hi @Vikas Singh , SplitJson Processor is used to split Json Array and EvaluateJsonPath is used to extract Json fields as attribute or content. In your case: Step 1:Use EvaluateJsonPath Processor to extract info fields of Json. For example: is you want to extarct info fileds: .$info.seed,.$info.page,.$info.version,.$info.results and save it as Flowfile Attribute. Step 2:Use SplitJson Processor :$.results Step 3:USe EvaluateJsonPath Processor to extract Json Array fields. ... View more