@Jeong Jong-Duk Using external Database: There are LookupRecord/LookupAttribute/SelectHiveQL/FetchHbase processor available in NiFi as you can setup lookupservice then you are able to retrieve the avro schema from Database. (or) Using NiFi processors: You can add Avro schema attribute by using NiFi UpdateAttribute Advanced usage by checking the filename you can add the appropriate avro schema as the flowfile attribute then in your Record Reader/Writer controller services uses that schema. This is basically a case statement in UpdateAttribute processor based on filename we are making decision which schema we need to add to the flowfile, you need to define schema with avro logical type(timestamo) and configure record reader/writer controller services with the timestamp/time/date formats so that processor can read/write the flowfile content to postgressql. Refer to this link to configure and usage of UpdateAttribute Advanced usage. ... View more

@Mahendra Hegde AFAIK we cannot extract only the value using Jolt transform. To achieve this extract the array from the json using `EvaluateJsonPath` processor EvaluateJsonPath Configs: Now using ReplaceText processor replace ",[,],space from the extracted value using NiFi expression language. ReplaceText Configs: Search Value (?s)(^.*$) Replacement Value ${country_list:replaceAll('(\[|\]|"|\s)','')} Character Set UTF-8 Maximum Buffer Size 1 MB Replacement Strategy Always Replace Evaluation Mode Entire text Input: { "organization_id": "life-360", "message_type_id": "MSG01", "consent_required": "1", "country_list": [ "GB", "IT", "BG" ] } Output: GB,IT,BG Flow: 1.other processors 2.EvaluateJsonPath //extract the json array as attribute value 3.ReplaceText //replace the content of flowfile with extracted value 4..other processors ... View more

@Kei Miyauchi If you want to trigger minor/major compactions from NiFi then feed the Success relationship PutHiveStreaming processor to Replace Text process and configure ReplaceText processor with below configs: replacement strategy always replace and replacement value as alter table <db_name>.<table_name> compact 'minor'; Then using PutHiveQL processor execute the minor compaction. Flow: --other processors --> PutHivestreaming --> ReplaceText processor --> PutHiveQL By following this way we are initializing minor compaction from NiFi. Take a look into this SupportKB related to Minor Compactions are not working in Hive, set the recommended global configs to make minor compactions work. ... View more

@Mahendra Hegde it's possible in NiFi but it depends on how frequently you are going to restart NiFi instance. Use GenerateFlowFile (or) other processors and schedule the processor as shown below With the above schedule GenerateFlowFile processor going to run once we start processor and next run will be after 1111111110 sec (or) when you restart NiFi instance. For your case adjust the Run Schedule time based on your NiFi restart time. ... View more