@Ron Labau Try to use Cast the fields names to Varchar in your QueryDatabaseTable processor configs in Columns to Return. If the above method doesn't work then Use ConvertRecord processor to read your Incoming Avro data and in Record SetWriter configure the field names to have string datatype. So that we are going have OutputFlowfile in Avro format with string datatype. Refer to this link for configuring/usage of the ConvertRecord processor. ... View more

@Raj ji Merge functionality i.e. if matched then Update/Delete, If not then insert is very efficient if you are running from Hive. We cannot run Update/Delete/Insert statements from NiFi to Hive. in my article i'm creating partitions in NiFi using Partition Record processor and storing them in HDFS directly then By using PutHiveQL processor i'm executing Msck Repair table to add newly created partitions in HDFS directory to Hive table. Try with some different ways using Merge strategy: 1.Create your source table as partitioned then target table also partitioned then use the partitioned column in your Merge statement, as this will boost your hive job to run more efficiently. Set the auto.convert.join property value to false. hive> set Hive.auto.convert.join = false; (or) 2. Create smaller chunks of files in HDFS instead of one big file then when you are running merge use MERGE INTO merge_data.transactions AS T USING (select * from source_table where input__file__name="<HDFS-file-path>") AS S By performing the above input__file__name predicate hive merge will be performed only on the particular file not on the full data set of the table. If you are following this way first you need to extract all distinct filenames from the source table then run merge on each individual file at once. (or) 3.Storing all the data into target table with some timestamp then while processing the data get only the latest record by using `window function.` (or) 4. Store all the data to target table then recreate(by performing update/delete/insert) the table every day once your ingestion is completed. ... View more

@Josh Nicholson To get ClientID and revision number of the processor group make GET RestAPI call for Process group. curl -X GET http://<url>/nifi-api/flow/process-groups/<process-groupid>; This Get method results you the clientid and revision number of the process group so when we are deleting particular process group then we need to pass these values in to the delete rest api call curl -X DELETE '<url>/nifi-api/process-groups/<process-groupid>?version=<version_number>&clientId=<clientid>' Example: i have a process group with id 5eb0e7db-0165-1000-e40b-62a0bae27474 to get clientid and version details: $ curl -X GET http://localhost:8081/nifi-api/flow/process-groups/5eb0e7db-0165-1000-e40b-62a0bae27474 [{"revision":{"clientId":"5ea367bc-0165-1000-74ab-d0e36167547e","version":1},"id":"5eb10684-0165-1000-fee6-01cbff4570a6","uri":"http://localhost:8081/nifi-api/output-ports/5eb10684-0165-1000-fee6-01cbff4570a6"} As you can see my version is 1 so in my Delete call i'm using same client id and version number curl -X DELETE 'http://localhost:8081/nifi-api/process-groups/5eb0e7db-0165-1000-e40b-62a0bae27474?version=2&clientId=5ea367bc-0165-1000-74ab-d0e36167547e' Then the process group will be deleted. if you are creating a shell script then you need to extract the clientid and version store it in a variable and pass those variables in your Delete call, so that you can automate your job using shell script. ... View more

@Sai Krishna Makineni You can use List/Fetch SFTP processors and then Store the Fetched files into HDFS by using PutHDFS processor. Flow: 1.ListSFTP (or) ListFTP //list the files fro remote path 2.Remote Processor group //distribute the load across the cluster 3.FetchSFTP (or) FetchFTP //fetch the files from remote 4.PutHDFS //store the fetched files into HDFS Please refer to this link for more details regards to fetching files from remote path and usage of List/Fetch processors in NiFi. ... View more

@sandra Alvarez You can use RouteText processor Routes textual data based on a set of user-defined rules. Each line in an incoming FlowFile is compared against the values specified by user-defined Properties. The mechanism by which the text is compared to these user-defined properties is defined by the 'Matching Strategy'. The data is then routed according to these rules, routing each line of the text individually. Example: I have an input data as follows: nifi hdf kyjkglhkghfgdsa hdp jdfdf nifi RouteText configs: Route text processor having alot of ways to determine Matching Strategy and i'm using contains strategy and added new dynamic property as required nifi the matched relationship from RouteText processor will give nifi hdf hdp jdfdf nifi Only the above two lines having nifi in them so processor given results as expected. Like this way you can choose any of Matching Strategy and add dynamic properties based on the strategies(i.e match regular expression means your dynamic property will be REGEX). - If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues. ... View more

@Hariprasanth Madhavan PutHiveQL processor is used to: Executes a HiveQL DDL/DML command (UPDATE, INSERT, e.g.). The content of an incoming FlowFile is expected to be the HiveQL command to execute. - If you want to insert data into hive table directly then use PutHiveStreaming processor instead of PutHiveQL. Puthivestreaming processor expects the incoming data in AVRO format and table needs to Transactional enabled, so based on the KafkaConsumer format of data use ConvertRecord processor to Convert the source data into AVRO format then feed the Avro data into PutHiveStreaming processor. Flow: 1.ConsumeKafka 2.ConvertRecord //convert the outgoing flowfile into AVRO format 3.PutHiveStreaming Refer to this link for hive transactional tables and this link for ConvertRecord processor usage. - If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues. ... View more