@Shantanu kumar 1/1 is nothing but daily it is similar to * i.e 0 0-1 10 * * ? * (or) 0 0-1 10 1/1 * ? *. It runs daily at 10:00,10:01 am, you can create/check all cron tab expressions in the following link. http://www.cronmaker.com/ For more details about cron expression https://community.hortonworks.com/questions/63513/helping-setting-up-cron-based-nifi-processor.html https://nifi.apache.org/docs/nifi-docs/html/user-guide.html ... View more

@Mark McGowan You can do merging by using Correlation attribute name property in merge content processor, to use this property we cannot use convert record processor because each flowfile we need to keep date,hour attrbiutes as one attribute and use the attribute in merge content correlation attribute name. Then the merge content processor will merge with same attribute flowfiles into one flowfile. Please refer to below links for more details about correlation attribute usage https://community.hortonworks.com/questions/161827/mergeprocessor-nifi-using-the-correlation-attribut.html https://community.hortonworks.com/questions/55926/using-the-mergecontent-processor-in-nifi-can-i-use.html https://community.hortonworks.com/questions/87178/merge-fileflow-files-based-on-time-rather-than-siz.html For reference Convert record xml 178086-json-to-csv.xml ... View more

@Shantanu kumar Yes, if you change threshold value to 1 min then we need to run Monitor Activity processor for 2 mins i.e 0 0-1 10 1/1 * ? *, for the first minute checks flow's activity for 1 minute and second minute(10:01am) to trigger a inactivity flowfile. Now processor will run at 10:00am with this configuration window will be 1 min and checks if there are any flowfiles and 10:01 am(sends an flowfile for inactivity relation if there are no flow files). ... View more

@Sami Ahmad Query Database table processor stores the last state value(if we mention Maximum-value Columns) and run incrementally based on your run schedule.If no columns are provided in Maximum value columns then all rows from the table will be considered, which could have a performance impact. NOTE: It is important to use consistent max-value column names for a given table for incremental fetch to work properly. So let's consider you have some incremental column in your source table and you have mentioned incremental column as Maximum value column in query databasetable processor. For the first time this processor pulls all the records and updates the state (consider your last state is 2018-03-22 19:11:00), for the next run this processor only pulls the columns that have incremental column value more than stored state value i.e 2018-03-22 19:11:00. if there are no records that have new updated incremental column value then this processor won't return any flowfiles(because no records got added/updated). for more reference about querydatabasetable https://community.hortonworks.com/articles/51902/incremental-fetch-in-nifi-with-querydatabasetable.html In your screenshot you have connected success and failure relationships again to puthdfs processor ,even if the flowfiles has successfully stored into hdfs then those flowfiles route to success relation and you have looped back it to same processor again which will try to keep the same file again and again if the conflict resulution strategy is set to fail then you will end up with filling logs with this error. it's better to use retry loop for failure relation and for success relation just auto terminate (or) drag a funnel and feed success relation to funnel. Retry loop references https://community.hortonworks.com/articles/51902/incremental-fetch-in-nifi-with-querydatabasetable.html There are some links which can gives insights how put hive streaming processor works https://community.hortonworks.com/questions/84948/accelerate-working-processor-puthivestreaming.html https://issues.apache.org/jira/browse/NIFI-3418 https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest ... View more

@umang s That's expected behavior from Hbase table because we are having unique Row Key for each record in Hbase table now. But your data that you are writing to this category1 table is duplicated data and Hbase only overwrites the existing data if it found same Row key already exists in the table. But in your case we are using UUID i.e unique id for the each flowfile in NiFi, so we will have unique id for each flowfile (although the content of the flowfile is same). ... View more

@umang s Are you using UUID as Row Identifier? Could you please share your PutHbaseCell processor configs.. ... View more

@umang s I think your input json messages are enclosed in an array [] like [{"id":"1334134","name":"Apparel Fabric","path":"Arts, Crafts & Sewing/Fabric/Apparel Fabric"},{"id":"412","name":"Apparel Fabric","path":"Arts, Crafts & Sewing/Fabric/Apparel Fabric"}] In this case use Split Json processor before PutHbasecell processor with below configs Use Splits relation from splitjson processor to PutHbase cell processor in this case Split json processor splits array of json messages to individual messages. Input:- [{"id":"1334134","name":"Apparel Fabric","path":"Arts, Crafts & Sewing/Fabric/Apparel Fabric"},{"id":"412","name":"Apparel Fabric","path":"Arts, Crafts & Sewing/Fabric/Apparel Fabric"}] Output:- flowfile1:- {"id":"1334134","name":"Apparel Fabric","path":"Arts, Crafts & Sewing/Fabric/Apparel Fabric"} Flowfile2:- {"id":"412","name":"Apparel Fabric","path":"Arts, Crafts & Sewing/Fabric/Apparel Fabric"} ... View more