@Nidutt you should be able to use NifI expression language in the flow to change date int to ISO timestamps.    Here is a template you can use that shows many examples of timestamp formatting:   https://github.com/steven-matison/NiFi-Templates/blob/master/Working_with_TimeStamps.xml   I think you may find that nifi attributes remain strings in your flow without a strict date type, after all an ISO timestamp is really a string, your end point database just knows it is a "timestamp"....       If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

I have flow files with different dimensions. But they have a common id column. I want to use that to join the flowfiles and pick specific columns. How can I use mergeContent in this case? ... View more

Thanks for your point and if you got time, please read the solution that i found out somewhere in the internet. ... View more

In reality despite ConsumeIMAP is stateless, the IMAP server holds /Recent flag that is system read-only flag (theoretically within IMAP session, but how Gmail detect unique session is a BlackBox and I was not able to manage it with NiFi ConsumeIMAP client).   If you want to re-fetch all your historical emails I recommend using ConsumePOP3 NiFi component, with the following setting on Gmail server: Gmail - settings - all - forwarding - Enable POP for all mail After performing such historical fetch you can disable ConsumePOP3 and continue to work with ConsumeIMAP to consume new messages. ... View more

@Mondi you should be able to enable the HBASE plugin by editing the hue.ini file from your admin console and telling it the HBASE Thrift Server.     Reference https://docs.gethue.com/administrator/configuration/connectors/#hbase for the information below.   HBase Specify the comma-separated list of HBase Thrift servers for clusters in the format of “(name|host:port)": [hbase] hbase_clusters=(Cluster|localhost:9090) In the full reference above there are some additional hbase artifacts for impersonization and kerberos. ... View more

Hi @rmtonkin, I think I understood the problem and the solution is currently working for me pretty well. The culprit property here is Scroll Duration in the processor ScrollElasticSearchHttp. The property default is 1m(minute) and by definition it means How long each search context is kept in memory (of ES itself). In my configuration, Page Size is 100, means 100 records per flow file.   Problem(for example): 1. For some reason, fetching a page(*current cycle) took more than 1 minute. 2. 1 minute passed means, since Scroll Duration is set to 1 minute, ES removed that ScrollId(in View State) from memory and no longer recognises it. 3. Nifi processor uses the ScrollId to fetch the next page after completing the *current cycle. 4. Gets 404 because there's no such ScrollId in ES memory.   Solution(working for me): Increase the value of Scroll Duration to 5m and hope each cycle completes within 5 mins(which IMO is a pretty good time).   NOTE: Don't increase the value much or it might be a huge pressure on the ES and might stall it(haven't tried, just a theory).   Hope this solves your problem too. You can try and let me know. Best of luck.   For all others, please understand this is NOT a solution to the original question. This is about how to avoid the scenario mentioned in it.   Cheers. ... View more

Thanks @stevenmatison for your response on this ... View more

@redmonc2   You should update the post with the input data, and a screen shot of your flow, for better responses from your peers.  If you provide this info I will update my response below.   Without being able to see the input data I believe you just need to adjust your flow so that you are breaking up the input data into multiple flowfiles.   For example, if that input data is lines of dates you want to change formats, your flow should split the lines with SplitText, then get each split FlowFiles date to an attribute called date (${date}) with ExtractText that uses regex to get the entire split content to the date attribute.  With an attribute called date in each flowfile,  you can then use your expression language in updateAttribute:   ${date:toDate("ddHHmm:ssMMMyy"):format("yyyy/MM/dd HH:mm:ss")}   Once you have the format correct for each date, you can proceed with the dates downstream as attributes or write them back to the content of the flowfile and merge them together.   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven @ DFHZ ... View more

Too cool man,  great work! ... View more

@dewi  A management pack is 1 or more Custom Ambari Services which are easily added to many different ambari versions via the management pack install command.  A custom service is something you would manually add to a single ambari installation.   So the management pack just makes it easier, more applicable to multiple versions, and stacks. ... View more