Thanks @mgilman That was helpful. So for ReportingTask, how frequent should I run it? Lets say I am running every 10 seconds, and we receive 100 errors/Warn/info bulletins every minute (just for example), do you think we might loose messages, are we guaranteed to receive all bulletins? ... View more

Thanks mgilman: Yes that is what I am trying to achieve, a ReportingTask that is fetching events from BulletinRepository. The issue is that it always returns at-max 5, so I cannot report all arrors (assume you have more then 5 errors for a processor etc) so which API should I use to extract all bulletins? essentially, how to do "exfil the bulletins as they occur"? It would be great if you could elaborate, thanks ... View more

@Bryan Bende: I wrote a ReportingTask to retrieve messages from BulletinRepository and report Error/Warn/Info metrics, however the repository would always return at-max 5 messages per component no matter how I use the API 😞 (https://community.hortonworks.com/questions/72411/nifi-bulletinrepository-api-returns-maximum-5-bull.html#answer-72488) Do you think this might be by design? So would you recommend Nifi REST api or should I look for any other solution (start monitoring actual logs etc) ... View more

Ok, actually I verified that I am still getting total of 5 messages, no matter how many times I call it, apologies ! So now, I need to re-think how I monitor errors. Do you think through Rest API I would be able to get all error messages? Or should I monitor logs for errors? ... View more

UPDATE: FOLLOWING SOLUTION DOSENT WORKS : We always get at-max 5 messages, Thanks @Timothy Spann: Here is what I found out (there is a bulletin id maintained for each bulletin messages, and it is always increasing. By using .after(id) I am able to fetch all messages in repeated calls: /** * Retrieve bulletins from ButtetinRepository * These bulletin messages are used to generate system health metrics (Errors/Warns/Info) * @param context * @param previousBulletinId * @param maxBulletins * @return */ public static List<Bulletin> findBulletins(ReportingContext context, ComponentType componentType, long previousBulletinId, final int maxBulletins){ ArrayList<Bulletin> bulletinsList = new ArrayList<>(); BulletinRepository repository = context.getBulletinRepository(); int bulletinsFound = 0; do{ bulletinsFound = 0; final BulletinQuery queryProcessor= new BulletinQuery.Builder().sourceType(componentType).after(previousBulletinId).build(); List<Bulletin> bulletinsThisQuery = repository.findBulletins(queryProcessor); if(bulletinsThisQuery != null && bulletinsThisQuery.size() > 0){ bulletinsFound = bulletinsThisQuery.size(); previousBulletinId = bulletinsThisQuery.get(0).getId(); /** Retrieve bulletin id*/ bulletinsList.addAll(bulletinsThisQuery); } } while(bulletinsFound > 0 && bulletinsList.size() < maxBulletins); return bulletinsList; } ... View more

Following what Bryan Bende mentioned (in the case of a cluster), You need to make sure all cluster nodes are a part of the policy. In my case, I created a new Group 'Cluster' and added all the nodes in this group. Then I went ahead and added this Group to a processor group (added this group for pilicies: view the data and modify the data) ... View more

Awesome, thanks for the pointers, ... View more

@Joshua Adeleke : Yes I was able to send Email alerts through PutEmail (some time back), however I dont use it for alerts (actually still looking for a better solution). Current approach: I implemented a ReportingTask, to send metrics to InfluxDB (particularly success/failure connection metrics) and use Capacitor for alerts (and you could use any other system to monitor metrics). So for example you could issue alerts if there are flowfiles landing on any Processor's failure channel (dosent works all the time since some processors dont have failure relationships). However, for a more better approach, checkout Bryan's comments above ! ... View more

Thanks a lot @Bryan Bende for sharing your thoughts, - Could you also recommend how we should monitor dataflows for detecting all failures (Other then PutEmail, would you also recommend monitoring Nifi Logs, or do you think PutEmail is a good enough solution)? - Another idea I wanted to discuss/share: Write a reporting task, and report failures/errors to configured Email/Slack etc, this way you would not need to hookup PutEmail with each processor (considering you have many processors, connecting all with PutEmail make it look complicated/complex). by default, you could get alerts for any failure without configuring/changing flows, Any thoughts ? Thanks again Obaid ... View more

Thanks, that was very helpful, ... View more

sure, no problem ... View more

Great, Thanks for your response, Do you think that there is a relationship with Cores and RAM, meaning if you have X cores then you should have X+ RAM etc, is there any dependency or good practice? We can think of minimum requirements, assuming we will be running a lot of light-weight flows (batch, scheduled). I mean, more cores will let us run more flows, so just thinking if 32GB RAM will be enough for 20cores if I go for HDF2.x.x. Say in the future all 20cores become busy, then would RAM be an issue? Thanks ... View more

Thanks a lot @Attila Kanto for a detailed response, Let me ask another cost related question, which is an important factor for making a decision on which technology to use: How would you compare EMR vs Cloudbreak (or Hortonworks Data Cloud) in-terms of cost? Obaid ... View more

Thanks @Dominika B, Thanks for sharing the link, seems interesting. So I have a very basic question: Amazon EMR lets you launch manage Hadoop and Spark clusters, so what would be the benefit of using Hortonworks cloud vs just using EMR? Thanks Obaid ... View more

Thanks a lot @Matt Burgess and @Pierre Villard for a quick response, ... View more

@Sam Hjelmfelt So far I have not being able to find a feasible way for sending alerts through Nifi cluster, and am curious to know how should I deploy alerts in production Nifi cluster. Thanks ... View more

Thanks @Sam Hjelmfelt for your reply, Yes if the data lands on Primary node, PutEmail works as expected. However id the data lands on a Slave node, no Email is generated and flowfiles get stuck on connection for ever (i.e Slave nodes are not able to talk to primary node). Following is an example flow (template is attached, please check it out): In the below dataflow, we generate flow files and run MurgeContent (every 20 seconds) and then pass the result on to two PutEmail processors in parallel. First PutEmail is running on Primary, where as the second PutEmail processor is running on all Slave nodes (Timer event). For PutEmail on primary, it seems like for 9 generated files, only 1 got processed where as 8 got stuck on the connection (seems like slave nodes not able to talk to primary). Second PutEmail worked just fine i.e it processed all 9 flowfiles. So, is there a way to generate 1 Email alert if a processor fails in a cluster? PS: putemaillimitations.xml ... View more