Member since
07-25-2016
55
Posts
28
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5717 | 07-26-2016 12:31 AM |
12-20-2016
07:40 AM
Thanks @mgilman That was helpful. So for ReportingTask, how frequent should I run it? Lets say I am running every 10 seconds, and we receive 100 errors/Warn/info bulletins every minute (just for example), do you think we might loose messages, are we guaranteed to receive all bulletins?
... View more
12-19-2016
05:02 PM
Thanks mgilman: Yes that is what I am trying to achieve, a ReportingTask that is fetching events from BulletinRepository. The issue is that it always returns at-max 5, so I cannot report all arrors (assume you have more then 5 errors for a processor etc) so which API should I use to extract all bulletins? essentially, how to do "exfil the bulletins as they occur"? It would be great if you could elaborate, thanks
... View more
12-18-2016
08:52 PM
@Bryan Bende: I wrote a ReportingTask to retrieve messages from BulletinRepository and report Error/Warn/Info metrics, however the repository would always return at-max 5 messages per component no matter how I use the API 😞 (https://community.hortonworks.com/questions/72411/nifi-bulletinrepository-api-returns-maximum-5-bull.html#answer-72488) Do you think this might be by design? So would you recommend Nifi REST api or should I look for any other solution (start monitoring actual logs etc)
... View more
12-18-2016
08:46 PM
Ok, actually I verified that I am still getting total of 5 messages, no matter how many times I call it, apologies ! So now, I need to re-think how I monitor errors. Do you think through Rest API I would be able to get all error messages? Or should I monitor logs for errors?
... View more
12-18-2016
08:17 PM
1 Kudo
UPDATE: FOLLOWING SOLUTION DOSENT WORKS : We always get at-max 5 messages, Thanks @Timothy Spann: Here is what I found out (there is a bulletin id maintained for each bulletin messages, and it is always increasing. By using .after(id) I am able to fetch all messages in repeated calls: /**
* Retrieve bulletins from ButtetinRepository
* These bulletin messages are used to generate system health metrics (Errors/Warns/Info)
* @param context
* @param previousBulletinId
* @param maxBulletins
* @return
*/
public static List<Bulletin> findBulletins(ReportingContext context, ComponentType componentType, long previousBulletinId, final int maxBulletins){
ArrayList<Bulletin> bulletinsList = new ArrayList<>();
BulletinRepository repository = context.getBulletinRepository();
int bulletinsFound = 0;
do{
bulletinsFound = 0;
final BulletinQuery queryProcessor= new BulletinQuery.Builder().sourceType(componentType).after(previousBulletinId).build();
List<Bulletin> bulletinsThisQuery = repository.findBulletins(queryProcessor);
if(bulletinsThisQuery != null && bulletinsThisQuery.size() > 0){
bulletinsFound = bulletinsThisQuery.size();
previousBulletinId = bulletinsThisQuery.get(0).getId(); /** Retrieve bulletin id*/
bulletinsList.addAll(bulletinsThisQuery);
}
}
while(bulletinsFound > 0 && bulletinsList.size() < maxBulletins);
return bulletinsList;
}
... View more
12-18-2016
01:12 AM
1 Kudo
Hi, I have written a ReportingTask service for Nifi, where I use 'BulletinRepository.findBulletins(queryProcessor);' to retrieve all bulletins and report metrics regarding number of errors/warns/info. The issue is that I am receiving at-max 5 bulletins/messages per component though more messages are viewable through the Bulletin UI. Following is how I construct and execute my query: BulletinRepository repository = context.getBulletinRepository();
final BulletinQuery queryProcessor= new BulletinQuery.Builder().
sourceType(ComponentType.PROCESSOR).
limit(500).build();
bulletinsList.addAll(repository.findBulletins(queryProcessor));
How do I get ALL the messages from Bulletin repository rather then just receiving 5? Thanks Obaid
... View more
Labels:
- Labels:
-
Apache NiFi
11-29-2016
06:37 AM
Following what Bryan Bende mentioned (in the case of a cluster), You need to make sure all cluster nodes are a part of the policy. In my case, I created a new Group 'Cluster' and added all the nodes in this group. Then I went ahead and added this Group to a processor group (added this group for pilicies: view the data and modify the data)
... View more
11-15-2016
06:17 AM
Awesome, thanks for the pointers,
... View more
11-15-2016
06:13 AM
@Joshua Adeleke : Yes I was able to send Email alerts through PutEmail (some time back), however I dont use it for alerts (actually still looking for a better solution). Current approach: I implemented a ReportingTask, to send metrics to InfluxDB (particularly success/failure connection metrics) and use Capacitor for alerts (and you could use any other system to monitor metrics). So for example you could issue alerts if there are flowfiles landing on any Processor's failure channel (dosent works all the time since some processors dont have failure relationships). However, for a more better approach, checkout Bryan's comments above !
... View more
11-15-2016
12:13 AM
Thanks a lot @Bryan Bende for sharing your thoughts, - Could you also recommend how we should monitor dataflows for detecting all failures (Other then PutEmail, would you also recommend monitoring Nifi Logs, or do you think PutEmail is a good enough solution)? - Another idea I wanted to discuss/share: Write a reporting task, and report failures/errors to configured Email/Slack etc, this way you would not need to hookup PutEmail with each processor (considering you have many processors, connecting all with PutEmail make it look complicated/complex). by default, you could get alerts for any failure without configuring/changing flows, Any thoughts ? Thanks again Obaid
... View more