Member since
03-29-2018
7
Posts
0
Kudos Received
0
Solutions
06-06-2018
10:30 PM
@Bryan Bende Similar question along same lines. I want to report metrics from nifi at the processor level and processor group level and I was able to get this metrics from the rest API, but I am when I get aggregate snapshot I am worried that all nodes will send exactly the same metrics which might show wrong values after aggregation in graphite or I planned to use node level metrics snapshot, if I can compare the node which is sending data can only send its specific metrcis not all nodes metrics. Is this possible ? or there is a better solution to get fine level metrics ?
... View more
06-06-2018
04:16 PM
statuses variable not in code is a copy paste error. I did add more logging and I see the statuses is empty which is why nothing logs.
... View more
06-05-2018
02:43 PM
I have implemented a custom reporting task, very simple one to start with, which just logs all details that come in the context of `onTrigger` method. I see the nar being unpacked and the reporting task being started when I start from the settings, but nothing logs after that. My Run Schedule is 30 sec (I also tried changing this to 1 mins and 5 mins), but nothing ever logs, but reporting task shows as running. No errors in logs whatsoever. Only log lines I see in nifi-app.log are these org.apache.nifi.metrics.reporting.task.MetricsReportingTask
org.apache.nifi:nifi-metrics-reporting-nar:1.6.0-SNAPSHOT || ./work/nar/extensions/nifi-metrics-reporting-nar-1.6.0-SNAPSHOT.nar-unpacked
org.apache.nifi.reporting.ambari.AmbariReportingTask
org.apache.nifi:nifi-ambari-nar:1.6.0-SNAPSHOT || ./work/nar/extensions/nifi-ambari-nar-1.6.0-SNAPSHOT.nar-unpacked
org.apache.nifi.reporting.datadog.DataDogReportingTask
org.apache.nifi:nifi-datadog-nar:1.6.0-SNAPSHOT || ./work/nar/extensions/nifi-datadog-nar-1.6.0-SNAPSHOT.nar-unpacked
org.apache.nifi.controller.ControllerStatusReportingTask
org.apache.nifi:nifi-standard-nar:1.6.0-SNAPSHOT || ./work/nar/extensions/nifi-standard-nar-1.6.0-SNAPSHOT.nar-unpacked
org.apache.nifi.controller.MonitorDiskUsage
org.apache.nifi:nifi-standard-nar:1.6.0-SNAPSHOT || ./work/nar/extensions/nifi-standard-nar-1.6.0-SNAPSHOT.nar-unpacked
com.blah.cloud.datamanager.tasks.CSReportingTask
com.blah.cloud.datamanager:nifi-csreporting-nar:1.0-SNAPSHOT || ./work/nar/extensions/nifi-csreporting-nar-1.0-SNAPSHOT.nar-unpacked<br>2018-06-05 04:03:29,056 INFO [Timer-Driven Process Thread-9] o.a.n.c.s.TimerDrivenSchedulingAgent CSReportingTask[id=ce15e357-0163-1000-0000-0000139a1779] started.
My Reporting Task code public class CSReportingTask extends AbstractReportingTask {
@Override
public void onTrigger(ReportingContext context) {
EventAccess eventAccess = context.getEventAccess();
ProcessGroupStatus groupStatus = eventAccess.getControllerStatus();
Collection<ProcessorStatus> processorStatus = groupStatus.getProcessorStatus();
String nodeId = context.getClusterNodeIdentifier();
for (ProcessorStatus status : statuses) {
String groupID = status.getGroupId();
String name = status.getName();
String execNode = status.getExecutionNode().name();
String id = status.getId();
String type = status.getType();
String runStatus = status.getRunStatus().name();
getLogger().info("id: {}; name: {}; groupID: {}; execNode: {}; type: {}; runStatus: {}; nodeID: {}", new Object[]{id, name, groupID, execNode, type, runStatus, nodeId});
}
}
}
... View more
Labels:
- Labels:
-
Apache NiFi
04-26-2018
11:36 PM
@Shu Never noticed there is another publish processor for Kafka, PublishKafka_0_11 seems to be working fine. Thanks for the answer!
... View more
04-26-2018
10:16 PM
I am trying to fetch files (very small files, but millions of them) from S3 and put it into kafka and Kafka consumer will read each file from Kafka and process the data inside the file. I noticed that publish kafka 0_11 processor forces to use record reader/writer, but for data i am trying to put into Kafka, I am not sure what will be my schema. Can I just avoid this and NiFi not try to parse this data and just pass through the flow file content and publish to kafka?
... View more
Labels:
- Labels:
-
Apache NiFi