How do I configure Metricbeat to pass its default system metrics data through NiFi to Elasticsearch without half the metrics disappearing / not properly updating in Elasticsearch and therefore not appearing in Kibana?
- Without NiFi, I can easily configure Metricbeat to use a remote Kibana server and have it's output to go directly to Elasticsearch. In this configuration, all metrics show up as expected in Kibana.
- I'm trying to do the same thing through NiFi. I changed the metricbeat.yml to output to Logstash at an open port and, in NiFi, I ingest that with the ListenBeats processor on the same port. That data is then passed to the PutElasticsearcHttp processor for the correct metricbeat-* index.
-- After this, when viewing the data in Kibana's "Metricbeat host overview" dashboard, the device still shows connected, but only populates some of the expected metrics.
-- Basically, only the network metrics and total processes continue to be updated through NiFi. The list of running processes and most other displays just stop updating.
-- If I revert Metricbeat's config to output directly to Elasticsearch again, all metrics work normally again
Any idea why Metricbeat Logstash output running through NiFi then Elasticsearch is different than Metricbeat outputting directly to Elasticsearch?
Is there a way to fix or compensate for the differences to make the NiFi connection work the same when viewed in Kibana?