Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache Nifi - How to co-relate IDs in logs to actual Processor names (or parent process groups)

Highlighted

Apache Nifi - How to co-relate IDs in logs to actual Processor names (or parent process groups)

Contributor

Hi all,

I am planning to push Nifi logs to a persistent storage (InfludDB or ELK) and deploy alerts on top of these systems (Like you could count the number or Errors on InfluxDB or Elastic Search and alert (Email) if it reaches a certain threshold).

However, in Nifi logs, I only see IDs and not processor/Processor Group names:

2016-09-03 08:03:34,062 ERROR [Timer-Driven Process Thread-9] o.a.n.processors.standard.MergeContent MergeContent[id=51a435a0-40ca-4254-8fce-efae24c24677] Cannot Defragment StandardFlowFileRecord[uuid=2f3cb5b1-33d3-4251-8bf3-8b6608c9de07,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1472861719946-2897, container=default, section=849], offset=460800, length=0],offset=0,name=askdone_50246.esclf_S.201607190600-0700-0.gz,size=0] because it does not have an integer value for the fragment.index attribute; routing 4161 FlowFiles to failure

To generate more meaningful Email alerts, I would like to store Processor Name (Or ProcessorGroupName) rather then id=51a435a0-40ca-4254-8fce-efae24c2467. Hence, email alerts would specify the name of the Processor rather then ID.

What would be the bast way to co-relate ID with processor/progess group names?

Thanks

Obaid

7 REPLIES 7
Highlighted

Re: Apache Nifi - How to co-relate IDs in logs to actual Processor names (or parent process groups)

Contributor

Hello,

Did this ever get answered?

Highlighted

Re: Apache Nifi - How to co-relate IDs in logs to actual Processor names (or parent process groups)

Rising Star

@Timothy Spann @Pierre Villard i have exactly the same problem, ideally i would like the Process group names in the logs so that they can be extracted to a seperate logfile, having one logfile per flow/process group eventually

Highlighted

Re: Apache Nifi - How to co-relate IDs in logs to actual Processor names (or parent process groups)

Super Guru

You should be able to grab the ids.

I think ideally you parse the logs with Apache NiFi and insert them as Hive tables then you can do group by and other SQL to order.

You should be able to map with the REST API

https://nifi.apache.org/docs/nifi-docs/rest-api/

I would love all the processor group and processors with their id into a Hive lookup table. Then you can join them in queries

Highlighted

Re: Apache Nifi - How to co-relate IDs in logs to actual Processor names (or parent process groups)

Rising Star

@Timothy Spann Thanks for the response, just saw the relevant JIRA tickets, hope you/someone will find some time at some point to implement this!

Further, another useful thing would be to be able to grab at least the logs created by Cron-triggered flows, but i can only see "TimerDriver Scheduling" info in the created logs. Cron-triggered flows create no logs?

Highlighted

Re: Apache Nifi - How to co-relate IDs in logs to actual Processor names (or parent process groups)

Super Guru

there is some talk about this. if you want to add a JIRA or update one you found.

The code to lookup names from the REST API is pretty straight forward. You could use this to enhance the logs. I like the use case of augmenting the logs and use Hive or HBase/Phoenix to query them.

Highlighted

Re: Apache Nifi - How to co-relate IDs in logs to actual Processor names (or parent process groups)

Super Guru
  1. pip install nipyapi
    python
    Python 2.7.12 (default, Jun 29 2016, 14:05:02)
    ...
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from nipyapi.canvas import *
    >>> {i.status.id:i.status.name for i in list_all_processors()}
    {'c27747fe-0160-1000-3213-c5520cc523da': 'root_AttributeRollingWindow', 'c28aca06-0160-1000-2604-f1de5dec00c4': 'root_AttributesToJSON', '30b58a25-0160-1000-d637-20de36a133fb': 'test_ListenSyslog'}
    

Re: Apache Nifi - How to co-relate IDs in logs to actual Processor names (or parent process groups)

It'll also work for process groups:

{i.status.id:i.status.name for i in canvas.list_all_process_groups()}
{'4d5dcf9a-015e-1000-097e-e505ed0f7fd2': 'NiFi Flow', '30b4a370-0160-1000-10e5-ccfb95cefcec': 'test'}
Don't have an account?
Coming from Hortonworks? Activate your account here