Support Questions
Find answers, ask questions, and share your expertise

about - NiFi Monitoring - ControllerStatusReportingTask with Ambari

Contributor

Hi guys!

Please, see this tutorial - https://community.hortonworks.com/articles/79849/nifi-monitoring-controllerstatusreportingtask-with.... before you read my question. ()

I tried this method but it doesn't works. A separate log file is not even created. I've all ready tried many more modifications of this configuration but still nothing. What I am doing wrong? The key moment is - "A separate log file is not even created".

PS

I've already asked this question inside this tutorial but still I have no answer.

Thanks for help!

20 REPLIES 20

Re: about - NiFi Monitoring - ControllerStatusReportingTask with Ambari

Hi @Alex Coast

I followed the same tutorial and had issues getting the file generated too, initially.
Had to make a few minor adjustments, and following works well for me;

<appender name="5MINUTES_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> 
<file>${org.apache.nifi.bootstrap.config.log.dir}/5minutesStatistics.log</file> 
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> 
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/5minutesStatistics_%d.log</fileNamePattern> 
<maxHistory>5</maxHistory> 
</rollingPolicy> 
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> 
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern> 
</encoder> 
</appender> 
<logger name="org.apache.nifi.controller.ControllerStatusReportingTask" level="INFO" additivity="false"> 
<appender-ref ref="5MINUTES_FILE" /> 
</logger> 

PS. If the above helps you to resolve your issue please take a moment to log in and mark the answer as accepted 🙂

Re: about - NiFi Monitoring - ControllerStatusReportingTask with Ambari

Contributor

Hi @Jonathan Sneep. Thanks for your answer, but I've allready did like that but still nothing.

My steps:

1. Add the config to logback.xml

2. Restart service nifi

3. Do something in NiFi (like start some dataflow)

And the result is nothing - No new log file with separate logs which I configure

What I'm doing wrong?

Re: about - NiFi Monitoring - ControllerStatusReportingTask with Ambari

Master Guru

@Alex Coast

What version of NiFi are you running?

Thanks,

Matt

Re: about - NiFi Monitoring - ControllerStatusReportingTask with Ambari

@Alex Coast

What version of HDP/HDF are you using?
And just to be sure, you are adding this via Ambari right? Under Advanced nifi-node-logback-env of the NiFi config?
If you share your HDP/HDF version and your current nifi-node-logback-env (logback.xml) I will try to reproduce your scenario exactly 🙂

Re: about - NiFi Monitoring - ControllerStatusReportingTask with Ambari

Contributor

@Jonathan Sneep

No, via Ambari it's too simple :))) I'm trying via command line hard code 🙂 Maybe that's why it doesn't work. If so how can I configure logback.xml by my method (via cli) ? Thanks!

Re: about - NiFi Monitoring - ControllerStatusReportingTask with Ambari

Contributor

I have no any Ambari that's why I need to do this configuration without it. Is anybody know how to do this right?

Re: about - NiFi Monitoring - ControllerStatusReportingTask with Ambari

Master Guru

@Alex Coast

-

I just enabled what you are trying to do using Apache NiFi 1.8 without any issue.

             <appender name="STATUS_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
            <file>/var/log/nifi/nifi-status.log</file>
            <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!--
            For daily rollover, use 'user_%d.log'.
            For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
            To GZIP rolled files, replace '.log' with '.log.gz'.
            To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>/var/log/nifi/nifi-status_%d.log</fileNamePattern>
            <!-- keep 30 log files worth of history -->
            <maxHistory>2</maxHistory>
            </rollingPolicy>
            <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
            </encoder>
            </appender>

-

I then added separate loggers for for Connections and Processors

            <logger name="org.apache.nifi.controller.ControllerStatusReportingTask.Connections" level="INFO" additivity="false">
            <appender-ref ref="STATUS_FILE"/>
            </logger>
            <logger name="org.apache.nifi.controller.ControllerStatusReportingTask.Processors" level="INFO" additivity="false">
            <appender-ref ref="STATUS_FILE"/>
            </logger>

-

I successfully see my nifi-status.log being created and containing proper output from these loggers.

-

1. Double check your logback.xml for any formatting mistakes. If a formatting mistake is found, you will need to restart nifi after fixing that formatting mistake. For example: forgetting to close a code block like "<logger" without corresponding "</logger>".

2. Make sure you do not have any comment blocks that were started and not properly ended.
This starts a comment block: <!--

This ends that comment block: -->

3. Make sure there are no comment blocks started within existing comment blocks.

4. Make sure the user running your NiFi service has permissions to create log in configured directory.

-

93267-screen-shot-2018-11-15-at-95240-am.png

Thank you,

Matt

Re: about - NiFi Monitoring - ControllerStatusReportingTask with Ambari

Contributor

Hi @Matt Clarke !

Thank you for your detailed answer. I tried 3 or 4 different ways to configure this logs! And yes?I always check from 5 to 10 times my configs before restart nifi service.

Here is my permissions and config(bottom):

I still have no any file with this configured separate status logs.

93634-logback1.png

93635-logback2.png

           <!-- Test configuration for ControllerStatusReportingTask 2018-11-07 by AL -->


    <appender name="STATUS_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-status.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>/var/log/nifi/nifi-status_%d.log</fileNamePattern>
            <!-- keep 5  log files worth of history -->
            <maxHistory>5</maxHistory>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>
    </appender>



    <!-- Logger to redirect controller status/reporting task Connections output to a different file. -->
    <logger name="org.apache.nifi.controller.ControllerStatusReportingTask.Connections" level="INFO" additivity="false">
        <appender-ref ref="STATUS_FILE"/>
    </logger>
    <logger name="org.apache.nifi.controller.ControllerStatusReportingTask.Processors" level="INFO" additivity="false">
        <appender-ref ref="STATUS_FILE"/>
    </logger> 

Re: about - NiFi Monitoring - ControllerStatusReportingTask with Ambari

Master Guru

@Alex Coast

You screenshot indicates you have .swp file associated to your logback.xml file. When using vi for example to edit the logback.xml file, this swp file is created to hold you changes until you exit vi. I see that both your swap file and the original logback.xml are very different in sizes. Appears as though your NiFi may still be using the logback.xml which does not contian your changes that exist in the swp file only.

-

Thank you,

Matt