@myuintelli2021  Logback is not something written by or for Apache NiFi.   NiFi's default logback has changed very little for nifi-app.log, nifi-user.log, and nifi-bootstrap.log.   All of which by default use the "RollingFileAppender". In the latest releases you may see that some additional appenders are now present; however, it is not uncommon for users to create additional appenders themselves in the logback.xml so that specific loggers can output to user defined log files rather than the defaults.  Logback supports numerous rolling policies: https://logback.qos.ch/manual/appenders.html These have existed along as NiFi has been using Logback. Not sure if maybe the issue is specific to the version of Windows or JDK you are using? If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@myuintelli2021  I just installed Apache NiFi 1.19.1 on my MAC and only modified the <maxFileSize> from default 100MB to 2 MB.  Log rotation is working as expected. My current logback configuration for nifi-app.log: <configuration scan="true" scanPeriod="30 seconds"> <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" /> <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator"> <resetJUL>true</resetJUL> </contextListener> <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>2MB</maxFileSize> <!-- keep 30 log files worth of history --> <maxHistory>30</maxHistory> </rollingPolicy> <immediateFlush>true</immediateFlush> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> Log directory showing logs rotated based on size (adds .<num> before .log suffix) reaching max: -rw-r--r-- 1 nifiadmin nifi 452B Jan 17 16:39 nifi-user.log -rw-r--r-- 1 nifiadmin nifi 2.0M Jan 17 16:53 nifi-app_2023-01-17_16.0.log -rw-r--r-- 1 nifiadmin nifi 2.0M Jan 17 16:58 nifi-app_2023-01-17_16.1.log drwxr-xr-x 13 nifiadmin nifi 416B Jan 17 16:58 . -rw-r--r-- 1 nifiadmin nifi 96K Jan 17 16:58 nifi-request.log -rw-r--r-- 1 nifiadmin nifi 178K Jan 17 16:58 nifi-app.log   I don't have access to a Windows environment to test there as it appears that is where you are testing, but it also works as expected in a linux environment I also have. I recommend opening a community question if you are looking for assistance with your setup. A community article comments is not he correct place to troubleshoot configurations or environmental issues.  I encourage you to include details when you raise that question to include at least your complete OS version, complete java (JDK) version being used, and complete NiFi version. Thank you, Matt ... View more

@srilakshmi  Logging does not happen at the process group level.  Processors logging is based on the processor class.  So there is nothing in the log output produced by a processor within a process group that is going to tell you in which process group that particular processor belongs.  That being said, you may be able to prefix every processor's name within the same Process group with some string that identifies the process group.  This processor name would generally be included in the the log output produced by the processor.  Then you may be able to use logback filters (have not tried this myself) to filter log output based on these unique strings. https://logback.qos.ch/manual/filters.html   NiFi bulletins (bulletins are log output to the NiFi UI and have a rolling 5 minute life in the UI) however do include details about the parent Process Group in which the component generating the bulletin resides.   You could build a dataflow in yoru NiFi to handle bulletin notification through the use of the SiteToSiteBulletinReportingTask which is used to send bulletin to a destination remote import port on a target NiFi.  A dataflow on the target NiFi could be built to parse the received bulletin records by the bulletinGroupName json path property so that all records from same PG are kept together.  These  'like' records could then be written out to local filesystem based on the PG name, remote system, used to send email notifications, etc... Example of what a Bulletin sent using the SiteToSiteBulletinReportingTask looks like: { "objectId" : "541dbd22-aa4b-4a1a-ad58-5d9a0b730e42", "platform" : "nifi", "bulletinId" : 2200, "bulletinCategory" : "Log Message", "bulletinGroupId" : "7e7ad459-0185-1000-ffff-ffff9e0b1503", "bulletinGroupName" : "PG2-Bulletin", "bulletinGroupPath" : "NiFi Flow / Matt's PG / PG2-Bulletin", "bulletinLevel" : "DEBUG", "bulletinMessage" : "UpdateAttribute[id=8c5b3806-9c3a-155b-ba15-260075ce9a6f] Updated attributes for StandardFlowFileRecord[uuid=1b0cb23a-75d8-4493-ba82-c6ea5c7d1ce3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1672661850924-5, container=default, section=5], offset=969194, length=1024],offset=0,name=bulletin-${nextInt()).txt,size=1024]; transferring to 'success'", "bulletinNodeId" : "e75bf99f-095c-4672-be53-bb5510b3eb5c", "bulletinSourceId" : "8c5b3806-9c3a-155b-ba15-260075ce9a6f", "bulletinSourceName" : "PG1-UpdateAttribute", "bulletinSourceType" : "PROCESSOR", "bulletinTimestamp" : "2023-01-04T20:38:27.776Z" } In the above produced bulletin json you see the BulletinGroupName and the BulletinMessage (the actual log output).   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@davehkd  No, do not set root node to a path.  The root node should be set to same value and "/nifi" is fine. This is the root node that all NiFi instance will use.  Lets say you were using the recommended external Zookeeper rather then the internal Zookeeper.  You may choose to use that one Zookeeper cluster to support multiple independent NiFi clusters.  To prevent ZK from thinking all nodes are part of same cluster, each NiFi cluster would use a different root node value.  So for second NiFi cluster you might use "/nifi2".  This value has nothing to do with an install path. Thanks, Matt ... View more

@davehkd You are definitely having issue with your embedded zookeeper. The shared exception indicates NIFi is not able to communicate with it.  The KeeperLoss exception thrown by the ZK client in NiFi most commonly is seen when the ZK does not have quorum because not enough nodes are part of ZK cluster or not all nodes are able to talk to one another. A New NiFi with no flows to load should start very quickly (never 12+ hours). Yours is unable to connect to the zookeeper that is included with and started as part of the NiFi process startup. 1. I'd start with making sure all of your nodes can resolve the zookeeper hostnames (nificlient1, nificlient2, and nificlient3) to proper reachable IP addresses.  If not make sure you add these to your local hosts file on each server so this is possible. 2. I'd check to see if zookeeper server is running on each host and listening on the configured ports. Make sure no other processes are using those ports thus blocking zookeeper from being able to bind to them. 3. Make sure you don't have any firewalls blocking connections to the zk ports.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@RodolfoE  Try using the absolute path to the certificate pem file rather than just the filename. If you are executing your curl command from within the directory where your pem file resides, try using "./<pem filename>". Otherwise curl may try looking up yoru pem filename as if it were an alias/nickname it expects find in some NSS database. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more