Member since
07-30-2019
3429
Posts
1632
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 100 | 01-27-2026 12:46 PM | |
| 506 | 01-13-2026 11:14 AM | |
| 1107 | 01-09-2026 06:58 AM | |
| 944 | 12-17-2025 05:55 AM | |
| 448 | 12-17-2025 05:34 AM |
01-17-2023
01:23 PM
@srilakshmi Logging does not happen at the process group level. Processors logging is based on the processor class. So there is nothing in the log output produced by a processor within a process group that is going to tell you in which process group that particular processor belongs. That being said, you may be able to prefix every processor's name within the same Process group with some string that identifies the process group. This processor name would generally be included in the the log output produced by the processor. Then you may be able to use logback filters (have not tried this myself) to filter log output based on these unique strings. https://logback.qos.ch/manual/filters.html NiFi bulletins (bulletins are log output to the NiFi UI and have a rolling 5 minute life in the UI) however do include details about the parent Process Group in which the component generating the bulletin resides. You could build a dataflow in yoru NiFi to handle bulletin notification through the use of the SiteToSiteBulletinReportingTask which is used to send bulletin to a destination remote import port on a target NiFi. A dataflow on the target NiFi could be built to parse the received bulletin records by the bulletinGroupName json path property so that all records from same PG are kept together. These 'like' records could then be written out to local filesystem based on the PG name, remote system, used to send email notifications, etc... Example of what a Bulletin sent using the SiteToSiteBulletinReportingTask looks like: {
"objectId" : "541dbd22-aa4b-4a1a-ad58-5d9a0b730e42",
"platform" : "nifi",
"bulletinId" : 2200,
"bulletinCategory" : "Log Message",
"bulletinGroupId" : "7e7ad459-0185-1000-ffff-ffff9e0b1503",
"bulletinGroupName" : "PG2-Bulletin",
"bulletinGroupPath" : "NiFi Flow / Matt's PG / PG2-Bulletin",
"bulletinLevel" : "DEBUG",
"bulletinMessage" : "UpdateAttribute[id=8c5b3806-9c3a-155b-ba15-260075ce9a6f] Updated attributes for StandardFlowFileRecord[uuid=1b0cb23a-75d8-4493-ba82-c6ea5c7d1ce3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1672661850924-5, container=default, section=5], offset=969194, length=1024],offset=0,name=bulletin-${nextInt()).txt,size=1024]; transferring to 'success'",
"bulletinNodeId" : "e75bf99f-095c-4672-be53-bb5510b3eb5c",
"bulletinSourceId" : "8c5b3806-9c3a-155b-ba15-260075ce9a6f",
"bulletinSourceName" : "PG1-UpdateAttribute",
"bulletinSourceType" : "PROCESSOR",
"bulletinTimestamp" : "2023-01-04T20:38:27.776Z"
} In the above produced bulletin json you see the BulletinGroupName and the BulletinMessage (the actual log output). If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
01-12-2023
01:30 PM
@SachinMehndirat There is NO replication of data from the four NiFi repositories across all NiFi nodes in a NiFi cluster. Each NiFi node in the cluster is only aware of and only excutes against the FlowFile on that specific node. As such, NiFi nodes can not share a common set of repositories. Each node must have their own repositories and it is important to protect those repositories from data loss (flowfile_repository and content_repository being most important). - flowfile_repository - contain metadata/attributes about FlowFiles actively processing thorugh your NiFi dataflow(s). This includes metadata on location of content of queued FlowFiles. - content_repository - contains content claims that can hold the content for 1 too many FlowFiles actively being processed or temporarily archived post termination at end of dataflow(s) - provenance_repository - contains historical lineage information about FlowFile currently or previously processed through your NiFi dataflows. - database_repository - contains flow configuration history which is a record of changes made via NiFi UI (adding, modifying, deleting, stopping, starting, etc...). Also contain info about users currently authenticated in to the NiFi node. Processors that record cluster wide state would use zookeeper to store and retrieve that stored state needed by all nodes. Processors that use local state will write that state to NiFi locally configured state directory. So in addition to protect the repositories mentioned above from dataloss, you'll also want to make sure local state (unique to each node in the NiFi cluster) directory is also protected. The embedded documentation in NiFi for each component has a section "State management:" that will tell you if that component use local and/or cluster state. You may find some of the info found in the following articles useful: https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999 https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418 https://blogs.apache.org/nifi/entry/load-balancing-across-the-cluster If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
01-12-2023
01:08 PM
1 Kudo
@AndreyN Each processor by default uses Timer Driven Scheduling strategy and a run Schedule of 0 secs. This means that each processor is constantly requesting threads from the Max Timer Driven Thread pool and checking for work (work being any FlowFiles on inbound connections or in case of an ingest processor, connecting to that ingest point whether local dir or remote service to check for data). While generally these checks for work take micro seconds or longer depending on processor, NiFi does have a global setting for yielding the processors when the previous run resulted in no FlowFiles processed/produced. To prevent excessive latency this back duration by default is very short (10 ms). To adjust this setting, you can change the following property in the nifi.properties file: nifi.bored.yield.duration found in Core Properties section of admin guide. Keep in mind that this setting impacts all processors. So higher you set the more latency that could exist before new work is found after a run that resulted in no work. You can also selectively adjust the run schedule on select processors. 0 sec run schedule means to run as fast as possible. So as soon as one thread completes, request another (unless thread results in no work wait bored duration before requesting next thread). So if you have flows that are always very light and latency is not a concern, you could set those processors to only get scheduled to execute every 1 sec or longer. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
01-10-2023
09:19 PM
@RodolfoE, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
01-05-2023
11:51 AM
@SachinMehndirat The exception comes from java and not NiFi's core code base. The exception is possibly being thrown as a result of a failed mutual TLS handshake. Without the complete stack trace and preceding log output it is difficult to give a detailed response here, so below is based on assumptions from what little has been shared. However, when you see "chain" issues (keeping in mind that output shared is incomplete), that points at missing TrustedCertEntry(s) in the truststore used one side or the other of that TLS handshake. Both the client and server sides of a TLS handshake present their clientAuth or serverAuth certificate info. These PrivateKeyEntry certificates will have an owner and issuer (signing Certificate Authority (CA) for the owner). In order for the receiving sides to trust the certificate presented from the other side, it must trust that CA. This means that within its truststore, it must have a TrustedCertEntry for the CA that signed the PrivateKey presented in the TLS handshake. BUT... it does not end there. The CA may be an intermediate CA meaning that the certificate for that CA was signed by yet another CA. In that case the truststore would also need the TrustedCertEntry for the next CA in that chain of authority. A complete trust "chain" would consist of all CAs from signer of PrivateKey to the root CA (root CA public key will have same Distinguished Name (DN) for both owner and signer of the TrustedCertEntry). So on one side or the other you do not have the complete trustchain in the truststore. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
01-03-2023
08:55 AM
@Mostafa12345678 Can you share full stack trace for review?
... View more
01-03-2023
07:46 AM
@doubtguy Is there a consistency in the timing of when it stops tailing the file (after 15 minutes of starting processor for example)? Interesting that it starts tailing again when you cd to the mounted directory. This seems characteristic of a SMB/CIFS mount that has gone idle. If you set up a process to touch that file or cd to that mounted directory every 2 mins or so, does the issue go away? If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
01-02-2023
11:21 PM
@GosuNadir, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.
... View more
12-30-2022
02:16 AM
Hi @MattWho , After i Enabled Retrieve site-to-site details for the Input port BulleInfo, then i was able to receive the BulletinInfo. Thank you so much for you suggestions!! Regards, Pradeep
... View more