Member since
06-14-2023
96
Posts
34
Kudos Received
9
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1739 | 11-04-2024 06:51 PM | |
| 6067 | 12-29-2023 09:36 AM | |
| 9415 | 12-28-2023 01:01 PM | |
| 1980 | 12-27-2023 12:14 PM | |
| 1194 | 12-08-2023 12:47 PM |
12-01-2023
08:57 AM
@Jisson I don't see ExecuteStream in the thread dump provided. Let's clarify first what you mean by "stuck"... When the processor is in this "stuck" state, does the processor indicate that it has an active thread? A NiFi processor will show a small number in its upper right corner when it has an active thread(s). Below example shows an ExecuteStreamCommand processor with "1" active thread: If your processor has no active threads, it is not stuck/hung. It is simply does not have a thread to execute the command. This could happen if all thread from the max timer driven thread pool in NiFi are already being used by other components. We would call this a thread starved processor. If your CPU load average is good, you could increase the size of the thread pool to see if that helps. NiFi out-of-the-box sets the "Maximum Timer Driven Thread Count" Pool to 10. You can change this from the NiFi Ui --> global menu (upper right corner) --> Controller Settings --> General tab. If your processor does show an active threads, i'd expect to see that thread in the thread dump. Also keep in mind that a single thread dump is not very useful. A thread may not be HUNG, but rather long running for example. So getting a series of thread dumps spread out to compare would allow you to see if the thread stack is changing over time indicating not hung but slow. In the case of your ExecuteStreamCommand processor, it is calling a custom python script and the waits for the return from that script. Then comes the challenge is the thread dump indicates it is waiting on your python script return to figure out why your python scripts is hanging or taking a very long time all of a sudden. Not something that can be troubleshot through NiFi. Hope this helps you in your troubleshooting journey. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-01-2023
08:21 AM
@SAMSAL The managed Authorizer uses the file-access-policy-provider (generates the authorizations.xml if it does no already exist) and then a user-group-provider. In your case that would make most sense to be the ldap-user-group-provider. You may also want to use the Composite-configurable-user-group-provider (configure it with ldap-user-group-provider and file-user-group-provider). Having both a file based provider and ldap provider allows sycning of ldap users and groups form ldap automatically as well as the file provider allowing you to manually add non ldap user/client identities for authorization as well. Non ldap client/user identities might be certifcate based clients like other NiFi nodes/instance, etc.. Within the file-access-policy-provider you define the initial admin identity. That user identity could be set to your ldap user account identity. Then on first start up with managed provider, it generates the authorizations.xml file seeded with the policies necessary for that initial admin user identity to act as admin. So you could skip the single-user-provider step. Matt
... View more
11-30-2023
12:26 AM
yea I tried using KeyScanOptions as well but the same error
... View more
11-29-2023
04:39 AM
CHeck out this link @joseomjr shared in another post: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1 And here is a great blog from Pierre Villard: https://medium.com/cloudera-inc/getting-ready-for-apache-nifi-2-0-5a5e6a67f450 Nifi 2.0 brings python native, where jython was very limited library set.
... View more
11-29-2023
01:01 AM
Hi! Thanks for your response. Now the problem is solved! I was not able to connect due to that nifi was effectively installed in a docker in my computer, so, even the computer had access to the mongo database, the docker not. Now I solved the problem and can continue working on it! Thanks again!
... View more
11-28-2023
05:47 AM
@joseomjr Is on to the right solution here. Your regex statement should match "kafka.topic" not "${kafka.topic}". A quick test in regex101.com confirms "kafka\.topic" should match.
... View more
11-27-2023
08:10 AM
Your original example had everything at +0000 but the error has +0700 so perhaps change the pattern to yyyy-MM-dd'T'HH:mm:ssZ
... View more
11-22-2023
09:52 PM
^ I've attached the image above. this is how the data looks. I want to clean the first 7 rows and let the 8th row (header row) be first.
... View more
11-22-2023
02:46 PM
Yes, turns out the groovy-yaml module is an optional module in Groovy 3, however groovy-yaml-3.0.17.jar is actually included in nifi-scripting-nar-1.23.2 but the ExecuteScript processor when started throws the exception I mentioned. Possibly this indicates there are other dependencies that would need to be explicitly included in the classpath (e.g. jackson-databind, jackson-dataformat-yaml, groovy-json) I was hoping to use YamlSlurper followed by JsonBuilder to do YAML to JSON conversions. In the end I used the Jackson YamlFactory and ObjectMapper to achieve the desired result.
... View more
11-21-2023
01:56 PM
What's wrong with 1:1 relationship? If you're concerned with performance, then leveraging "Message Demarcator" for the Consume and Publish will provide the best throughput.
... View more