Member since
07-30-2019
2917
Posts
1446
Kudos Received
848
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
98 | 04-26-2024 06:40 AM | |
171 | 04-25-2024 06:16 AM | |
430 | 04-23-2024 05:56 AM | |
64 | 04-22-2024 06:13 AM | |
210 | 04-17-2024 11:30 AM |
08-18-2023
02:36 AM
1 Kudo
Hi, If you're using Apache NiFi and the token you're trying to capture with the InvokeHTTP processor is too large to be stored as an attribute, you can follow the steps below to work around this limitation: Keep the token in the content of the FlowFile if it's returned by the InvokeHTTP processor. You can use processors like ReplaceText to wrap the token in the header format you need. For instance, if you need the header to be Authorization: Bearer {token}, then you can configure a ReplaceText processor to replace the content (i.e., the token) to match this format.
... View more
08-17-2023
11:17 PM
I was able to resolve this error, by making the below configuration to ExecuteStreamCommand: Command Path: <complete path of python.exe> Command Arguments: <complete path of your script>;<other arg>;<other arg>
... View more
08-16-2023
06:57 AM
I set up the authorizers.xml file as you suggested and it's working perfectly, Thank you very much @MattWho !!
... View more
08-15-2023
06:23 AM
@Tenda Since you are saying you can freely navigate the NiFi UI when in this "stuck" state, NiFi is not stuck as both the UI and processor components all operate within the same JVM. What you circled indicates that at the exact moment (last time browser refreshed) there were 24 active threads out of the 32 configured in the Max Timer Driven Thread pool settings. Milliseconds later that could still be 24 active threads but consumed by different components. The NiFi processors will all show small a small number in the upper right corner if they have an active threads, so step one is determining which processors are holding these 24 threads for a long time. Then looking at those processors and the thread dumps to figure out why those threads are long running. Typically we would see this when external service connections are made which are unstable, network issues, local NiFi repo I/O, NiFi CPU utilization, or long or very frequent GC pauses, or even OOMs. So you have ruled out a few of these so far it sounds. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
08-11-2023
08:18 AM
1 Kudo
@Madhav_VD Apache NiFi contains no native processors that utilize Apache Tika other than IdentifyMimeType (this processor does not do any extraction), but you can find others in the Apache that have created custom processors that utilize Apache Tika. Adding custom nars to Apache NiFi is as easy as adding the custom nar to the auto-load directory: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#autoloading-processors While I have no experience with any of these custom nars, you can give them a try to see if they meet your needs. If not they may provide you with a stepping stone for creating your own custom variant. https://github.com/tspannhw/nifi-extracttext-processor/releases/tag/html https://community.cloudera.com/t5/Community-Articles/ExtractText-NiFi-Custom-Processor-Powered-by-Apache-Tika/ta-p/249392 https://community.cloudera.com/t5/Community-Articles/Creating-HTML-from-PDF-Excel-and-Word-Documents-using-Apache/ta-p/247968 https://github.com/tspannhw/nifi-extracttext-processor If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
08-11-2023
06:05 AM
Ok, thanks for the info. I thought it would have been an elegant solution to be able to use the nifi-api to transfer files directly to a port. I have tried ListenHTTP, which suits my task perfectly.
... View more
08-10-2023
01:30 PM
Hi @MattWho, Thank you for your prompt response. The solution you have provided actually worked. thanks for your help!
... View more
08-10-2023
06:52 AM
1 Kudo
@Anderosn In-between your SplitJson and PuSQL processors are you rebalancing the FlowFile across multiple nodes in a NiFi cluster? Are you routing any of the split Json messges down a different dataflow path that does not lead to this pusSQL processor? The reason I ask is because the splitJson processor will write the following FlowFile attributes to each new FlowFile created (each split): The fragment.identifier value and fragment.count are used by the putSQL processor when "Support FragmentTransactions" is set to "true" (default). This means that, if not all split jsons are present at this putSQL and located on the same node of the NiFi cluster, the FlowFiles part of the same fragment.identifier will not be processed and remain on the inbound connection to the PutSQL. I'd start my listing the connection and checking these attributes to verify the fragment.count is "10", the fragment.identifier has same value on all 10, and fragment.index value shows numbers 1 to 10 across those 10 FlowFiles. If making sure all fragments are processed in same transaction is not a requirement for your dataflow, try changing "Support Fragmented Transactions" to false and see if these 10 FlowFiles get successfully executed by your putSQL processor. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
08-09-2023
01:33 AM
Hey Matt! I am truly grateful for your answer. You made me realize what I was doing wrong and I learned a lot while reading trough your explanation. I am going to revise the documentation again, as I am realizing that I did not had some concepts crystal clear before. Thanks again and I wish you a good day.
... View more
08-08-2023
12:01 PM
Yep! Had moved on to that after your response and written up a JIRA for my team to build the needed tool. Thanks again for the response. My plan is the below > Find all the processors which may have state, then check if each has state, and if so, clear it. Find processors of used types which can contain state: e.g., QueryDatabaseTableRecord For each, pull the state by the ID and check if it has any stored state if totalEntryCount of cluster or local state is non-zero If so, call the clear state endpoint Query URL: /nifi-api/flow/search-results?q=QueryDatabaseTableRecord Pull Processor State: /nifi-api/processors/41584e33-adc6-171d-0000-0000581caccb/state Clear Processor State: POST /nifi-api/processors/41584e33-adc6-171d-0000-0000581caccb/state/clear-requests
... View more