Created on 03-28-2024 08:29 AM - edited 03-28-2024 08:33 AM
I am using Apache nifi invokeHttp processor to call Google API. The url contains a path which i need to encode.So in the URL,only path is encoded e.g. resources%2F8cb90ae6-16cc-4460-99f4-3ae0689b3b66%2Fdata_asset%2FCETIvowels-38.pdf
However whenver I set encoded url, the processor internally replaces %2F with %252F which results in HTTP 404 response.
E.g. Nifi convert URL - https://storage.googleapis.com/storage/v1/b/t-bis-1hg-jk/o/resources%2F8cb90ae6-16cc-4460-99f4-3ae06...
to
Created 05-06-2024 10:40 AM
@jonay__reyes or @DeepakDonde
The issue you are encountering if caused by a code change so the the invokeHTTP would encode URLs automatically. This issue is triggered if your URL is already encoded. The URL encoding change will convert all '%' to '%25'.
Workarounds/solutions:
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created 03-28-2024 10:10 AM
@DeepakDonde
The issue you are describing was caused by a change in Apache NiFi InvokeHTTP processor that tries to URL encode the URL entered.
https://issues.apache.org/jira/browse/NIFI-12513
The fix for this is in https://issues.apache.org/jira/browse/NIFI-12785 which will be part of the Apache NiFi 1.26 and Apache NiFi 2.0.0-M3 releases.
Since the change that caused this issue was added to Apache NiFi 1.25 and Apache NiFi 2.0.0-M2, you could use and earlier version like Apache NiFi 1.24 or Apache NiFi 2.0.0-M1 to get around the issue until the two above mentioned versions are released.
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created 03-28-2024 10:27 PM
Changing Nifi version seems overhead as its running on kubernetes and there are many process groups running on that version. And i am not sure if doing that won't introduce the issues from earlier version.
Is there any work around for this?
Can I use GetHttp instead of InvokeHttp?
Created 03-29-2024 07:06 AM
@DeepakDonde
https://issues.apache.org/jira/browse/NIFI-12513 does not mention GetHTTP processor, so you could certainly try that processor to see if you experience the same issue.
Downgrade would lose all improvements and bug fixes introduced in Apache NiFi 1.25. Otherwise you could wait until 1.26 is released that contains the fix.
The InvokeHTTP processor is part of the NiFi Standard nar which includes a lot of NiFi components. You could also try Downloading just the 1.24.0 standard nar from the maven central repository and adding it the extensions folder of your 1.25.0 NiFi. This would make both the 1.24 and 1.25 versions of many components available in your NiFi. You could then use the 1.24 version of the invokeHTTP over the 1.25 version that has issue. This would allow you to continue to use 1.25 version for all other components. While i have added multiple version of the same nar to my NiFi installations in the past, I have not done so with the standard nar. If you have issues, you can stop your NiFi, remove the added nar and restart so thing go back to previous.
https://mvnrepository.com/artifact/org.apache.nifi/nifi-standard-shared-nar/1.24.0
https://repo1.maven.org/maven2/org/apache/nifi/nifi-standard-shared-nar/1.24.0/nifi-standard-shared-...
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created 05-06-2024 09:30 AM
Hi @MattWho! I tried to download the 1.24 InvokeHTTP to my M2 but doesn't seem to work:
While loading 'org.apache.nifi:nifi-standard-shared-nar:1.24.0' unable to locate exact NAR dependency 'org.apache.nifi:nifi-standard-services-api-nar:1.24.0'. Only found one possible match 'org.apache.nifi:nifi-standard-services-api-nar:2.0.0-M2'. Continuing...
then:
Loaded NAR file: /usr/local/lib/nifi-2.0.0-M2/./work/nar/extensions/nifi-standard-shared-nar-1.24.0.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-standard-shared-nar-1.24.0.nar-unpacked]
then:
Loaded extensions for org.apache.nifi:nifi-standard-shared-nar:1.24.0 in 23 millis
and finally:
Found existing bundle with coordinate org.apache.nifi:nifi-standard-shared-nar:1.24.0, will not load /usr/local/lib/nifi-2.0.0-M2/./extensions/nifi-standard-shared-nar-1.24.0.nar
Downloading also the mentioned nifi-standard-services-api-nar logs this:
org.apache.nifi.nar.NarAutoLoaderTask Found ./extensions/nifi-standard-services-api-nar-1.24.0.nar in auto-load directory
and:
Starting load process for 1 NARs...
Creating class loaders for 1 NARs...
While loading 'org.apache.nifi:nifi-standard-services-api-nar:1.24.0' unable to locate exact NAR dependency 'org.apache.nifi:nifi-jetty-bundle:1.24.0'. Only found one possible match 'org.apache.nifi:nifi-jetty-bundle:2.0.0-M2'. Continuing...
Loaded NAR file: /usr/local/lib/nifi-2.0.0-M2/./work/nar/extensions/nifi-standard-services-api-nar-1.24.0.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-standard-services-api-nar-1.24.0.nar-unpacked]
Successfully created class loaders for 1 NARs, 0 were skipped
Loaded extensions for org.apache.nifi:nifi-standard-services-api-nar:1.24.0 in 16 millis
Finished NAR loading process!
any ideas how this could actually be done?
Thanks in advance!
Created on 05-06-2024 10:14 AM - edited 05-06-2024 10:41 AM
@jonay__reyes
Apache NiFi 1.x and NiFi 2.x are major release versions, I would not expect that you could mix component versioned nars successfully between those versions.
My suggestion in the above post was you could try:
1. If using Apache NiFi 1.25, you could add the 1.24 nar.
2. If using Apache NiFi 2.0.0-M2, you could add the Apache NiFi 2.0.0.-M1 nar.
The NiFi standard nar contains so many core libraries and components, I can't guarantee
it will load successfully, but have done is successfully in the older releases of Apache NIFi 1.x versions. Never tried with Apache NiFi 2.x major milestone (M1, M2, ...) release versions.
With above being said, Nothing you shared above appears to be an ERROR. So I am no clear in exactly what you mean when you say "doesn't seem to work". Did not startup? Could the UI be accessed? When adding a new component do you see multiple versions for the components included in the standard nar?
When you load in to duplicate nars of different versions, NiFi does not change anything about the components loaded on the canvas. What you should see when dragging and adding new components to the canvas is multiple version options for the same component class to choose from. If you are still using the 2.0.0-M2 version of invokeHTTP, it is still going to have issues.
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created 05-06-2024 10:40 AM
@jonay__reyes or @DeepakDonde
The issue you are encountering if caused by a code change so the the invokeHTTP would encode URLs automatically. This issue is triggered if your URL is already encoded. The URL encoding change will convert all '%' to '%25'.
Workarounds/solutions:
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created on 05-07-2024 01:30 AM - edited 05-07-2024 01:41 AM
Thanks @MattWho for the quick response. The short version of "why does it doesn't work" means is that nifi keeps working, recognizes the addition of the NARs but no "1.24" versions of components are available in the Processor selection window. I promptly tried the "M1" download workaround and found this as soon as I wget'ed the NAR:
text version of the above:
INFO Found ./extensions/nifi-standard-shared-nar-2.0.0-M1.nar in auto-load directory
INFO Starting load process for 1 NARs...
INFO Creating class loaders for 1 NARs...
WARN Unable to resolve required dependency 'nifi-standard-services-api-nar'. Skipping NAR '/usr/local/lib/nifi-2.0.0-M2/./work/nar/extensions/nifi-standard-shared-nar-2.0.0-M1.nar-unpacked'
INFO Successfully created class loaders for 0 NARs, 1 were skipped
INFO Finished NAR loading process!
so, since it asks for nifi-standard-shared, I'll download it also and:
text version:
INFO Found ./extensions/nifi-standard-services-api-nar-2.0.0-M1.nar in auto-load directory
INFO Starting load process for 1 NARs...
INFO Including 1 previously skipped bundle(s)
INFO Creating class loaders for 2 NARs...
WARN While loading 'org.apache.nifi:nifi-standard-services-api-nar:2.0.0-M1' unable to locate exact NAR dependency 'org.apache.nifi:nifi-jetty-bundle:2.0.0-M1'. Only found one possible match 'org.apache.nifi:nifi-jetty-bundle:2.0.0-M2'. Continuing...
INFO Loaded NAR file: /usr/local/lib/nifi-2.0.0-M2/./work/nar/extensions/nifi-standard-services-api-nar-2.0.0-M1.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-standard-services-api-nar-2.0.0-M1.nar-unpacked]
INFO Loaded NAR file: /usr/local/lib/nifi-2.0.0-M2/./work/nar/extensions/nifi-standard-shared-nar-2.0.0-M1.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-standard-shared-nar-2.0.0-M1.nar-unpacked]
INFO Successfully created class loaders for 2 NARs, 0 were skipped
EDIT: I also downloaded the jetty package, but:
INFO Starting load process for 1 NARs...
ERROR Found a Jetty NAR, will not auto-load /usr/local/lib/nifi-2.0.0-M2/./extensions/nifi-jetty-bundle-2.0.0-M1.nar
INFO No NARs were unpacked, nothing to do
so, it looks like it's acknowledging the NARs I provide, but after reloading the interface:
You've been very helpful (and quick) providing the causes and several workarounds (I'll go the "build your URL on the fly" way), so thank you a lot and my double "accept as solution" click 🙂