Silver ⭐️⭐️

Solved

Invoke Http with url containing %2F

7 replies
6800 views
Translate All

I am using Apache nifi invokeHttp processor to call Google API. The url contains a path which i need to encode.So in the URL,only path is encoded e.g. resources%2F8cb90ae6-16cc-4460-99f4-3ae0689b3b66%2Fdata_asset%2FCETIvowels-38.pdf

However whenver I set encoded url, the processor internally replaces %2F with %252F which results in HTTP 404 response.

E.g. Nifi convert URL - https://storage.googleapis.com/storage/v1/b/t-bis-1hg-jk/o/resources%2F8cb90ae6-16cc-4460-99f4-3ae0689b3b66%2Fdata_asset%2FCETIvowels-38.pdf?alt=media

to

https://storage.googleapis.com/storage/v1/b/t-bis-1hg-jk/o/resources%252F8cb90ae6-16cc-4460-99f4-3ae0689b3b66%252Fdata_asset%252FCETIvowels-38.pdf?alt=media

Apache NiFi

MattWhoAnswer

Titanium ⭐️⭐️⭐️

@jonay__reyes or @DeepakDonde

The issue you are encountering if caused by a code change so the the invokeHTTP would encode URLs automatically. This issue is triggered if your URL is already encoded. The URL encoding change will convert all '%' to '%25'.

Workarounds/solutions:

Remove your url encoding and allow processor to do that encoding.
If your URL is not URL encoded already and happens to contain '%' in the URL you can do the following:
- If using Apache NiFi 1.25 verions:
  - Upgrade to NIFi 1.26 which contains fix NIFI-12842. (now released)
  - Try adding Apache NiFi 1.26 version of the NiFi standard nar to your 1.25 install.
  - Downgrade to Apache NiFi 1.24
- If using Apache NiFi 2.0.0-M2
  - Wait for upcoming release of NiFi 2.0.0-M3 and upgrade, which will contain fix.
  - Downgrade to Apache NiFi 2.0.0-M1
  - Try adding Apache NiFi 2.0.0-M1 Standard nar to your 2.0.0-M2 install and switch to using older 2.0.0-M1 invokeHTTP processor.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

MattWho

Titanium ⭐️⭐️⭐️

@DeepakDonde

The issue you are describing was caused by a change in Apache NiFi InvokeHTTP processor that tries to URL encode the URL entered.
https://issues.apache.org/jira/browse/NIFI-12513

The fix for this is in https://issues.apache.org/jira/browse/NIFI-12785 which will be part of the Apache NiFi 1.26 and Apache NiFi 2.0.0-M3 releases.

Since the change that caused this issue was added to Apache NiFi 1.25 and Apache NiFi 2.0.0-M2, you could use and earlier version like Apache NiFi 1.24 or Apache NiFi 2.0.0-M1 to get around the issue until the two above mentioned versions are released.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

DeepakDondeAuthor

Silver ⭐️⭐️

Changing Nifi version seems overhead as its running on kubernetes and there are many process groups running on that version. And i am not sure if doing that won't introduce the issues from earlier version.

Is there any work around for this?

Can I use GetHttp instead of InvokeHttp?

Translate

MattWho

Titanium ⭐️⭐️⭐️

@DeepakDonde

https://issues.apache.org/jira/browse/NIFI-12513 does not mention GetHTTP processor, so you could certainly try that processor to see if you experience the same issue.

Downgrade would lose all improvements and bug fixes introduced in Apache NiFi 1.25. Otherwise you could wait until 1.26 is released that contains the fix.

The InvokeHTTP processor is part of the NiFi Standard nar which includes a lot of NiFi components. You could also try Downloading just the 1.24.0 standard nar from the maven central repository and adding it the extensions folder of your 1.25.0 NiFi. This would make both the 1.24 and 1.25 versions of many components available in your NiFi. You could then use the 1.24 version of the invokeHTTP over the 1.25 version that has issue. This would allow you to continue to use 1.25 version for all other components. While i have added multiple version of the same nar to my NiFi installations in the past, I have not done so with the standard nar. If you have issues, you can stop your NiFi, remove the added nar and restart so thing go back to previous.

https://mvnrepository.com/artifact/org.apache.nifi/nifi-standard-shared-nar/1.24.0
https://repo1.maven.org/maven2/org/apache/nifi/nifi-standard-shared-nar/1.24.0/nifi-standard-shared-nar-1.24.0.nar

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

jonay__reyes

Silver ⭐️⭐️⭐️

Hi @MattWho! I tried to download the 1.24 InvokeHTTP to my M2 but doesn't seem to work:

 While loading 'org.apache.nifi:nifi-standard-shared-nar:1.24.0' unable to locate exact NAR dependency 'org.apache.nifi:nifi-standard-services-api-nar:1.24.0'. Only found one possible match 'org.apache.nifi:nifi-standard-services-api-nar:2.0.0-M2'. Continuing...

then:

Loaded NAR file: /usr/local/lib/nifi-2.0.0-M2/./work/nar/extensions/nifi-standard-shared-nar-1.24.0.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-standard-shared-nar-1.24.0.nar-unpacked]

then:

Loaded extensions for org.apache.nifi:nifi-standard-shared-nar:1.24.0 in 23 millis

and finally:

Found existing bundle with coordinate org.apache.nifi:nifi-standard-shared-nar:1.24.0, will not load /usr/local/lib/nifi-2.0.0-M2/./extensions/nifi-standard-shared-nar-1.24.0.nar

Downloading also the mentioned nifi-standard-services-api-nar logs this:

org.apache.nifi.nar.NarAutoLoaderTask Found ./extensions/nifi-standard-services-api-nar-1.24.0.nar in auto-load directory

and:

Starting load process for 1 NARs...
Creating class loaders for 1 NARs...
While loading 'org.apache.nifi:nifi-standard-services-api-nar:1.24.0' unable to locate exact NAR dependency 'org.apache.nifi:nifi-jetty-bundle:1.24.0'. Only found one possible match 'org.apache.nifi:nifi-jetty-bundle:2.0.0-M2'. Continuing...
Loaded NAR file: /usr/local/lib/nifi-2.0.0-M2/./work/nar/extensions/nifi-standard-services-api-nar-1.24.0.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-standard-services-api-nar-1.24.0.nar-unpacked]
Successfully created class loaders for 1 NARs, 0 were skipped
Loaded extensions for org.apache.nifi:nifi-standard-services-api-nar:1.24.0 in 16 millis
Finished NAR loading process!

any ideas how this could actually be done?

Thanks in advance!

Translate

MattWho

Titanium ⭐️⭐️⭐️

@jonay__reyes

Apache NiFi 1.x and NiFi 2.x are major release versions, I would not expect that you could mix component versioned nars successfully between those versions.

My suggestion in the above post was you could try:
1. If using Apache NiFi 1.25, you could add the 1.24 nar.
2. If using Apache NiFi 2.0.0-M2, you could add the Apache NiFi 2.0.0.-M1 nar.

The NiFi standard nar contains so many core libraries and components, I can't guarantee
it will load successfully, but have done is successfully in the older releases of Apache NIFi 1.x versions. Never tried with Apache NiFi 2.x major milestone (M1, M2, ...) release versions.

With above being said, Nothing you shared above appears to be an ERROR. So I am no clear in exactly what you mean when you say "doesn't seem to work". Did not startup? Could the UI be accessed? When adding a new component do you see multiple versions for the components included in the standard nar?

When you load in to duplicate nars of different versions, NiFi does not change anything about the components loaded on the canvas. What you should see when dragging and adding new components to the canvas is multiple version options for the same component class to choose from. If you are still using the 2.0.0-M2 version of invokeHTTP, it is still going to have issues.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

MattWhoAnswer

Titanium ⭐️⭐️⭐️

@jonay__reyes or @DeepakDonde

The issue you are encountering if caused by a code change so the the invokeHTTP would encode URLs automatically. This issue is triggered if your URL is already encoded. The URL encoding change will convert all '%' to '%25'.

Workarounds/solutions:

Remove your url encoding and allow processor to do that encoding.
If your URL is not URL encoded already and happens to contain '%' in the URL you can do the following:
- If using Apache NiFi 1.25 verions:
  - Upgrade to NIFi 1.26 which contains fix NIFI-12842. (now released)
  - Try adding Apache NiFi 1.26 version of the NiFi standard nar to your 1.25 install.
  - Downgrade to Apache NiFi 1.24
- If using Apache NiFi 2.0.0-M2
  - Wait for upcoming release of NiFi 2.0.0-M3 and upgrade, which will contain fix.
  - Downgrade to Apache NiFi 2.0.0-M1
  - Try adding Apache NiFi 2.0.0-M1 Standard nar to your 2.0.0-M2 install and switch to using older 2.0.0-M1 invokeHTTP processor.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

jonay__reyes

Silver ⭐️⭐️⭐️

Thanks @MattWho for the quick response. The short version of "why does it doesn't work" means is that nifi keeps working, recognizes the addition of the NARs but no "1.24" versions of components are available in the Processor selection window. I promptly tried the "M1" download workaround and found this as soon as I wget'ed the NAR:

text version of the above:

INFO Found ./extensions/nifi-standard-shared-nar-2.0.0-M1.nar in auto-load directory
INFO Starting load process for 1 NARs...
INFO Creating class loaders for 1 NARs...
WARN Unable to resolve required dependency 'nifi-standard-services-api-nar'. Skipping NAR '/usr/local/lib/nifi-2.0.0-M2/./work/nar/extensions/nifi-standard-shared-nar-2.0.0-M1.nar-unpacked'
INFO Successfully created class loaders for 0 NARs, 1 were skipped
INFO Finished NAR loading process!

so, since it asks for nifi-standard-shared, I'll download it also and:

text version:

INFO Found ./extensions/nifi-standard-services-api-nar-2.0.0-M1.nar in auto-load directory
INFO Starting load process for 1 NARs...
INFO Including 1 previously skipped bundle(s)
INFO Creating class loaders for 2 NARs...
WARN While loading 'org.apache.nifi:nifi-standard-services-api-nar:2.0.0-M1' unable to locate exact NAR dependency 'org.apache.nifi:nifi-jetty-bundle:2.0.0-M1'. Only found one possible match 'org.apache.nifi:nifi-jetty-bundle:2.0.0-M2'. Continuing...
INFO Loaded NAR file: /usr/local/lib/nifi-2.0.0-M2/./work/nar/extensions/nifi-standard-services-api-nar-2.0.0-M1.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-standard-services-api-nar-2.0.0-M1.nar-unpacked]
INFO Loaded NAR file: /usr/local/lib/nifi-2.0.0-M2/./work/nar/extensions/nifi-standard-shared-nar-2.0.0-M1.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-standard-shared-nar-2.0.0-M1.nar-unpacked]
INFO Successfully created class loaders for 2 NARs, 0 were skipped

EDIT: I also downloaded the jetty package, but:

INFO Starting load process for 1 NARs...
ERROR Found a Jetty NAR, will not auto-load /usr/local/lib/nifi-2.0.0-M2/./extensions/nifi-jetty-bundle-2.0.0-M1.nar
INFO No NARs were unpacked, nothing to do

so, it looks like it's acknowledging the NARs I provide, but after reloading the interface:

You've been very helpful (and quick) providing the causes and several workarounds (I'll go the "build your URL on the fly" way), so thank you a lot and my double "accept as solution" click 🙂

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.