Error in Base64EncodeContent processor after configuring repository encryption- protocol version 1

[#1]

I have enabled the repository Encryption with protocol version 1, after that  there is below error in Base64EncodeContent  for some of the flowfiles  and other flowfiles gets successfully processed.Can someone please help on this.

2023-08-08 19:26:07,368 ERROR [Timer-Driven Process Thread-5] o.a.n.p.standard.Base64EncodeContent Base64EncodeContent[id=7ba6e988-8bc9-392a-1ae8-f38733a69144] Processing halted: yielding [1 sec]
org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find content for FlowFile
at org.apache.nifi.controller.repository.StandardProcessSession.handleContentNotFound(StandardProcessSession.java:3701)
at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3454)
at org.apache.nifi.processors.standard.Base64EncodeContent.onTrigger(Base64EncodeContent.java:135)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1360)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.nifi.controller.repository.ContentNotFoundException: Could not find content for StandardContentClaim [resourceClaim=StandardResourceClaim[id=1691502935403-13, container=default, section=13], offset=47426, length=13851]: Stream contained only 525 bytes but should have contained 904
at org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.ensureAllContentRead(FlowFileAccessInputStream.java:49)
at org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:84)
at org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.apache.nifi.processors.standard.util.ValidatingBase64InputStream.read(ValidatingBase64InputStream.java:37)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.nifi.processors.standard.util.ValidatingBase64InputStream.read(ValidatingBase64InputStream.java:53)
at org.apache.commons.codec.binary.BaseNCodecInputStream.read(BaseNCodecInputStream.java:173)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.nifi.processors.standard.Base64EncodeContent$2.process(Base64EncodeContent.java:141)
at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3436)
... 12 common frames omitted

[#2]

@tej_s 

I recommend against switching to encrypted repositories while you still have content in your flowfile_repository, content_repository, and provenance_repository.   What version of Apache NiFi are you using?

If you stop NiFi and delete the contents of these encrypted repositories and startup clean, do you encounter and issues like above with all new data being processed through your dataflows?

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#3]

@MattWho,

Thanks for your response!

The NiFi version that I am using is NiFi-1.17.0. However I tested the same with NiFi-1.15.0, NiFi-1.18.0 and NiFi-1.22.0. Still the same error occurs.

I configured the repository data encryption immediately after setting up the nifi when there was no data flowing through the system. Next time I tried with cleaning up the existing data in NiFi repositories and sending the new data still same issue persists.

Is there any other scenario that you think I should try and that might work. If yes please suggest that.

I am not sure, if this is any functional issue where Base64 decoding doesn't work with the AES GCM 256 encryption algorithm for repository encryption.

[#4 · Accepted solution]

@tej_s @jakese 
There are some recently discovered issues seen as more individuals start using the encrypted repository capability that have been addressed in Apache NiFi 1.23.  I encourage you to see if you can reproduce your issue with release 1.23 or newer.

Some specific related fix that are part of 1.23:
https://issues.apache.org/jira/browse/NIFI-10235

https://issues.apache.org/jira/browse/NIFI-11670
11670 fixed some offset tracking issues and content handling issues that could very likely be causing your issue.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#5]

Upgrading NiFi to latest version NiFi-1.23.2 helped me on this issue and there are no issues with Base64EncodeContent processor after configuring repository encryption

@MattWho Thanks you!

[#6]

this also happen to me