Cloudera Community
Luwi
user rank
Contributor
My Accepted Solutions
Show All

ConsumeIMAP with OAuth2accesstokenprovider works w...

by Contributor in Support Questions
‎12-06-2023 10:21 PM
‎12-06-2023 10:21 PM
Hi, I have a ConsumeIMAP processor downloading emails from office 365 mailbox using StandardOAuth2AccessTokenProvider. Here is a simple structure, consisting of few processors, including custom one. When testing various scenarios I also removed my custom processor to make sure my problem is not caused by this. The thing is, that when testing and starting ConnsumeIMAP via "run once" option, it works perfectly and it gives me my 10 emails as set in Fetch Size. But when I start processor using Start option so it uses cron frequency 2 minutes, it returns only one email and the rest only when I stop the ConsumeIMAP processor. I tried to schedule to 5, 10, 15 minutes, but without effect. Also tried for accesstokenprovider processor various setting in refresh window property from 0 s to 60 s, but also without any effect. Also, after some time, and only sometimes, I get Authentization fail error, see below. When I stop the processor and use "run once", it starts to work immediately as I already mentioned above. Asked IT to check logs on exchange server, but there is nothing. I'm a bit desperate by now, as I googled, chatgpted a lot, so you guy are my last chance before I get mad. Thanks for all suggestions you might have. Here are parameters of ConsumeIMAP and StandardOAuth2AccessTokenProvider:   org.apache.nifi.processor.exception.ProcessException: Failed to receive messages from Email server: [javax.mail.AuthenticationFailedException - AUTHENTICATE failed. at org.apache.nifi.processors.email.AbstractEmailProcessor.fillMessageQueueIfNecessary(AbstractEmailProcessor.java:386) at org.apache.nifi.processors.email.AbstractEmailProcessor.receiveMessage(AbstractEmailProcessor.java:439) at org.apache.nifi.processors.email.AbstractEmailProcessor.onTrigger(AbstractEmailProcessor.java:242) at org.apache.nifi.processors.email.ConsumeIMAP.onTrigger(ConsumeIMAP.java:32) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1361) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:247) at org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:130) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: javax.mail.AuthenticationFailedException: AUTHENTICATE failed. at com.sun.mail.imap.IMAPStore.protocolConnect(IMAPStore.java:732) at javax.mail.Service.connect(Service.java:366) at javax.mail.Service.connect(Service.java:246) at javax.mail.Service.connect(Service.java:195) at org.springframework.integration.mail.AbstractMailReceiver.connectStoreIfNecessary(AbstractMailReceiver.java:334) at org.springframework.integration.mail.AbstractMailReceiver.openFolder(AbstractMailReceiver.java:345) at org.springframework.integration.mail.AbstractMailReceiver.receive(AbstractMailReceiver.java:373) at org.apache.nifi.processors.email.AbstractEmailProcessor.fillMessageQueueIfNecessary(AbstractEmailProcessor.java:381)   ... View more
Labels:

Re: PutEmail frozen

by Contributor in Support Questions
‎07-26-2023 01:02 PM
‎07-26-2023 01:02 PM
Hi Matt,   thanks for your reply and suggestions, appreciate it. Server where NiFi is running is restarted every night. I had to terminate it to release processor, so it can work again. I set the processor on Debug as suggested by @cotopaul and will see tomorrow more details I believe. Yes, there is probably something with exchange communication, but it should not be some kind of quota on max emails, as when I terminate it and run again, it works for several hour again. Also, it rarely process more then 10 emails at once and we check this one. Thanks for info about dynamic smtp mail properties, I will go through this. Will inform tomorrow on my progress. 🙂   Thanks a lot! Ludvik ... View more

Re: PutEmail frozen

by Contributor in Support Questions
‎07-26-2023 08:48 AM
‎07-26-2023 08:48 AM
So, there is some development on this. After several hours of working without issues on changed port 25 it got stuck again. So I changed STARTTLS to false and got error below. Then changed it back to "true" and got another error related to the port, se below as well.   Emails has always subject like this: (NiFi) FP05-5213923, 88918, User Name, 101 City Body is always one line: Nové číslo faktury: 5213923     ... View more

Re: PutEmail frozen

by Contributor in Support Questions
‎07-26-2023 04:19 AM
‎07-26-2023 04:19 AM
Hi, thanks a lot for your reply. This setup worked for month with no issues until last Friday, since then it got stuck twice. I changed port to 25 yesterday and no problems so far, so hopefully this will solve it. Thanks for pointing me to SMTP Starttls, in case of troubles I will definitely try to set it to false. I will monitor it closely and if no issued within next month, I will accept you comment as solution, because your suggested changing port too. Thanks again, I will drop a comment later so this topic is updated. ... View more

PutEmail frozen

by Contributor in Support Questions
‎07-24-2023 11:38 PM
‎07-24-2023 11:38 PM
Hello,   I have a simple process getting data from sql server and send them by email. Short emails, nothing heavy and not that many during the day. But few days ago, it suddenly started to be frozen and it happened again today. It does not show any error and I do not see any error in logs either.   I checked this thread, maybe it's the same, but there is no real solution and in my logs I can't see anything with SMTP or that it's "waiting", so would like to ask you for help or to point me to the right direction. Maybe I'm not checking everything...     Thanks Rgds Ludvik   ... View more
Labels:

Re: Unable to write flowfile content to content re...

by Contributor in Support Questions
‎07-08-2022 12:16 AM
1 Kudo
‎07-08-2022 12:16 AM
1 Kudo
@MattWho ,   I'm processing invoice data from accounting system where I ask for invoices of two years (one by one) and then split them to invoice level and process each invoice separately. So it's a lot of small flowfiles and the whole process takes around 25 minutes. I think this is what you are referring to as active claims, right?  Content_repository resides on the server disk which is used for other things. That is my current client's setup. if I disable archiving, then this issue will be solved, but I will not have a possibility to check the content in provenance history, correct? Thanks, Ludvik ... View more

Re: Unable to write flowfile content to content re...

by Contributor in Support Questions
‎07-08-2022 12:06 AM
‎07-08-2022 12:06 AM
@MattWho , thanks. I used previous release, but I did not upgraded, I downloaded it and run on a new server and moved only my processes from old to then new NiFi.   Ludvik ... View more

Re: Unable to write flowfile content to content re...

by Contributor in Support Questions
‎07-06-2022 09:19 PM
‎07-06-2022 09:19 PM
@araujo , thanks. @MattWho is mentioning "nifi.content.repository.archive.backpressure.percentage", but I can't see it in my nifi.properties file. (I use NiFi 1.16.3). May I add it manually?   Thanks Ludvik ... View more

Unable to write flowfile content to content reposi...

by Contributor in Support Questions
‎07-06-2022 12:09 PM
‎07-06-2022 12:09 PM
Hi,   I'm getting this message when running process with more than 10000 flowfiles. unable to write flowfile content to content repository container default due to archive file size constraints   I googled and read more about content repository and content claims, but still don't get it. My archive folder is like 100MB.   Can you please help me understand how to configure NiFi to get rid of this message?   Thanks a lot.   My settings which could affect this is: nifi.flow.configuration.archive.max.time=30 days nifi.flow.configuration.archive.max.storage=500 MB nifi.flow.configuration.archive.max.count=   nifi.queue.backpressure.count=10000 nifi.queue.backpressure.size=1 GB   nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true   ... View more
Labels:

Re: ExecuteScript error - ECMAScript is missing

by Contributor in Support Questions
‎06-26-2022 09:20 AM
‎06-26-2022 09:20 AM
ok, I found this https://en.wikipedia.org/wiki/Nashorn_(JavaScript_engine) With the release of Java 11, Nashorn was deprecated citing challenges to maintenance, and has been removed from JDK 15 onwards. ... View more
Contact Me
Online
Offline
Last Visited
‎12-28-2023 03:16 PM
Community Statistics
Member Since ‎04-15-2020 11:57 AM
Last Visited ‎12-28-2023 03:16 PM
Posts 14
Kudos received 2