Member since
03-05-2021
19
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 677 | 07-22-2022 08:59 AM |
01-10-2023
06:10 PM
1 Kudo
Hi @edaley, I have a couple instances runnning on Windows and all of them rotates uses the TimeBasedRollingPolicy with the following configuration <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!-- file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file -->
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log.zip</fileNamePattern>
<maxHistory>30</maxHistory>
</rollingPolicy>
<immediateFlush>true</immediateFlush>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder> This configuration works over the nifi-app.log file and move it into a new file nifi-app_date.log.zip at midnight. Hope this helps.
... View more
10-06-2022
03:00 PM
Hi, This week I started facing an issue with a PutFile processor that was running for over a year without problems. This processor is getting blocked and start enqueuing flowfiles. When I tried to terminate the process inside Nifi it ends; but no matter how many times i try to restart it, the processor never works again. My next step is to shutdown Nifi. The shutdown process ends OK, but the Windows process never ends and kept temporary files locked and RAM memory allocate. But the worse part is that this process is impossible to kill. I tried to kill it from the task manager and got access error. Also tried to kill it from cmd (as administrator) using taskkill and wimc commands but the process does not died. Until now, the only solution that I have found is to restart the server. But even this solution works half way, because if I send the reboot from the server; this one stays (I guess) trying to kill the process and doesn't shut down. The only thing that has worked is to send the reboot from the administration console of my hosting provider. The following is the nifi-dump log: "Timer-Driven Process Thread-1 <Terminated Task>" Id=61 RUNNABLE (in native code)
at java.base@11.0.15/sun.nio.fs.WindowsNativeDispatcher.CreateFile0(Native Method)
at java.base@11.0.15/sun.nio.fs.WindowsNativeDispatcher.CreateFile(WindowsNativeDispatcher.java:71)
at java.base@11.0.15/sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:308)
at java.base@11.0.15/sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:168)
at java.base@11.0.15/sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(WindowsFileSystemProvider.java:330)
at java.base@11.0.15/sun.nio.fs.WindowsFileSystemProvider.checkAccess(WindowsFileSystemProvider.java:367)
at java.base@11.0.15/sun.nio.fs.AbstractFileSystemProvider.exists(AbstractFileSystemProvider.java:151)
at java.base@11.0.15/java.nio.file.Files.exists(Files.java:2435)
at org.apache.nifi.processors.standard.PutFile.onTrigger(PutFile.java:315)
at app//org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base@11.0.15/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base@11.0.15/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.15/java.lang.Thread.run(Thread.java:829)
Number of Locked Synchronizers: 1
"Timer-Driven Process Thread-10" Id=84 RUNNABLE (in native code)
at java.base@11.0.15/java.io.WinNTFileSystem.list(Native Method)
at java.base@11.0.15/java.io.File.normalizedList(File.java:1165)
at java.base@11.0.15/java.io.File.listFiles(File.java:1258)
at org.apache.nifi.processors.standard.GetFile.performListing(GetFile.java:321)
at org.apache.nifi.processors.standard.GetFile.onTrigger(GetFile.java:386)
at app//org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base@11.0.15/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base@11.0.15/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.15/java.lang.Thread.run(Thread.java:829)
Number of Locked Synchronizers: 2
- java.util.concurrent.locks.ReentrantLock$NonfairSync@6d7d76f3
- java.util.concurrent.ThreadPoolExecutor$Worker@153e2f2f
"Timer-Driven Process Thread-11" Id=6746 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bb8fee2
at java.base@11.0.15/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.15/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11.0.15/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.15/java.lang.Thread.run(Thread.java:829)
"Timer-Driven Process Thread-12 <Terminated Task>" Id=6750 RUNNABLE (in native code)
at java.base@11.0.15/sun.nio.fs.WindowsNativeDispatcher.CreateFile0(Native Method)
at java.base@11.0.15/sun.nio.fs.WindowsNativeDispatcher.CreateFile(WindowsNativeDispatcher.java:71)
at java.base@11.0.15/sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:308)
at java.base@11.0.15/sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:168)
at java.base@11.0.15/sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(WindowsFileSystemProvider.java:330)
at java.base@11.0.15/sun.nio.fs.WindowsFileSystemProvider.checkAccess(WindowsFileSystemProvider.java:367)
at java.base@11.0.15/sun.nio.fs.AbstractFileSystemProvider.exists(AbstractFileSystemProvider.java:151)
at java.base@11.0.15/java.nio.file.Files.exists(Files.java:2435)
at org.apache.nifi.processors.standard.PutFile.onTrigger(PutFile.java:315)
at app//org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent$$Lambda$1526/0x0000000800f54040.run(Unknown Source)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base@11.0.15/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base@11.0.15/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.15/java.lang.Thread.run(Thread.java:829)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@bbdcc1d
"Timer-Driven Process Thread-13" Id=6751 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bb8fee2
at java.base@11.0.15/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.15/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11.0.15/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.15/java.lang.Thread.run(Thread.java:829) In the Nifi-app.log does not exists a related message. As I mentioned before, this processor (or its process) has no had changes recently. But a new process that has several PutFile processor with high usage was recently deployed on the Nifi instance. However this new process is not having issues and is working without problem while the old process is having the described problem. I'll appreciate your help -------------------------- Nifi 1.14.0 OpenJDK 64-Bit Server VM Microsoft-32930 (build 11.0.15+10-LTS, mixed mode) Windows Server Std 2016 (10.0.14.393 b 14393) Allocate RAM 8G
... View more
Labels:
- Labels:
-
Apache NiFi
07-22-2022
08:59 AM
Hi All, The problem was on the JDK version. We were using OpenJDK 11.0.2 which had a bug in the TLS handshake. Solution: Upgrade JDK (now using 11.0.15).
... View more
07-12-2022
06:44 PM
Does the target URL hostname go to a LB that can route that request to any number of endpoint servers? I don't really know, but thing that this is the scenario. Perhaps not all those endpoint servers use the same serverAuth certificate or are not all signed by the same authority or an authority known the the truststore configured in the SSLContextService you have configured in your invokeHTTP processor. Is possible. I added the root certificate available on CA Authority to the keystore and will monitor the behavior.
... View more
07-12-2022
05:44 PM
Hi @araujo, Just today I detected more than 150 errors over around 400 HTTP request.
... View more
07-06-2022
12:38 PM
Hi, I am very frequently getting the following exception whenever I call the URL seen in the message. 2022-07-06 15:24:56,138 ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=33351c02-15ee-1ccc-6c00-dda0e16110ca] Routing to Failure due to exception: javax.net.ssl.SSLPeerUnverifiedException: Hostname creatorapp.zoho.com not verified (no certificates): javax.net.ssl.SSLPeerUnverifiedException: Hostname creatorapp.zoho.com not verified (no certificates)
javax.net.ssl.SSLPeerUnverifiedException: Hostname creatorapp.zoho.com not verified (no certificates)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:396)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:850)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834) Some details about it: 1. The URL certificate is valid 2. The request is processed correctly on the second or third try and if several flowfiles are queued, the following messages may or may not present the same behavior. But usually the request is always processed. 3. I am using the self-signed certificate generated by Nifi. Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
04-02-2022
02:14 PM
Hi André, It works, however I'm still facing problems to start the MiNiFi, now I'm seeing the following errors in the nifi-app.log 2022-04-02 17:25:56,124 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 22, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,140 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 75, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,140 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 126, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,140 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 177, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,155 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 212, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,155 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 242, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,155 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 277, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,155 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 359, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,155 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 407, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,155 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 446, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,171 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 486, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,171 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 549, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,171 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 612, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected.
2022-04-02 17:25:56,191 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 680, col 28: cvc-complex-type.2.4.a: Invalid content was found starting with element 'networkInterface'. One of '{inputPort, outputPort}' is expected.
2022-04-02 17:25:56,311 INFO [main] o.a.nifi.headless.HeadlessNiFiServer Loading Flow...
2022-04-02 17:25:56,343 WARN [main] o.a.nifi.headless.HeadlessNiFiServer Failed to start headless server... shutting down.
java.lang.Exception: Unable to load flow due to: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required
at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:166)
at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:163)
at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:64)
at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265)
Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required
at org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42)
at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:125)
... 3 common frames omitted
2022-04-02 17:25:56,343 INFO [Thread-1] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server...
2022-04-02 17:25:56,343 WARN [Thread-1] o.a.nifi.headless.HeadlessNiFiServer Problem occurred ensuring flow controller or repository was properly terminated due to java.lang.NullPointerException
2022-04-02 17:25:56,343 ERROR [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi
2022-04-02 17:25:56,343 INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise). About the warnings I could not find which processors in the flow has the maxconcurrentask unset, all have one as value (1). About the network interface, input and output ports warnings I suppose that it refers to the remote process group. In this point, I understand that, at least the network interface, is an optional parameter. The final error, nifi.sensitive.props.key. I seted this property but when minifi try to starts set the value to empty again and then fails. Thanks again.
... View more
04-01-2022
07:05 PM
Hi, I'm trying to deploy a minifi instance in Windows Server. The process deployed basically waits for a file, read it and send the content to the Nifi instance in other server in the same network. Template's transformation finished OK and none changes have been made D:\minifi-1.14.0\minifi-toolkit-1.14.0\bin>config.bat transform Captura_Log_de_Errrores_Remotos.xml Captura_Log_de_Errrores_Remotos.yml
No validation errors found in converted configuration. My problem is that MiNiFi is not starting with the following exception: c:\minifi\bin>run-minifi.bat
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/minifi/lib/bootstrap/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/minifi/lib/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
Exception in thread "main" java.io.IOException: Unable to successfully transform the provided configuration
at org.apache.nifi.minifi.bootstrap.RunMiNiFi.performTransformation(RunMiNiFi.java:1703)
at org.apache.nifi.minifi.bootstrap.RunMiNiFi.start(RunMiNiFi.java:1180)
at org.apache.nifi.minifi.bootstrap.RunMiNiFi.main(RunMiNiFi.java:244)
Caused by: java.io.IOException: org.yaml.snakeyaml.error.YAMLException: java.nio.charset.MalformedInputException: Input length = 1
at org.apache.nifi.minifi.commons.schema.serialization.SchemaLoader.loadYamlAsMap(SchemaLoader.java:64)
at org.apache.nifi.minifi.commons.schema.serialization.SchemaLoader.loadConvertableSchemaFromYaml(SchemaLoader.java:79)
at org.apache.nifi.minifi.bootstrap.util.ConfigTransformer.transformConfigFile(ConfigTransformer.java:94)
at org.apache.nifi.minifi.bootstrap.RunMiNiFi.performTransformation(RunMiNiFi.java:1693)
... 2 more
Caused by: org.yaml.snakeyaml.error.YAMLException: java.nio.charset.MalformedInputException: Input length = 1
at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:200)
at org.yaml.snakeyaml.reader.StreamReader.peek(StreamReader.java:146)
at org.yaml.snakeyaml.scanner.ScannerImpl.scanPlain(ScannerImpl.java:2000)
at org.yaml.snakeyaml.scanner.ScannerImpl.fetchPlain(ScannerImpl.java:1046)
at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:401)
at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:226)
at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:557)
at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:157)
at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:147)
at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:224)
at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:155)
at org.yaml.snakeyaml.composer.Composer.composeSequenceNode(Composer.java:199)
at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:153)
at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:246)
at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:237)
at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:225)
at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:155)
at org.yaml.snakeyaml.composer.Composer.composeSequenceNode(Composer.java:199)
at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:153)
at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:246)
at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:237)
at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:225)
at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:155)
at org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122)
at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105)
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120)
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:450)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:381)
at org.apache.nifi.minifi.commons.schema.serialization.SchemaLoader.loadYamlAsMap(SchemaLoader.java:55)
... 5 more
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at org.yaml.snakeyaml.reader.UnicodeReader.read(UnicodeReader.java:123)
at java.io.Reader.read(Reader.java:140)
at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:184)
... 33 more Nifi's configuration is: Port 10000 is up on Nifi's server and can be accessed from MiniFi's server... I'll appreciate your help
... View more
- Tags:
- minifi
- windows server
Labels:
- Labels:
-
Apache NiFi
03-31-2022
10:06 AM
1 Kudo
Hi André, That is what I needed. Thanks a lot. For those who are facing the same problem in windows; @araujo's solution also works with Powershell
... View more
03-29-2022
12:34 PM
Hi, An external process creates an error.file on certain events and updates the process log file. The process that I'm creating starts when the error.file is created and need to read the last 10 - 15 lines of the log.file. My problem is that the log.file does not rollover and just grows continually and neither follows a log standard in order to use SysLog processors. How can I get that last 10 - 15 lines of the log file? Something like ( "tail -15 log.file" ). Thanks in advance.
... View more
- Tags:
- NiFi
Labels:
- Labels:
-
Apache NiFi
03-03-2022
06:07 PM
1 Kudo
Hi @araujo , It is major head ache for me your findings as this is a crucial step in the process. But, at least I know what is happening. I will try to use another smtp server to check if the problem decreases. Unfortunately we don't have Cloudera support. I just opened the issue NIFI-9758 Thanks a lot for your help. Regards, Javier
... View more
03-03-2022
05:55 AM
Hi @araujo , Here you can find the nifi-app.log in Debug mode. This is the flowfile that was enqueue (just one flowfile when I checked) In the logs appears that the processor received the flowfile at 16:31:37,479 but almost instantly the process dyed -without clear reason- , and the mail only was send until 17:42:20,913 after I terminated/restarted the processor. The dump did not show anything different from my first post. As I can see, in the mean time the other Processor's task (Concurrent tasks = 2) continue working without problem. Also I noticed that while I was attempting to terminate the Processor the queue was appearing with one flow file enqueued, but when I tried to list the queue I got the message "The queue has no FlowFiles". I don't know if this is the normal behavior. Thanks
... View more
03-01-2022
08:16 AM
@araujo After month close, I have an approved manteinance window for tomorrow night. I tried to enable debug in in test instance, changing the root tag in the logback.xml <root level="DEBUG"> <appender-ref ref="APP_FILE" /> </root> But I got several hundreds of Mb in the log files in a few minutes. As I don't when the issue will appear; does exist some way to increase the log level in the nifi-app, but only for the PutEmail processor or its associate classes? I can see that related clasess to PutEmail are (until I can see): com.sun.mail.smtp jakarta.activation com.sun.mail.smtp jakarta.mail thanks
... View more
02-23-2022
02:54 PM
hi @araujo, Thanks for your reply. I can't see anything relevant on nifi-app or in the PutEmail Debug. Following logs are from an event that happened earlier today were 3 flow files were enqueued. I terminate/start the PutEmail processor and just 2 flow files were processed before the processor dyed again. I repeated the terminate/start action and the processor have been working without problem since then. PutEmail Debug Sent email as a result of receiving StandardFlowFileRecord[uuid=41e2361c-a94f-4b4c-b8d3-bdb558c92ae4,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1645643212828-491, container=default, section=491], offset=105253, length=104579],offset=0, name=33_76209474-6_41326.pdf, size=104579] Sent email as a result of receiving StandardFlowFileRecord[uuid=e05651dd-a507-4873-a291-fc840ce60b89,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1645643170184-490, container=default, section=490], offset=511202, length=118027],offset=0,name=34_76209474-6_45376.pdf, size=118027] <the processor just dyed, no goodbye, no error, no nothing!!> [Terminated Process] - DEBUG - Reloading component 8d061908-32f1-1b27-7653-ff7e9404c34a to type org.apache.nifi.processors.standard.PutEmail from bundle org.apache.nifi:nifi-standard-nar:1.14.0 Sent email as a result of receiving StandardFlowFileRecord[uuid=a3ec8b22-850a-4788-b191-cb1faef454f9,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1645643289194-492, container=default, section=492], offset=868, length=103410],offset=0,name=33_83574800-6_41327.pdf, size=103410] Sent email as a result of receiving StandardFlowFileRecord[uuid=e43076fc-5baa-4d3b-8f50-e198524dff93,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1645643289194-492, container=default, section=492], offset=104278, length=116147],offset=0,name=34_83574800-6_45377.pdf, size=116147] [Terminated Process] - DEBUG - Reloading component 8d061908-32f1-1b27-7653-ff7e9404c34a to type org.apache.nifi.processors.standard.PutEmail from bundle org.apache.nifi:nifi-standard-nar:1.14.0 Sent email as a result of receiving StandardFlowFileRecord[uuid=4478a2ce-e9f6-4bd5-b63b-6549531e697a,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1645641281073-449, container=default, section=449], offset=127565, length=102849],offset=0,name=33_79921700-7_41302.pdf, size=102849] Sent email as a result of receiving StandardFlowFileRecord[uuid=880053b8-336b-4ceb- a7eb-371118014b1b,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1645643289319-493, container=default, section=493], offset=620752, length=99647],offset=0,name= 2022-02-23T16_28_31_T34F8251_76450608-1.pdf, size=99647] nifi-app Does not exist any different message in the log while the event happened INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 124 records in 0 milliseconds INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 124 records in 0 milliseconds INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5a5cd0af checkpointed with 334 Records and 0 Swap Files in 55 milliseconds (Stop-the-world time = 24 milliseconds, Clear Edit Logs time = 24 millis), max Transaction ID 38437 Thanks
... View more
02-18-2022
08:46 AM
Hi, In my process I'm triggering some emails at certain points. My problem is that the PutEmail processor just keep dying frequently (even two or three times per day)!!! Nifi Version 1.14.0 When processor dyes I have to terminate and restart it. This is the dump of the last crash: "Timer-Driven Process Thread-1" Id=68 RUNNABLE (in native code) at java.base@11.0.2/java.net.SocketInputStream.socketRead0(Native Method) at java.base@11.0.2/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) at java.base@11.0.2/java.net.SocketInputStream.read(SocketInputStream.java:168) at java.base@11.0.2/java.net.SocketInputStream.read(SocketInputStream.java:140) at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:102) at java.base@11.0.2/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) at java.base@11.0.2/java.io.BufferedInputStream.read(BufferedInputStream.java:271) - waiting on java.io.BufferedInputStream@40399b64 at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:100) at com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:2456) at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2172) at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:722) - waiting on com.sun.mail.smtp.SMTPTransport@4fa5e86a at jakarta.mail.Service.connect(Service.java:364) - waiting on com.sun.mail.smtp.SMTPTransport@4fa5e86a at jakarta.mail.Service.connect(Service.java:222) at jakarta.mail.Service.connect(Service.java:171) at jakarta.mail.Transport.send0(Transport.java:230) at jakarta.mail.Transport.send(Transport.java:100) at org.apache.nifi.processors.standard.PutEmail.send(PutEmail.java:524) at org.apache.nifi.processors.standard.PutEmail.onTrigger(PutEmail.java:392) at app//org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run (TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.2/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:305) at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@39b265c9 "Timer-Driven Process Thread-10 <Terminated Task>" Id=92 RUNNABLE (in native code) at java.base@11.0.2/java.net.SocketInputStream.socketRead0(Native Method) at java.base@11.0.2/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) at java.base@11.0.2/java.net.SocketInputStream.read(SocketInputStream.java:168) at java.base@11.0.2/java.net.SocketInputStream.read(SocketInputStream.java:140) at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:102) at java.base@11.0.2/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) at java.base@11.0.2/java.io.BufferedInputStream.read(BufferedInputStream.java:271) - waiting on java.io.BufferedInputStream@7fe5124a at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:100) at com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:2456) at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2172) at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:722) - waiting on com.sun.mail.smtp.SMTPTransport@17e56d13 at jakarta.mail.Service.connect(Service.java:364) - waiting on com.sun.mail.smtp.SMTPTransport@17e56d13 at jakarta.mail.Service.connect(Service.java:222) at jakarta.mail.Service.connect(Service.java:171) at jakarta.mail.Transport.send0(Transport.java:230) at jakarta.mail.Transport.send(Transport.java:100) at org.apache.nifi.processors.standard.PutEmail.send(PutEmail.java:524) at org.apache.nifi.processors.standard.PutEmail.onTrigger(PutEmail.java:392) at app//org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run (TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.2/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.2/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@176d79dc I can find the reason (and the corrective action to take) why is this happening as the remain processors works fine. I'll appreciate your help pointing me in the right direction.
... View more
Labels:
- Labels:
-
Apache NiFi
06-10-2021
03:02 PM
Issue still present in 1.13.2. No workaround yet. It appears that will be fixed in 1.14 according to Jira NIFI-8400
... View more
06-09-2021
06:18 PM
Thanks for this excellent post, it works almost perfect in 1.13.0 version. The only problem I had was when did the last step (13) and the ConvertRecord processor became invalid with the following message: I found the solution to this issue in this post of @pvillard where he updates the schema.name property before de ConvertRecord processor. I didn't complete understand the reason but it works.
... View more
03-05-2021
01:23 PM
Hello, I am deploying NIFI 1.13.0 on Windows 10 and Windows Server and as soon as the instance is active it starts logging the following error: 2021-03-05 15:57:39,728 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.controller.FlowController Failed to capture component stats for Stats History
java.lang.NullPointerException: null
at org.apache.nifi.diagnostics.SystemDiagnostics.getOpenFileHandles(SystemDiagnostics.java:224)
at org.apache.nifi.controller.FlowController.getNodeStatusSnapshot(FlowController.java:3011)
at org.apache.nifi.controller.FlowController.access$400(FlowController.java:230)
at org.apache.nifi.controller.FlowController$2.run(FlowController.java:697)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834) This behavior is the same in an open box instance (without any configuration) and in secured instances (tls-toolkit); and at first glance it doesn't seem to have any impact on the behavior of the instance functionality, but it causes rapid growth of the nifi-app.log file and makes it difficult to read. This error does not appear in 1.12. I'll appreciate your help. Thanks, Temp
... View more
Labels:
- Labels:
-
Apache NiFi
