HI ALL, i am try to read multiple files from particular location. Logs will generate every 1 minute , it will create new file. i am getting below error.  ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.TailFile TailFile[id=3f714fb2-72eb-1376-037b-167383110223] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.util.regex.PatternSyntaxException: Unknown character property name {In/IsY} near index 14 D:\Softwares\PYTHON-YOUTUBE\SIMULATION\modbus_logs\Q\\E^modbus_log\.\d{4}-\d{2}-\d{2}_\d{2}-\d{2}$               ^ java.util.regex.PatternSyntaxException: Unknown character property name {In/IsY} near index 14 D:\Softwares\PYTHON-YOUTUBE\SIMULATION\modbus_logs\Q\\E^modbus_log\.\d{4}-\d{2}-\d{2}_\d{2}-\d{2}$               ^     at java.base/java.util.regex.Pattern.error(Pattern.java:2027)     at java.base/java.util.regex.Pattern.family(Pattern.java:2916)     at java.base/java.util.regex.Pattern.sequence(Pattern.java:2155)     at java.base/java.util.regex.Pattern.expr(Pattern.java:2068)     at java.base/java.util.regex.Pattern.compile(Pattern.java:1782)     at java.base/java.util.regex.Pattern.<init>(Pattern.java:1428)     at java.base/java.util.regex.Pattern.compile(Pattern.java:1068)     at org.apache.nifi.processors.standard.TailFile.getFilesToTail(TailFile.java:525)     at org.apache.nifi.processors.standard.TailFile.lookup(TailFile.java:386)     at org.apache.nifi.processors.standard.TailFile.recoverState(TailFile.java:410)     at jdk.internal.reflect.GeneratedMethodAccessor1142.invoke(Unknown Source)     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.base/java.lang.reflect.Method.invoke(Method.java:566)     at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)     at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)     at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)     at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)     at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$8(StandardProcessorNode.java:1736)     at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)     at java.base/java.lang.Thread.run(Thread.java:834)     How to resolve this issue.   in modbus_logs dir file name are like modbus_log.2023-09-06_16-16,modbus_log.2023-09-06_16-17,modbus_log.2023-09-06_16-18,modbus_log.2023-09-06_16-19. ... View more

Hi all,    Putelasticsearchhttp delete and upsert is  not working.  Fetchelasticsearch is giving error code 400.   Thanks  Kumar  ... View more

Hi, we are running nifi 1.11.4 version .we had resource issue before recently RAM and CPU core increase in server. we changed bootstrap.conf file  java.arg.2=-Xms20g java.arg.3=-Xmx40g Total 1100 processors are running in nifi,  around 150+ processor disable (in disable processors have queue with 3GB). After restart nifi it is working fine for sometime ,after that Nifi ui response is too slowly. Even for refresh option it is taking 1 minute +. After sometime we are unable to access nifi ui. it is not loading anything but in server nifi is running telnet is working fine. os: redhat RAM : 64 GB CPU : 16GB java:1.8 unable to find the actually issue.   nifi-api/system-diagnostics {"systemDiagnostics":{"aggregateSnapshot":{"totalNonHeap":"366 MB","totalNonHeapBytes":383778816,"usedNonHeap":"273.25 MB","usedNonHeapBytes":286525256,"freeNonHeap":"92.75 MB","freeNonHeapBytes":97253560,"maxNonHeap":"-1 bytes","maxNonHeapBytes":-1,"totalHeap":"38.34 GB","totalHeapBytes":41163423744,"usedHeap":"35.97 GB","usedHeapBytes":38624675432,"freeHeap":"2.36 GB","freeHeapBytes":2538748312,"maxHeap":"38.34 GB","maxHeapBytes":41163423744,"heapUtilization":"94.0%","availableProcessors":20,"processorLoadAverage":14.13,"totalThreads":338,"daemonThreads":155,"uptime":"91:55:36.089","flowFileRepositoryStorageUsage":{"freeSpace":"145.38 GB","totalSpace":"300 GB","usedSpace":"154.62 GB","freeSpaceBytes":156101574656,"totalSpaceBytes":322122547200,"usedSpaceBytes":166020972544,"utilization":"52.0%"},"contentRepositoryStorageUsage":[{"identifier":"default","freeSpace":"145.38 GB","totalSpace":"300 GB","usedSpace":"154.62 GB","freeSpaceBytes":156101574656,"totalSpaceBytes":322122547200,"usedSpaceBytes":166020972544,"utilization":"52.0%"}],"provenanceRepositoryStorageUsage":[{"identifier":"default","freeSpace":"145.38 GB","totalSpace":"300 GB","usedSpace":"154.62 GB","freeSpaceBytes":156101574656,"totalSpaceBytes":322122547200,"usedSpaceBytes":166020972544,"utilization":"52.0%"}],"garbageCollection":[{"name":"PS Scavenge","collectionCount":2458,"collectionTime":"00:52:34.376","collectionMillis":3154376},{"name":"PS MarkSweep","collectionCount":4935,"collectionTime":"64:44:42.397","collectionMillis":233082397}],"statsLastRefreshed":"10:40:31 IST","versionInfo":{"niFiVersion":"1.11.4","javaVendor":"Oracle Corporation","javaVersion":"1.8.0_242","osName":"Linux","osVersion":"3.10.0-1062.12.1.el7.x86_64","osArchitecture":"amd64","buildTag":"nifi-1.11.4-RC1","buildTimestamp":"03/18/2020 12:42:54 IST"}}}} ... View more

We are using ubuntu:20.04 OS. Any configuration missed out? Any solution for this issue. Zookeeper cluster is working well.when we try to use zookeeper in nifi at the point of time nifi is giving error. zookeeper configuration : same configuration is available in 3 servers  tickTime=2000 dataDir=/opt/zookeeper/data clientPort=2181 maxClientCnxns=60 initLimit=10 syncLimit=5 server.1=nifi-zoo:2888:3888 server.2=nifi-zoo-2:2888:3888 server.3=nifi-zoo-3:2888:3888 nifi configuration nifi-config-1 # State Management # #################### nifi.state.management.configuration.file=./conf/state-management.xml nifi.state.management.provider.local=local-provider nifi.state.management.provider.cluster=zk-provider nifi.state.management.embedded.zookeeper.start=false nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties # Site to Site properties nifi.remote.input.host=nifi-zoo nifi.remote.input.secure=false nifi.remote.input.socket.port= nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.contents.cache.expiration=30 secs # web properties # nifi.web.http.host=nifi-zoo nifi.web.http.port=8080 nifi.web.http.network.interface.default= nifi.web.https.host= nifi.web.https.port= nifi.web.https.network.interface.default= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=400 nifi.web.max.header.size=16 KB nifi.web.proxy.context.path= nifi.web.proxy.host= nifi.web.max.content.size= nifi.web.max.requests.per.second=30000 nifi.web.should.send.server.version=true # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=nifi-zoo nifi.cluster.node.protocol.port=12000 nifi.cluster.node.protocol.threads=20 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=60 sec nifi.cluster.node.read.timeout=60 sec nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=30 sec nifi.cluster.flow.election.max.candidates= # cluster load balancing properties # nifi.cluster.load.balance.host= nifi.cluster.load.balance.port=6342 nifi.cluster.load.balance.connections.per.node=1 nifi.cluster.load.balance.max.thread.count=8 nifi.cluster.load.balance.comms.timeout=30 sec # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=nifi-zoo:2181,nifi-zoo-2:2181,nifi-zoo-3:2181 nifi.zookeeper.connect.timeout=60 secs nifi.zookeeper.session.timeout=60 secs nifi.zookeeper.root.node=/nifi nifi-config-2 # State Management # #################### nifi.state.management.configuration.file=./conf/state-management.xml nifi.state.management.provider.local=local-provider nifi.state.management.provider.cluster=zk-provider nifi.state.management.embedded.zookeeper.start=false nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties # Site to Site properties nifi.remote.input.host=nifi-zoo-2 nifi.remote.input.secure=false nifi.remote.input.socket.port= nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.contents.cache.expiration=30 secs # web properties # nifi.web.http.host=nifi-zoo-2 nifi.web.http.port=8080 nifi.web.http.network.interface.default= nifi.web.https.host= nifi.web.https.port= nifi.web.https.network.interface.default= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=400 nifi.web.max.header.size=16 KB nifi.web.proxy.context.path= nifi.web.proxy.host= nifi.web.max.content.size= nifi.web.max.requests.per.second=30000 nifi.web.should.send.server.version=true # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=nifi-zoo-2 nifi.cluster.node.protocol.port=12000 nifi.cluster.node.protocol.threads=20 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=60 sec nifi.cluster.node.read.timeout=60 sec nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=30 sec nifi.cluster.flow.election.max.candidates= # cluster load balancing properties # nifi.cluster.load.balance.host= nifi.cluster.load.balance.port=6342 nifi.cluster.load.balance.connections.per.node=1 nifi.cluster.load.balance.max.thread.count=8 nifi.cluster.load.balance.comms.timeout=30 sec # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=nifi-zoo:2181,nifi-zoo-2:2181,nifi-zoo-3:2181 nifi.zookeeper.connect.timeout=60 secs nifi.zookeeper.session.timeout=60 secs nifi.zookeeper.root.node=/nifi   nifi-config-3   # State Management # #################### nifi.state.management.configuration.file=./conf/state-management.xml nifi.state.management.provider.local=local-provider nifi.state.management.provider.cluster=zk-provider nifi.state.management.embedded.zookeeper.start=false nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties # Site to Site properties nifi.remote.input.host=nifi-zoo-3 nifi.remote.input.secure=false nifi.remote.input.socket.port= nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.contents.cache.expiration=30 secs # web properties # nifi.web.http.host=nifi-zoo-3 nifi.web.http.port=8080 nifi.web.http.network.interface.default= nifi.web.https.host= nifi.web.https.port= nifi.web.https.network.interface.default= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=400 nifi.web.max.header.size=16 KB nifi.web.proxy.context.path= nifi.web.proxy.host= nifi.web.max.content.size= nifi.web.max.requests.per.second=30000 nifi.web.should.send.server.version=true # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=nifi-zoo-3 nifi.cluster.node.protocol.port=12000 nifi.cluster.node.protocol.threads=20 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=60 sec nifi.cluster.node.read.timeout=60 sec nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=30 sec nifi.cluster.flow.election.max.candidates= # cluster load balancing properties # nifi.cluster.load.balance.host= nifi.cluster.load.balance.port=6342 nifi.cluster.load.balance.connections.per.node=1 nifi.cluster.load.balance.max.thread.count=8 nifi.cluster.load.balance.comms.timeout=30 sec # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=nifi-zoo:2181,nifi-zoo-2:2181,nifi-zoo-3:2181 nifi.zookeeper.connect.timeout=60 secs nifi.zookeeper.session.timeout=60 secs nifi.zookeeper.root.node=/nifi     NIFI ERROR:   2021-02-04 11:12:47,783 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED 2021-02-04 11:12:49,106 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED 2021-02-04 11:12:49,107 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to RECONNECTED 2021-02-04 11:12:49,107 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to RECONNECTED 2021-02-04 11:12:49,209 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2021-02-04 11:12:49,209 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to SUSPENDED 2021-02-04 11:12:49,209 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED 2021-02-04 11:12:49,506 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED 2021-02-04 11:12:49,507 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to RECONNECTED 2021-02-04 11:12:49,507 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to RECONNECTED 2021-02-04 11:12:49,508 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2021-02-04 11:12:49,508 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to SUSPENDED 2021-02-04 11:12:49,508 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED 2021-02-04 11:12:49,508 ERROR [main-EventThread] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:647) at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) at org.apache.curator.framework.imps.FindAndDeleteProtectedNodeInBackground$2.processResult(FindAndDeleteProtectedNodeInBackground.java:104) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:630) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2021-02-04 11:12:50,214 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED 2021-02-04 11:12:50,214 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to RECONNECTED 2021-02-04 11:12:50,214 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to RECONNECTED 2021-02-04 11:12:50,316 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2021-02-04 11:12:50,316 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to SUSPENDED 2021-02-04 11:12:50,316 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED 2021-02-04 11:12:50,434 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED 2021-02-04 11:12:50,434 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to RECONNECTED 2021-02-04 11:12:50,434 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to RECONNECTED 2021-02-04 11:12:50,536 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2021-02-04 11:12:50,536 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to SUSPENDED 2021-02-04 11:12:50,536 ERROR [main-EventThread] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:647) at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) at org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult(GetConfigBuilderImpl.java:222) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:601) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2021-02-04 11:12:50,536 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED 2021-02-04 11:12:51,990 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED 2021-02-04 11:12:51,990 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to RECONNECTED 2021-02-04 11:12:51,990 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to RECONNECTED 2021-02-04 11:12:51,991 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected 2021-02-04 11:12:52,092 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2021-02-04 11:12:52,093 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to SUSPENDED 2021-02-04 11:12:52,093 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED ... View more