Member since
06-11-2020
19
Posts
1
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5387 | 01-16-2022 08:51 PM | |
1301 | 02-01-2021 11:39 AM | |
946 | 01-06-2021 02:29 PM | |
1434 | 10-17-2020 08:11 AM | |
2085 | 09-28-2020 09:58 PM |
01-16-2022
08:51 PM
At the end of the day we had allocated more memory than was available on the server when under stress. I recommend reducing memory and restarting the cluster.
... View more
05-17-2021
06:03 AM
@MattWho , thanks for your response. Before I had heard from you I had checked /var/log/messages and searched for OOM messages without success. However, there was no indications of nifi dying in nifi-app.log so I dug a little farther. As it turned out I did find the OS killing the child process in NIFI. I reduced the JVM to 24gb and the OS no longer feels the need to kill it. So that problem is solved. As for the JVM, we were attempting to keep as much of our flow queues in memory to reduce disk i/o. Is there a better strategy?
... View more
05-14-2021
10:02 AM
NIFI 1.13.2 has nodes randomly restarting with connections refused. cluster information 12 nodes 8 cores 32 gb ram bootstrap.conf java=java run.as= lib.dir=./lib conf.dir=./conf graceful.shutdown.seconds=20 java.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=true java.arg.2=-Xms28g java.arg.3=-Xmx28gjava.arg.4=-Djava.net.preferIPv4Stack=true java.arg.5=-Dsun.net.http.allowRestrictedHeaders=true java.arg.6=-Djava.protocol.handler.pkgs=sun.net.www.protocol java.arg.7=-XX:ReservedCodeCacheSize=256m java.arg.8=-XX:CodeCacheMinimumFreeSpace=10m java.arg.9=-XX:+UseCodeCacheFlushingjava.arg.14=-Djava.awt.headless=true nifi.bootstrap.sensitive.key= java.arg.15=-Djava.security.egd=file:/dev/urandom java.arg.16=-Djavax.security.auth.useSubjectCredsOnly=true java.arg.17=-Dzookeeper.admin.enableServer=falsejava.arg.snappy=-Dorg.xerial.snappy.tempdir=/opt/nifi/tmp notification.services.file=./conf/bootstrap-notification-services.xml notification.max.attempts=5 # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true #nifi.cluster.node.address=ip-xx-xxx-xxx-xxx.us-gov-west-1.compute.internal nifi.cluster.node.address=192.170.108.140 nifi.cluster.node.protocol.port=11443 nifi.cluster.node.protocol.threads=100 nifi.cluster.node.protocol.max.threads=800 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=60 sec nifi.cluster.node.read.timeout=60 sec nifi.cluster.node.max.concurrent.requests=800 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=5 mins nifi.cluster.flow.election.max.candidates=7 # cluster load balancing properties # nifi.cluster.load.balance.host=192.170.108.140 nifi.cluster.load.balance.port=6342 nifi.cluster.load.balance.connections.per.node=50 nifi.cluster.load.balance.max.thread.count=600 nifi.cluster.load.balance.comms.timeout=45 sec # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=192.170.108.37:2181,192.170.108.67:2181,192.170.108.120:2181,192.170.108.104:2181,192.170.108.106:2181 nifi.zookeeper.connect.timeout=30 secs nifi.zookeeper.session.timeout=30 secs nifi.zookeeper.root.node=/nifi_tf Sample Error 2021-05-14 13:00:11,927 ERROR [Load-Balanced Client Thread-356] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to nifi-tf-11.bogus-dns.pvt:9443 for load balancing java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:482) at sun.nio.ch.Net.connect(Net.java:474) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:456) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:399) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:211) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-05-14 13:00:15,592 ERROR [Load-Balanced Client Thread-160] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to nifi-tf-11.bogus-dns.pvt:9443 for load balancing System Diagnostics
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
05-14-2021
09:41 AM
1 Kudo
We have a new 1.13.2 NIFI Cluster. The cluster has nodes randomly restarting and we are unable to determine the cause. We do see 'connection refused' errors intermittantly without explanation in the logs (attached). We don't know what additional steps are to take at this point. nifi.properties and latest nifi-app.log. Cluster Details 12 x 8 cores - 32gb of Memory These servers leverage a 5 node Zookeeper Quorum, we have not seen latency or file handler issues there. NIFI Properties File (redacted to protect the innocent) # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Core Properties # nifi.flow.configuration.file=./conf/flow.xml.gz nifi.flow.configuration.archive.enabled=true nifi.flow.configuration.archive.dir=./conf/archive/ nifi.flow.configuration.archive.max.time=30 days nifi.flow.configuration.archive.max.storage=500 MB nifi.flow.configuration.archive.max.count= nifi.flowcontroller.autoResumeState=true nifi.flowcontroller.graceful.shutdown.period=10 sec nifi.flowservice.writedelay.interval=500 ms nifi.administrative.yield.duration=30 sec # If a component has no work to do (is "bored"), how long should we wait before checking again for work? nifi.bored.yield.duration=5 millis nifi.queue.backpressure.count=20000 nifi.queue.backpressure.size=1 GB nifi.authorizer.configuration.file=./conf/authorizers.xml nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml nifi.templates.directory=./conf/templates nifi.ui.banner.text= nifi.ui.autorefresh.interval=30 sec nifi.nar.library.directory=./lib nifi.nar.library.autoload.directory=./extensions nifi.nar.working.directory=./work/nar/ nifi.documentation.working.directory=./work/docs/components #################### # State Management # #################### nifi.state.management.configuration.file=./conf/state-management.xml # The ID of the local state provider nifi.state.management.provider.local=local-provider # The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster. nifi.state.management.provider.cluster=zk-provider # Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server nifi.state.management.embedded.zookeeper.start=false # Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties # H2 Settings nifi.database.directory=./database_repository nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE # FlowFile Repository nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog nifi.flowfile.repository.directory=/opt/nifi/data/flow nifi.flowfile.repository.checkpoint.interval=20 secs nifi.flowfile.repository.always.sync=false nifi.flowfile.repository.encryption.key.provider.implementation= nifi.flowfile.repository.encryption.key.provider.location= nifi.flowfile.repository.encryption.key.id= nifi.flowfile.repository.encryption.key= nifi.flowfile.repository.retain.orphaned.flowfiles=true nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager nifi.queue.swap.threshold=50000 # Content Repository nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.claim.max.appendable.size=20 MB nifi.content.repository.directory.default=/opt/nifi/data/content nifi.content.repository.archive.max.retention.period=9 hours nifi.content.repository.archive.max.usage.percentage=95% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false nifi.content.viewer.url=../nifi-content-viewer/ nifi.content.repository.encryption.key.provider.implementation= nifi.content.repository.encryption.key.provider.location= nifi.content.repository.encryption.key.id= nifi.content.repository.encryption.key= # Provenance Repository Properties nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository nifi.provenance.repository.encryption.key.provider.implementation= nifi.provenance.repository.encryption.key.provider.location= nifi.provenance.repository.encryption.key.id= nifi.provenance.repository.encryption.key= # Persistent Provenance Repository Properties nifi.provenance.repository.directory.default=/opt/nifi/data/provenance nifi.provenance.repository.max.storage.time=6 hours nifi.provenance.repository.max.storage.size=595 GB nifi.provenance.repository.rollover.time=1 mins nifi.provenance.repository.rollover.size=1 GB nifi.provenance.repository.query.threads=2 nifi.provenance.repository.index.threads=1 nifi.provenance.repository.compress.on.rollover=true nifi.provenance.repository.always.sync=false nifi.provenance.repository.journal.count=16 # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are: # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship # FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type nifi.provenance.repository.indexed.attributes= # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository # but should provide better performance nifi.provenance.repository.index.shard.size=8 GB # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved. nifi.provenance.repository.max.attribute.length=65536 nifi.provenance.repository.concurrent.merge.threads=2 # Volatile Provenance Respository Properties nifi.provenance.repository.buffer.size=100000 # Component Status Repository nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository nifi.components.status.repository.buffer.size=1440 nifi.components.status.snapshot.frequency=1 min # Site to Site properties nifi.remote.input.host=ip-xx-xxx-xxx-xxx.us-gov-west-1.compute.internal nifi.remote.input.secure=true nifi.remote.input.socket.port=10443 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.contents.cache.expiration=30 secs # web properties # nifi.web.http.host= nifi.web.http.port= nifi.web.http.network.interface.default= nifi.web.https.host=nifi-tf-01.bogus-dns.pvt nifi.web.https.port=9443 nifi.web.https.network.interface.default= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=500 nifi.web.max.header.size=16 KB nifi.web.proxy.context.path= nifi.web.proxy.host= nifi.web.max.content.size= nifi.web.max.requests.per.second=30000 nifi.web.should.send.server.version=true # security properties # nifi.sensitive.props.key= nifi.sensitive.props.key.protected= nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL nifi.sensitive.props.provider=BC nifi.sensitive.props.additional.keys= nifi.security.keystore=/etc/nifi/keystore.jks nifi.security.keystoreType=jks nifi.security.keystorePasswd=pIAF20A4YrqZSkU+BOGUS nifi.security.keyPasswd=pIAF20A4YrqZSkU+BOGUS nifi.security.truststore=/etc/nifi/truststore.jks nifi.security.truststoreType=jks nifi.security.truststorePasswd=notreal+8Q nifi.security.user.authorizer=file-provider nifi.security.allow.anonymous.authentication=false nifi.security.user.login.identity.provider= nifi.security.ocsp.responder.url= nifi.security.ocsp.responder.certificate= # OpenId Connect SSO Properties # nifi.security.user.oidc.discovery.url= nifi.security.user.oidc.connect.timeout=5 secs nifi.security.user.oidc.read.timeout=5 secs nifi.security.user.oidc.client.id= nifi.security.user.oidc.client.secret= nifi.security.user.oidc.preferred.jwsalgorithm= nifi.security.user.oidc.additional.scopes= nifi.security.user.oidc.claim.identifying.user= # Apache Knox SSO Properties # nifi.security.user.knox.url= nifi.security.user.knox.publicKey= nifi.security.user.knox.cookieName=hadoop-jwt nifi.security.user.knox.audiences= # Identity Mapping Properties # # These properties allow normalizing user identities such that identities coming from different identity providers # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing # DNs from certificates and principals from Kerberos into a common identity string: # # nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$ # nifi.security.identity.mapping.value.dn=$1@$2 # nifi.security.identity.mapping.transform.dn=NONE # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$ # nifi.security.identity.mapping.value.kerb=$1@$2 # nifi.security.identity.mapping.transform.kerb=UPPER # Group Mapping Properties # # These properties allow normalizing group names coming from external sources like LDAP. The following example # lowercases any group name. # # nifi.security.group.mapping.pattern.anygroup=^(.*)$ # nifi.security.group.mapping.value.anygroup=$1 # nifi.security.group.mapping.transform.anygroup=LOWER # cluster common properties (all nodes must have same values) # nifi.cluster.protocol.heartbeat.interval=15 sec nifi.cluster.protocol.heartbeat.missable.max=20 nifi.cluster.protocol.is.secure=true # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true #nifi.cluster.node.address=ip-xx-xxx-xxx-xxx.us-gov-west-1.compute.internal nifi.cluster.node.address=192.170.108.140 nifi.cluster.node.protocol.port=11443 nifi.cluster.node.protocol.threads=100 nifi.cluster.node.protocol.max.threads=800 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=60 sec nifi.cluster.node.read.timeout=60 sec nifi.cluster.node.max.concurrent.requests=800 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=5 mins nifi.cluster.flow.election.max.candidates=7 # cluster load balancing properties # nifi.cluster.load.balance.host=192.170.108.140 nifi.cluster.load.balance.port=6342 nifi.cluster.load.balance.connections.per.node=50 nifi.cluster.load.balance.max.thread.count=600 nifi.cluster.load.balance.comms.timeout=45 sec # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=192.170.108.37:2181,192.170.108.67:2181,192.170.108.120:2181,192.170.108.104:2181,192.170.108.106:2181 nifi.zookeeper.connect.timeout=30 secs nifi.zookeeper.session.timeout=30 secs nifi.zookeeper.root.node=/nifi_tf # Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management # Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes # and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node # The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal # and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal # values configured on the zookeeper server). nifi.zookeeper.auth.type= nifi.zookeeper.kerberos.removeHostFromPrincipal= nifi.zookeeper.kerberos.removeRealmFromPrincipal= # kerberos # nifi.kerberos.krb5.file= # kerberos service principal # nifi.kerberos.service.principal= nifi.kerberos.service.keytab.location= # kerberos spnego principal # nifi.kerberos.spnego.principal= nifi.kerberos.spnego.keytab.location= nifi.kerberos.spnego.authentication.expiration=12 hours # external properties files for variable registry # supports a comma delimited list of file locations nifi.variable.registry.properties= # analytics properties # nifi.analytics.predict.enabled=false nifi.analytics.predict.interval=3 mins nifi.analytics.query.interval=5 mins nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares nifi.analytics.connection.model.score.name=rSquared nifi.analytics.connection.model.score.threshold=.90 nifi-app.log (excluding INFO entries) 2021-05-14 13:00:11,927 ERROR [Load-Balanced Client Thread-356] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to nifi-tf-11.bogus-dns.pvt:9443 for load balancing java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:482) at sun.nio.ch.Net.connect(Net.java:474) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:456) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:399) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:211) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-05-14 13:00:15,592 ERROR [Load-Balanced Client Thread-160] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to nifi-tf-11.bogus-dns.pvt:9443 for load balancing java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:482) at sun.nio.ch.Net.connect(Net.java:474) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:456) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:399) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:211) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-05-14 13:00:15,619 ERROR [Load-Balanced Client Thread-498] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to nifi-tf-11.bogus-dns.pvt:9443 for load balancing java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:482) at sun.nio.ch.Net.connect(Net.java:474) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:456) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:399) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:211) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-05-14 13:00:16,217 ERROR [Load-Balanced Client Thread-207] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to nifi-tf-11.bogus-dns.pvt:9443 for load balancing java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:482) at sun.nio.ch.Net.connect(Net.java:474) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:456) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:399) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:211) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-05-14 13:00:16,357 ERROR [Load-Balanced Client Thread-238] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Failed to communicate with Peer nifi-tf-11.bogus-dns.pvt:9443 java.io.IOException: Failed to decrypt data from Peer nifi-tf-11.bogus-dns.pvt:9443 because Peer unexpectedly closed connection at org.apache.nifi.controller.queue.clustered.client.async.nio.PeerChannel.decrypt(PeerChannel.java:269) at org.apache.nifi.controller.queue.clustered.client.async.nio.PeerChannel.read(PeerChannel.java:159) at org.apache.nifi.controller.queue.clustered.client.async.nio.PeerChannel.read(PeerChannel.java:80) at org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.confirmTransactionComplete(LoadBalanceSession.java:177) at org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.communicate(LoadBalanceSession.java:154) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:242) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
02-01-2021
11:39 AM
In this case per the admin guide you have to also copy the users.xml and authorizations.xml to the new node. For some reason these were not being created accurately on instantiation.
... View more
02-01-2021
10:55 AM
I am attempting to add a node to my cluster. My steps are: Unzip a fresh copy of the binary install Copy production authorizer.xml and flow.tar.gz to the new conf directory Update my nifi.properties for the new node this is similar to the a production node with the exception of references to the server itself. When I bring up the new node I see it in 'Connecting' state in cluster manager but it never completes. I receive the following error: 2021-02-01 18:52:13,164 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.NoClassDefFoundError: org/apache/nifi/security/xml/XmlUtils java.lang.NoClassDefFoundError: org/apache/nifi/security/xml/XmlUtils at org.apache.nifi.authorization.AbstractPolicyBasedAuthorizer.parsePoliciesUsersAndGroups(AbstractPolicyBasedAuthorizer.java:416) at org.apache.nifi.authorization.AbstractPolicyBasedAuthorizer.checkInheritability(AbstractPolicyBasedAuthorizer.java:348) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nifi.authorization.AuthorizerInvocationHandler.invoke(AuthorizerInvocationHandler.java:55) at com.sun.proxy.$Proxy82.checkInheritability(Unknown Source) at org.apache.nifi.authorization.AuthorizerFactory$1.checkInheritability(AuthorizerFactory.java:138) at org.apache.nifi.controller.inheritance.AuthorizerCheck.checkInheritability(AuthorizerCheck.java:61) at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:243) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1426) at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:792) at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:519) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1128) at org.apache.nifi.NiFi.<init>(NiFi.java:158) at org.apache.nifi.NiFi.<init>(NiFi.java:72) at org.apache.nifi.NiFi.main(NiFi.java:301) Caused by: java.lang.ClassNotFoundException: org.apache.nifi.security.xml.XmlUtils at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 20 common frames omitted I'm unsure where to start as this method has been successful to add a previous additional node. Any insights would be appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
01-06-2021
02:29 PM
Well, I didn't RATW's... My issue was NOT that my Initial Admin was wrong in the authorizers.xml file. It was that my nodes weren't specified with a space between the , and the OU. Make sure your authorizers.xml is EXACTLY as is created when you create your certificates... if you have issues stop NIFI fix the issue, delete users.xml and authorizations.xml and then restart NIFI. Nothing to see here move along.
... View more
01-06-2021
12:56 PM
First let me say I've searched the board and I've seen the responses and I have a dilemma. Before my cluster establishes I can connect to my servers leveraging Initial Admin and the p12 that I've created. I've verified that all of my nodes are in my authorizations.xml file and reference my users.xml accurately as /proxy resources If I'm not attached to a node prior to joining the cluster the error appears like this After clustering is complete attempt to connect yields the following response in the nifi-users.log 2021-01-06 20:38:38,206 WARN [NiFi Web Server-157] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response. java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi. I don't know where to go from here
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
10-21-2020
09:40 AM
Maybe this can't be done with ExtractText and there are other options
... View more
10-20-2020
12:40 PM
I have the below flowfile contents that I would like to manipulate without splitting the flowfile. There may be a million of these lines. {"ConfigBuild":"1007.8.0010703.9","ConfigStateHash":"1386539727","ContextProcessId":"317927403096194486","ContextThreadId":"0","ContextTimeStamp":"1603206060.314","Entitlements":"15","RawProcessId":"161622","TargetProcessId":"317927403096194486","aid":"1c41ff1d8eff49809ddb44d997ef2fa6","aip":"204.194.122.230","cid":"dff76f718745469994bd39c6a38579aa","event_platform":"Lin","event_simpleName":"TerminateProcess","id":"10c7f679-12e5-11eb-a7cc-0ab5dad5c022","name":"TerminateProcessLinV2","timestamp":"1603206060328"} {"CommandLine":"logger -p cron.notice -t run-parts(/etc/cron.hourly)[115758] starting 0anacron","ConfigBuild":"1007.8.0010703.9","ConfigStateHash":"1386539727","Entitlements":"15","GID":"0","ImageFileName":"/usr/bin/logger","MD5HashData":"79922c9c30f225086d8a7331f54427ae","ParentBaseFileName":"bash","ParentProcessId":"317927531071092650","ProcessEndTime":"1603206061.046","ProcessGroupId":"317927531071092650","ProcessStartTime":"1603206061.045","RGID":"0","RUID":"0","RawProcessId":"115761","SHA1HashData":"0000000000000000000000000000000000000000","SHA256HashData":"9ed6a92880974f41eef1dd11b80d883c26faf1951a4b6a12720ccce8d195042c","SVGID":"0","SVUID":"0","SessionProcessId":"317927531071092652","SourceProcessId":"317927531071092650","SourceThreadId":"0","TargetProcessId":"317927531071092653","UID":"0","aid":"3a6f13022fff41a7b016537ea3862935","aip":"204.194.122.230","cid":"dff76f718745469994bd39c6a38579aa","event_platform":"Lin","event_simpleName":"ProcessRollup2","id":"11869eb7-12e5-11eb-9385-02ed8d7af942","name":"ProcessRollup2LinV6","timestamp":"1603206061577"} {"CommandLine":"cat /var/spool/anacron/cron.daily","ConfigBuild":"1007.8.0010703.9","ConfigStateHash":"1386539727","Entitlements":"15","GID":"0","ImageFileName":"/usr/bin/cat","MD5HashData":"98b51c2a3efd088fe2ec632980c28e6c","ParentBaseFileName":"bash","ParentProcessId":"317927530540353195","ProcessEndTime":"1603206061.179","ProcessGroupId":"317927530540353191","ProcessStartTime":"1603206061.178","RGID":"0","RUID":"0","RawProcessId":"11630","SHA1HashData":"0000000000000000000000000000000000000000","SHA256HashData":"6825557ed76e1055d07a2cfd6ef753a931885daf4e102414724f9e812f48d538","SVGID":"0","SVUID":"0","SessionProcessId":"317927530540353193","SourceProcessId":"317927530540353195","SourceThreadId":"0","TargetProcessId":"317927532637505196","UID":"0","aid":"0a127135ef064baf8a4004991c43f2b7","aip":"204.194.122.230","cid":"dff76f718745469994bd39c6a38579aa","event_platform":"Lin","event_simpleName":"ProcessRollup2","id":"1199c591-12e5-11eb-b366-064134f5d9c0","name":"ProcessRollup2LinV6","timestamp":"1603206061703"} In each row there is a timestamp, for example in the first row it is "timestamp":"1603206061703" I'm doing an ExtractText manipulate the flowfile line by line with the goal of each line replaced with {"time": "THE_TIMESTAMP_VALUE/1000", "attribute a", "attribute_stuff_a", "event": $1} So I want to do a calculation based on a value found in the line-by-line and add a couple attributes. I've been able to do everything but add the calculated "time" value. I tried separating the ExtractText and adding all the attributes in one and then adding the timestamp using the following ExtractText
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)