NIFI UI is not working

[#1]

Hi Team,

We Could see our NIFI is not working for the last 2 days, Even we restarted NIFI services multiple times but nothing worked out as it is SSL enabled. Could you please help us to fix this.

@mattw 

[#2]

@mattw  Please find the error trace found in nifi-app.log

2020-08-18 07:00:34,997 ERROR [Timer-Driven Process Thread-2] o.a.n.p.standard.GenerateTableFetch GenerateTableFetch[id=bd177184-822e-32b9-983f-b76ae3bb613d] Failed to retrieve observed maximum values from the State Manager. Will not perform query until this is accomplished.: java.io.IOException: Failed to obtain value from ZooKeeper for component with ID bd177184-822e-32b9-983f-b76ae3bb613d with exception code CONNECTIONLOSS
java.io.IOException: Failed to obtain value from ZooKeeper for component with ID bd177184-822e-32b9-983f-b76ae3bb613d with exception code CONNECTIONLOSS
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:420)
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.getState(StandardStateManagerProvider.java:278)
at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63)
at org.apache.nifi.controller.lifecycle.TaskTerminationAwareStateManager.getState(TaskTerminationAwareStateManager.java:52)
at org.apache.nifi.processors.standard.GenerateTableFetch.onTrigger(GenerateTableFetch.java:247)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/components/bd177184-822e-32b9-983f-b76ae3bb613d
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403)
... 14 common frames omitted
2020-08-18 07:00:34,965 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Status of w0lxqhdp04.:9091 changed from NodeConnectionStatus[nodeId=w0lxqhdp04.:9091, state=CONNECTED, updateId=10] to NodeConnectionStatus[nodeId=w0lxqhdp04.:9091, state=DISCONNECTED, Disconnect Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from node in 70 seconds, updateId=11]
2020-08-18 07:00:58,612 WARN [Timer-Driven Process Thread-10] org.apache.hadoop.hdfs.DataStreamer Slow waitForAckedSeqno took 70145ms (threshold=30000ms). File being written: /data/operations/asop/psr_mongo_collab/psr_mongo_collab_2020_08_17, block: BP-1869526027-10.49.70.13-1580132125118:blk_1076124283_64574827, Write pipeline datanodes: [DatanodeInfoWithStorage[10.49.70.16:50010,DS-8d8421c2-b322-48e1-9922-5bdd4614667b,DISK], DatanodeInfoWithStorage[10.49.70.19:50010,DS-7791c66a-538f-4bb9-88e2-95f46ef522c2,DISK], DatanodeInfoWithStorage[10.49.70.13:50010,DS-35a98ef4-1d66-47f9-b29b-594b2a2fb308,DISK]].
2020-08-18 07:00:58,612 WARN [Timer-Driven Process Thread-9] org.apache.hadoop.hdfs.DataStreamer Slow waitForAckedSeqno took 47248ms (threshold=30000ms). File being written: /data/operations/asop/terminatn_status/terminatn_status_2020_08_17, block: BP-1869526027-10.49.70.13-1580132125118:blk_1076124281_64574829, Write pipeline datanodes: [DatanodeInfoWithStorage[10.49.70.16:50010,DS-8d8421c2-b322-48e1-9922-5bdd4614667b,DISK], DatanodeInfoWithStorage[10.49.70.17:50010,DS-d37465a6-e98d-495a-aa2e-d1bac42906d2,DISK], DatanodeInfoWithStorage[10.49.70.18:50010,DS-f28e02f8-bf18-421f-9c24-391f7a2ad881,DISK]].
2020-08-18 07:00:58,617 WARN [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for w0lxqhdp04.:9091 -- Node disconnected from cluster due to Have not received a heartbeat from node in 70 seconds

Bootstrap Log:

Bootstrap Config File: /usr/hdf/current/nifi/conf/bootstrap.conf

2020-08-18 06:44:16,813 ERROR [main] org.apache.nifi.bootstrap.Command Failed to send shutdown command to port 38531 due to java.net.SocketTimeoutException: Read timed out. Will kill the NiFi Process with PID 28486.

[#3]

@mattw 

Attaching the NIFI properties file for your reference.

cat nifi.properties
# Generated by Apache Ambari. Tue Aug 18 06:44:23 2020

nifi.administrative.yield.duration=30 sec
nifi.authorizer.configuration.file=/usr/hdf/current/nifi/conf/authorizers.xml
nifi.bored.yield.duration=10 millis
nifi.cluster.flow.election.max.candidates=1
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.is.node=true
nifi.cluster.node.address=w0lxqhdp04.
nifi.cluster.node.connection.timeout=30 sec
nifi.cluster.node.event.history.size=25
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.node.protocol.max.threads=
nifi.cluster.node.protocol.port=9088
nifi.cluster.node.protocol.threads=50
nifi.cluster.node.read.timeout=30 sec
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=True
nifi.components.status.repository.buffer.size=1440
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.snapshot.frequency=1 min
nifi.content.claim.max.appendable.size=1 MB
nifi.content.claim.max.flow.files=20
nifi.content.repository.always.sync=false
nifi.content.repository.archive.enabled=false
nifi.content.repository.archive.max.retention.period=6 hours
nifi.content.repository.archive.max.usage.percentage=25%
nifi.content.repository.directory.default=/data/content_repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.viewer.url=../nifi-content-viewer/
nifi.database.directory=/var/lib/nifi/database_repository
nifi.documentation.working.directory=/var/lib/nifi/work/docs/components
nifi.flow.configuration.archive.dir=/var/lib/nifi/archive/
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.max.count=
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.file=/var/lib/nifi/conf/flow.xml.gz
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.directory=/var/lib/nifi/flowfile_repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.partitions=256
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowservice.writedelay.interval=500 ms
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
nifi.kerberos.krb5.file=
nifi.kerberos.service.keytab.location=
nifi.kerberos.service.principal=
nifi.kerberos.spnego.authentication.expiration=12 hours
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.principal=
nifi.login.identity.provider.configuration.file=/usr/hdf/current/nifi/conf/login-identity-providers.xml
nifi.nar.library.directory=/usr/hdf/current/nifi/lib
nifi.nar.working.directory=/var/lib/nifi/work/nar
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.buffer.size=100000
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.concurrent.merge.threads=2
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.directory.default=/var/lib/nifi/provenance_repository
nifi.provenance.repository.encryption.key=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.index.shard.size=500 MB
nifi.provenance.repository.index.threads=1
nifi.provenance.repository.indexed.attributes=
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
nifi.provenance.repository.journal.count=16
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.warm.cache.frequency=1 hour
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB
nifi.queue.swap.threshold=20000
nifi.remote.contents.cache.expiration=30 secs
nifi.remote.input.host=
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.input.secure=True
nifi.remote.input.socket.port=
nifi.security.group.mapping.pattern.anygroup=
nifi.security.group.mapping.transform.anygroup=
nifi.security.group.mapping.value.anygroup=
nifi.security.identity.mapping.pattern.dn=
nifi.security.identity.mapping.pattern.kerb=
nifi.security.identity.mapping.transform.dn=
nifi.security.identity.mapping.transform.kerb=
nifi.security.identity.mapping.value.dn=
nifi.security.identity.mapping.value.kerb=
nifi.security.keyPasswd=lzvbGvxTu2jqYoiW||kF3EEKAgEgICkGdNcwI1ea7OwJ7M9zzIzYhXTGlUyBPt
nifi.security.keyPasswd.protected=aes/gcm/256
nifi.security.keystore=/usr/hdf/current/nifi/conf/keystore.jks
nifi.security.keystorePasswd=o6guFiFvjHZkd1sv||/PW2IpW+3uHDLfdYEYlGjJ5AwKfsY/JYlglLMUvzm9LB
nifi.security.keystorePasswd.protected=aes/gcm/256
nifi.security.keystoreType=jks
nifi.security.needClientAuth=False
nifi.security.ocsp.responder.certificate=
nifi.security.ocsp.responder.url=
nifi.security.truststore=/usr/hdf/current/nifi/conf/truststore.jks
nifi.security.truststorePasswd=GPdsLL2nuVPDna/8||D8l1yCf8Y2u0iijKajs+ujpJ0k8b6+9BYe9fwwXG3Fke
nifi.security.truststorePasswd.protected=aes/gcm/256
nifi.security.truststoreType=jks
nifi.security.user.authorizer=ranger-provider
nifi.security.user.knox.audiences=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.publicKey=
nifi.security.user.knox.url=
nifi.security.user.login.identity.provider=
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.read.timeout=5 secs
nifi.sensitive.props.additional.keys=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.key=+uS9mR1+fHmc7lmU||ZeyYH+ZsLF7S9y1MjrOz4D4zwcZQs3SeFsfz8kv2aUyPBxPz
nifi.sensitive.props.key.protected=aes/gcm/256
nifi.sensitive.props.provider=BC
nifi.state.management.configuration.file=/usr/hdf/current/nifi/conf/state-management.xml
nifi.state.management.embedded.zookeeper.properties=/usr/hdf/current/nifi/conf/zookeeper.properties
nifi.state.management.embedded.zookeeper.start=false
nifi.state.management.provider.cluster=zk-provider
nifi.state.management.provider.local=local-provider
nifi.swap.in.period=5 sec
nifi.swap.in.threads=1
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.swap.out.period=5 sec
nifi.swap.out.threads=4
nifi.templates.directory=/var/lib/nifi/templates
nifi.ui.autorefresh.interval=30 sec
nifi.ui.banner.text=
nifi.variable.registry.properties=
nifi.version=1.7.0.3.2.0.0-520
nifi.web.http.host=
nifi.web.http.network.interface.default=
nifi.web.http.port=
nifi.web.https.host=w0lxqhdp04.
nifi.web.https.network.interface.default=
nifi.web.https.port=9091
nifi.web.jetty.threads=200
nifi.web.jetty.working.directory=/var/lib/nifi/work/jetty
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=
nifi.web.war.directory=/usr/hdf/current/nifi/lib
nifi.zookeeper.connect.string=w0lxqhdp03.:2181,w0lxqhdp01.:2181,w0lxqhdp02.:2181
nifi.zookeeper.connect.timeout=60 secs
nifi.zookeeper.root.node=/nifi
nifi.zookeeper.session.timeout=60 secs

#nifi.security.ambari.hash.kspwd=60748b13f122c465a5b6f373cd82751fb803044db848ebb1d801281e3b48c154
#nifi.security.ambari.hash.kpwd=60748b13f122c465a5b6f373cd82751fb803044db848ebb1d801281e3b48c154
#nifi.security.ambari.hash.tspwd=60748b13f122c465a5b6f373cd82751fb803044db848ebb1d801281e3b48c154

[#4]

@saivenkatg55 Based on the error I seen the issue is with Zookeeper not NiFI:

zookeeper CONNECTIONLOSS

You can find more info in this post:  

https://community.cloudera.com/t5/Support-Questions/NiFi-Clustering-Issue-ConnectionLoss-Error/td-p/...

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

Thanks,

Steven @ DFHZ

[#5]

@stevenmatison @mattw 

do i need to enable this property  nifi.state.management.embedded.zookeeper.start=false

After enabling, seems NIFI is not able to run it's kept on failing.

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.ArrayIndexOutOfBoundsException: 1
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:55)
... 33 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.ArrayIndexOutOfBoundsException: 1
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
... 39 common frames omitted
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:188)
at org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:55)
at org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:189)
at org.apache.nifi.controller.FlowController.<init>(FlowController.java:620)
at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:452)
at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
... 46 common frames omitted
2020-08-18 07:32:30,284 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2020-08-18 07:32:30,291 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@5ec10ab1{HTTP/1.1,[http/1.1]}{w0lxqhdp04.:9090}
2020-08-18 07:32:30,292 INFO [Thread-1] org.eclipse.jetty.server.session Stopped scavenging

[#6]

@MattWho  Could you please help us to fix this issue