Ambari shows NIFI nodes running but unable to access the cluster UI

[#1]

Ambari shows NIFI nodes running but unable to access the cluster UI.

I have a 4 node nifi cluster, and noticed an error while access the Cluster ui (http://localhost/nifi).

when i check the nifi services in ambari, all services shown as "Running".  I then tried to restarted the nifi service one nodes at a time, and all services were restarted properly in ambari.  However, when I try to go back to the cluster ui again, i am no longer access the ui, not even the Nifi error page.  I need some help on troubleshoot this issue.

Thanks,

[#2]

@dtan 

1. What happens when you hit the URL using command line tools like wget/curl?
Please run the following kind of command from the host where you are opening the browser.

# curl -iLv http://$NIFI_HOST:9090/nifi/

Can you please share the output of the above commands? (this is to isolate the browser issue like any browser lever network proxy setting issue)

2. Also please try to run the same CURL command from any of the cluster node to see if it returns proper HTML page with Nifi UI html?

3. What happens when you try to check the NiFI Host / Port access from the machine where you are trying to open the NiFI UI in browser?  (this is to isolate the Firewall/Iptables issue) . 

# telnet  $NIFI_HOST  9090
(OR)
# nc -v  $NIFI_HOST  9090

4. On the NiFi host can you please confirm if the NiFi process is actually running and the port is listening?

# ps -ef | grep org.apache.nifi.NiFi
# netstat -tnlpa | grep 9090
# cat /var/run/nifi/nifi.status

5. Do we have firewall/iptables disabled on NiFI host? (just to make sure that from outside of that host we can access nifi ports.

6. Can you please share the screenshot / output of the NiFi UI  in the Browser Debugger console ... so that we can see if shows any error in browser debugger console?

[#3]

Please see response below.

Thanks again for your help.

1) the curl command failed from all sources, including one of the cluster node:

[root@hostnamm]# curl -iLv https://localhost:9090/nifi/
* About to connect() to localhost port 9090 (#0)
* Trying 127.0.0.1...
* Connection refused
* Failed connect to localhost:9090; Connection refused
* Closing connection 0
curl: (7) Failed connect to localhost:9090; Connection refused

3) Port 9090 doesn't seem to be running. That's the strange part. But Ambari indicated NIFI is running.

confirmed that FW is disabled:

[root@hostname]# systemctl status firewalld.service
â firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)

Jul 07 20:43:23 MS01-NIFI01 systemd[1]: Stopped firewalld - dynamic firewall daemon.
Jul 07 20:46:28 MS01-NIFI01 systemd[1]: Stopped firewalld - dynamic firewall daemon.

4) 

]# ps -ef | grep org.apache.nifi.NiFi
nifi 15495 15474 55 Aug26 ? 1-14:26:12 /usr/jdk64/jdk1.8.0_77/bin/java -classpath /usr/hdf/2.0.0.0-579/nifi/conf:/usr/hdf/2.0.0.0-579/nifi/lib/bcprov-jdk15on-1.54.jar:/usr/hdf/2.0.0.0-579/nifi/lib/commons-lang3-3.4.jar:/usr/hdf/2.0.0.0-579/nifi/lib/jcl-over-slf4j-1.7.12.jar:/usr/hdf/2.0.0.0-579/nifi/lib/jul-to-slf4j-1.7.12.jar:/usr/hdf/2.0.0.0-579/nifi/lib/log4j-over-slf4j-1.7.12.jar:/usr/hdf/2.0.0.0-579/nifi/lib/logback-classic-1.1.3.jar:/usr/hdf/2.0.0.0-579/nifi/lib/logback-core-1.1.3.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-properties-loader-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-api-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-runtime-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-documentation-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-framework-api-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-nar-utils-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-properties-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/slf4j-api-1.7.12.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx48g -Xms48g -Dambari.application.id=nifi -Dambari.metrics.collector.url=http://ms01-nifi01.hedgeservmgmt.com:6188/ws/v1/timeline/metrics -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:+UseG1GC -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/usr/hdf/2.0.0.0-579/nifi/conf/nifi.properties -Dnifi.bootstrap.listen.port=37158 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi org.apache.nifi.NiFi
root 62041 5945 0 12:56 pts/0 00:00:00 grep --color=auto org.apache.nifi.NiFi

ps -ef | grep org.apache.nifi.NiFi
nifi 15495 15474 55 Aug26 ? 1-14:26:12 /usr/jdk64/jdk1.8.0_77/bin/java -classpath /usr/hdf/2.0.0.0-579/nifi/conf:/usr/hdf/2.0.0.0-579/nifi/lib/bcprov-jdk15on-1.54.jar:/usr/hdf/2.0.0.0-579/nifi/lib/commons-lang3-3.4.jar:/usr/hdf/2.0.0.0-579/nifi/lib/jcl-over-slf4j-1.7.12.jar:/usr/hdf/2.0.0.0-579/nifi/lib/jul-to-slf4j-1.7.12.jar:/usr/hdf/2.0.0.0-579/nifi/lib/log4j-over-slf4j-1.7.12.jar:/usr/hdf/2.0.0.0-579/nifi/lib/logback-classic-1.1.3.jar:/usr/hdf/2.0.0.0-579/nifi/lib/logback-core-1.1.3.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-properties-loader-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-api-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-runtime-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-documentation-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-framework-api-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-nar-utils-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-properties-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/slf4j-api-1.7.12.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx48g -Xms48g -Dambari.application.id=nifi -Dambari.metrics.collector.url=http://ms01-nifi01.hedgeservmgmt.com:6188/ws/v1/timeline/metrics -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:+UseG1GC -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/usr/hdf/2.0.0.0-579/nifi/conf/nifi.properties -Dnifi.bootstrap.listen.port=37158 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi org.apache.nifi.NiFi
root 62041 5945 0 12:56 pts/0 00:00:00 grep --color=auto org.apache.nifi.NiFi
[root@MS01-NIFI01 nifi]# netstat -tnlpa | grep 9090
tcp 0 0 127.0.1.1:9090 0.0.0.0:* LISTEN 15495/java

cat /var/run/nifi/nifi.status
#Mon Aug 26 16:14:06 EDT 2019
port=44028
pid=15495
secret.key=<secret_key>

[#4]

Can anyone out there please help?  I have some business critical application using NiFi and its been down for a few days now.  Any help would be greatly appreciated.

[#5]

Hi,

Where is NIFI running? Is it on your own machine or somewhere else?

By default and if using HDF, if NIFI is not secured it will run on port 9090 but if you are secured it will run on port 9091.

Do you have a value on your machine in file /etc/hostname? on one of your nifi nodes?

I assume you are deploying HDF?

[#6]

NiFi is running in centos servers in our datacenter (on-prem).  One thing I like to add is that, it's been working for a long time and just stop working last week.  I restarted NiFi services in ambari and all the services look good (status is green), but I am unable to access the URL. I have don't a few diagnostic based on the earlier feedback.  One thing that's really puzzling is that I can't event telnet into the jetty server locally via port 9090, and I confirmed that the web server is setup to run on port 9090 in the nifi.properties file.  I reviewed the bootstrap log but not seeing any error related to the web server, but there are many errors.  I am thinking this is related to the flow file being out of sync in the cluster nodes but not sure how to validate the hypothesis??

[#7]

hmm to me it sounds like you have a DNS issue.

I would go to the logs directory of one of the NIFI hosts:

If using HDF it should be:

/usr/hdf/current/nifi/logs

and then do a grep like this:

grep "org.apache.nifi.web.server.JettyServer" *

One of the very last lines should have something that reads like so:

nifi-app_2019-07-25_13.0.log:2019-07-25 13:30:40,076 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:

nifi-app_2019-07-25_13.0.log:2019-07-25 13:30:40,076 INFO [main] org.apache.nifi.web.server.JettyServer http://SOME-URL:SOME-PORT/nifi

Once you have that then I would try to ping that FQDN or IP and if you cant then you either have something going on with DNS or need to add that FQDN and IP to your local machines /etc/host file, assuming you are trying to reach the UI on a *NIX machine

[#8]

This is not a DNS issue.  I am able to ping the short and fqdn name as well as able to resolve the address via nslookup.  I don't see anything in the bootstrap log indicates the jetty server was started, and telnet to 9090 from the server itself failed.  I am suspecting the jetty web server not able to start, but not sure how to fix it.  Also, ambari is running on the same host and its reachable a fully functional.

here's a snippet of the bootstrap log, show many errors but not sure what they are.

[user@MS01-NIFI01 logs]# cat nifi-bootstrap.log |more
2019-08-26 14:40:10,618 INFO [main] org.apache.nifi.bootstrap.RunNiFi A shutdown was initiated. Will not restart NiFi
2019-08-26 14:40:15,622 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: []
2019-08-26 14:40:15,625 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED
2019-08-26 14:40:15,625 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED
2019-08-26 14:40:15,625 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED
2019-08-26 14:40:15,631 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi...
2019-08-26 14:40:15,631 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /usr/hdf/2.0.0.0-579/nifi
2019-08-26 14:40:15,631 INFO [main] org.apache.nifi.bootstrap.Command Command: /usr/jdk64/jdk1.8.0_77/bin/java -classpath /usr/hdf/2.0.0.0-579/n
ifi/conf:/usr/hdf/2.0.0.0-579/nifi/lib/bcprov-jdk15on-1.54.jar:/usr/hdf/2.0.0.0-579/nifi/lib/commons-lang3-3.4.jar:/usr/hdf/2.0.0.0-579/nifi/lib
/jcl-over-slf4j-1.7.12.jar:/usr/hdf/2.0.0.0-579/nifi/lib/jul-to-slf4j-1.7.12.jar:/usr/hdf/2.0.0.0-579/nifi/lib/log4j-over-slf4j-1.7.12.jar:/usr/
hdf/2.0.0.0-579/nifi/lib/logback-classic-1.1.3.jar:/usr/hdf/2.0.0.0-579/nifi/lib/logback-core-1.1.3.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-prope
rties-loader-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-api-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-runtime-1.0.0
.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-documentation-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-framework-api-1.0.0.2
.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-nar-utils-1.0.0.2.0.0.0-579.jar:/usr/hdf/2.0.0.0-579/nifi/lib/nifi-properties-1.0.0.2.0.0.0-57
9.jar:/usr/hdf/2.0.0.0-579/nifi/lib/slf4j-api-1.7.12.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx48g -Xms48g -Dambari.application.id
=nifi -Dambari.metrics.collector.url=http://hostname:6188/ws/v1/timeline/metrics -Dsun.net.http.allowRestrictedHeaders=true
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:+UseG1GC -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.
path=/usr/hdf/2.0.0.0-579/nifi/conf/nifi.properties -Dnifi.bootstrap.listen.port=41630 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/va
r/log/nifi org.apache.nifi.NiFi
2019-08-26 14:40:16,291 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Boots
trap requests on port 39210
2019-08-26 14:41:06,392 ERROR [NiFi logging handler] org.apache.nifi.StdErr [Error] :161:54: cvc-pattern-valid: Value '1 hour' is not facet-vali
d with respect to pattern '\d+\s*(ns|nano|nanos|nanoseconds|ms|milli|millis|milliseconds|s|sec|secs|seconds|m|min|mins|minutes|h|hr|hrs|hours|d|
day|days)' for type 'TimePeriod'.
2019-08-26 14:41:06,392 ERROR [NiFi logging handler] org.apache.nifi.StdErr [Error] :161:54: cvc-type.3.1.3: The value '1 hour' of element 'pena
lizationPeriod' is not valid.
2019-08-26 14:41:06,407 ERROR [NiFi logging handler] org.apache.nifi.StdErr [Error] :1618:54: cvc-pattern-valid: Value '1 hour' is not facet-val
id with respect to pattern '\d+\s*(ns|nano|nanos|nanoseconds|ms|milli|millis|milliseconds|s|sec|secs|seconds|m|min|mins|minutes|h|hr|hrs|hours|d
|day|days)' for type 'TimePeriod'.
2019-08-26 14:41:06,407 ERROR [NiFi logging handler] org.apache.nifi.StdErr [Error] :1618:54: cvc-type.3.1.3: The value '1 hour' of element 'pen
alizationPeriod' is not valid.
2019-08-26 14:41:06,414 ERROR [NiFi logging handler] org.apache.nifi.StdErr [Error] :2404:54: cvc-pattern-valid: Value '1 hour' is not facet-val
id with respect to pattern '\d+\s*(ns|nano|nanos|nanoseconds|ms|milli|millis|milliseconds|s|sec|secs|seconds|m|min|mins|minutes|h|hr|hrs|hours|d
|day|days)' for type 'TimePeriod'.
2019-08-26 14:41:06,414 ERROR [NiFi logging handler] org.apache.nifi.StdErr [Error] :2404:54: cvc-type.3.1.3: The value '1 hour' of element 'pen
alizationPeriod' is not valid.
2019-08-26 14:41:06,420 ERROR [NiFi logging handler] org.apache.nifi.StdErr [Error] :2976:54: cvc-pattern-valid: Value '1 hour' is not facet-val
id with respect to pattern '\d+\s*(ns|nano|nanos|nanoseconds|ms|milli|millis|milliseconds|s|sec|secs|seconds|m|min|mins|minutes|h|hr|hrs|hours|d
--More--
 

[#9]

Hi,

What does the nifi-app.log say?

bootstrap will show JVM related messages but for this case inspect your nifi.app.log,