Nifi -taking so much time(misleading) to redirect to localhost:8080

[#1]

Hi,

I am running Nifi from CentOS , which I configured to run my Cloudera. To start Nifi, I change the Java version, exporting manually and start Nifi. I open the browser and could not load the Nifi GUI( http://localhost:8080/nifi/). I start tail -f for the log

"tail -f /home/cloudera/raja/nifi-1.5.0/logs/nifi-app.log" . After a long time , it redirects to http://localhost:8080/nifi/. So if I have to transfer Kafka topics from CentOS Cloudera to Nifi, HDP, it may be misleading for me as the latency is so high. Please suggest.

ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer 2018-05-29 06:57:50,071 INFO [Timer-Driven Process Thread-1] o.a.kafka.common.utils.AppInfoParser Kafka version : 0.10.2.1 2018-05-29 06:57:50,071 INFO [Timer-Driven Process Thread-1] o.a.kafka.common.utils.AppInfoParser Kafka commitId : e89bffd6b2eff799 2018-05-29 06:58:01,912 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1d306d27{/nifi-api,file:///home/cloudera/raja/nifi-1.5.0/work/jetty/nifi-web-api-1.5.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.5.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.5.0.war} 2018-05-29 06:58:04,866 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1800ms 2018-05-29 06:58:04,881 INFO [main] o.e.j.C./nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath 2018-05-29 06:58:05,001 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@5317a7ef{/nifi-content-viewer,file:///home/cloudera/raja/nifi-1.5.0/work/jetty/nifi-web-content-viewer-1.5.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.5.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.5.0.war} 2018-05-29 06:58:05,039 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.s.h.ContextHandler@53c39950{/nifi-docs,null,AVAILABLE} 2018-05-29 06:58:05,200 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=87ms 2018-05-29 06:58:05,331 INFO [main] o.e.jetty.ContextHandler./nifi-docs No Spring WebApplicationInitializer types detected on classpath 2018-05-29 06:58:05,364 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@3d056418{/nifi-docs,file:///home/cloudera/raja/nifi-1.5.0/work/jetty/nifi-web-docs-1.5.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.5.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.5.0.war} 2018-05-29 06:58:05,547 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=103ms 2018-05-29 06:58:05,623 INFO [main] org.eclipse.jetty.ContextHandler./ No Spring WebApplicationInitializer types detected on classpath 2018-05-29 06:58:05,650 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@35ab0bf3{/,file:///home/cloudera/raja/nifi-1.5.0/work/jetty/nifi-web-error-1.5.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.5.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.5.0.war} 2018-05-29 06:58:05,795 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@239ce32{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} 2018-05-29 06:58:05,795 INFO [main] org.eclipse.jetty.server.Server Started @467767ms 2018-05-29 06:58:05,832 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started.

The UI is available at the following URLs: 2018-05-29 06:58:05,832 INFO [main] org.apache.nifi.web.server.JettyServer http://192.168.142.134:8080/nifi 2018-05-29 06:58:05,832 INFO [main] org.apache.nifi.web.server.JettyServer http://127.0.0.1:8080/nifi 2018-05-29 06:58:05,855

INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap 2018-05-29 06:58:05,855 INFO [main] org.apache.nifi.NiFi Controller initialization took 116886957819 nanoseconds (116 seconds). 2018-05-29 06:58:36,124 WARN [NiFi Web Server-31] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [192.168.142.134:8080] different from web hostname [(:8080)]. Overriding to [:8080/nifi] 2018-05-29 06:58:36,126 WARN [NiFi Web Server-31] o.a.nifi.web.server.HostHeaderHandler Request host header [192.168.142.134:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/nifi] 2018-05-29 06:58:36,300 WARN [NiFi Web Server-28] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [192.168.142.134:8080] different from web hostname [(:8080)]. Overriding to [:8080/favicon.ico] 2018-05-29 06:58:36,301 WARN [NiFi Web Server-28] o.a.nifi.web.server.HostHeaderHandler Request host header [192.168.142.134:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/favicon.ico] 2018-05-29 06:58:36,308 WARN [NiFi Web Server-30] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [192.168.142.134:8080] different from web hostname [(:8080)]. Overriding to [:8080/favicon.ico] 2018-05-29 06:58:36,310 WARN [NiFi Web Server-30] o.a.nifi.web.server.HostHeaderHandler Request host header [192.168.142.134:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/favicon.ico] 2018-05-29 06:58:52,020 WARN [NiFi Web Server-25] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [127.0.0.1:8080] different from web hostname [(:8080)]. Overriding to [:8080/nifi] 2018-05-29 06:58:52,023 WARN [NiFi Web Server-25] o.a.nifi.web.server.HostHeaderHandler Request host header [127.0.0.1:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/nifi] 2018-05-29 06:58:52,153 WARN [NiFi Web Server-31] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [127.0.0.1:8080] different from web hostname [(:8080)]. Overriding to [:8080/favicon.ico] 2018-05-29 06:58:52,153 WARN [NiFi Web Server-31] o.a.nifi.web.server.HostHeaderHandler Request host header [127.0.0.1:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/favicon.ico] 2018-05-29 06:58:52,179 WARN [NiFi Web Server-26] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [127.0.0.1:8080] different from web hostname [(:8080)]. Overriding to [:8080/favicon.ico] 2018-05-29 06:58:52,180 WARN [NiFi Web Server-26] o.a.nifi.web.server.HostHeaderHandler Request host header [127.0.0.1:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/favicon.ico] 2018-05-29 06:58:58,281 WARN [NiFi Web Server-30] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [127.0.0.1:8080] different from web hostname [(:8080)]. Overriding to [:8080/nifi] 2018-05-29 06:58:58,281 WARN [NiFi Web Server-30] o.a.nifi.web.server.HostHeaderHandler Request host header [127.0.0.1:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/nifi] 2018-05-29 06:59:43,195 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@17d45cfb checkpointed with 7 Records and 0 Swap Files in 41 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 5 millis), max Transaction ID 20

[#2 · Accepted solution]

@Raja K

There are two Java process associated to a running NiFi instance.

The first JVM process is tied to the NiFi bootstrap. It is what you are interacting with when you issue the "start, stop, restart, dump, etc..." During the startup phase, the bootstrap will start the other NiFi JVM process.

-

During startup of the main NiFi process, there is a lot that goes on. For example:

- NiFi must unpack all the nars in the NiFi lib directory to the Nifi work directory and load relevant code in to memory.

- NiFi must join the cluster

----- If entire cluster has just been started/restarted, NiFi goes in to an election phase (Cluster needs to elect a cluster coordinator, primary node, a cluster flow). The election will not occur until all nodes have joined (based on nifi.cluster.flow.election.max.candidates=) or until election wait time expires (nifi.cluster.flow.election.max.wait.time=5 mins). If max.candidates is not set, you will always have a min 5 min latency here.

- Nodes joining the cluster must then compare their local flow with the cluster flow to make sure they match exactly.

- Once the node successfully passes these steps it must start loading flowfiles back in to the flow (these would be flowfiles still queued in connection when NiFi was last stopped/restarted).

- Processor components are then scheduled to run based on the cluster flow processor state (start all processors that are in a running state. Start primary node only processors on the elected primary node)

- Only then is the cluster as a whole in a state where user should be given access to the UI for interactive purposes. This does not mean that the flows were not already running before theUI was actually available to the user.

-

There are are things that happening there, but those are the biggest pieces.

Tailing the nifi-app/log watching for the "The UI is available at the following URLs:" is the easiest way to see when the UI will be available for the users.

Those UIs are based on the following properties from the nifi.properties file:

nifi.web.http.host=     <-- unsecured nifi

nifi.web.https.host=    <-- secured nifi

If they are left blank, NiFi will bind to all available interfaces that the JVM finds on the host machine (as your UIs shows). There is no redirection going on here.

-

Hope this helps explain what is occurring during the startup.

The size of your dataflow(s), election process, number of queued FlowFiles being loaded back in to JVM memory (queued connections),etc. will have some impact on startup time.

Your output above only has the timestamp for when the UI became available (2018-05-29 06:58:05,832), so it is not clear how much latency you are really seeing between the actual start command that timestamp. First entry in nifi-app.log should include (org.apache.nifi.NiFi Launching NiFi...).

-

Thank you,

Matt