Skip to main content
dulanga
dulanga
Silver ⭐️⭐️
August 20, 2023
Question

NIFI won't start throws java.util.concurrent.TimeoutException: null

I have install nifi-1.16.1 and used it for several ETL perviously. But suddly i'm experiencing it's restarting some times due to ot of memory issues.

but when i manually start the NIFI it logs below exceptions for all controller services


2023-08-19 21:11:43,480 INFO [main] o.a.n.c.s.StandardProcessScheduler Enabling StandardControllerServiceNode[service=DBCPConnectionPool[id=1654d4a2-0189-1000-7bbe-076de7850b24], name=DBCPConnectionPool, active=false]
2023-08-19 21:12:01,993 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@4d7f3adf checkpointed with 317 Records and 0 Swap Files in 19 milliseconds (Stop-the-world time = 13 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 19248
2023-08-19 21:12:03,810 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive
2023-08-19 21:12:13,481 ERROR [main] o.a.n.c.s.StandardControllerServiceProvider Failed to enable StandardControllerServiceNode[service=DBCPConnectionPool[id=1654d4a2-0189-1000-7bbe-076de7850b24], name=DBCPConnectionPool, active=true]
java.util.concurrent.TimeoutException: null
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
at org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:209)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer$FlowControllerComponentScheduler.enableNow(VersionedFlowSynchronizer.java:1022)
at org.apache.nifi.groups.AbstractComponentScheduler.resume(AbstractComponentScheduler.java:67)
at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:402)
at org.apache.nifi.groups.StandardProcessGroupSynchronizer.lambda$synchronize$0(StandardProcessGroupSynchronizer.java:221)
at org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:462)
at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:219)
at org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3832)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:358)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:186)
at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1524)
at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:815)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:538)
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008)
at org.apache.nifi.NiFi.<init>(NiFi.java:170)
at org.apache.nifi.NiFi.<init>(NiFi.java:82)
at org.apache.nifi.NiFi.main(NiFi.java:330)
2023-08-19 21:12:13,483 INFO [main] o.a.n.c.s.StandardProcessScheduler Enabling StandardControllerServiceNode[service=AvroReader[id=1654d4a3-0189-1000-7c27-2e120a44cc03], name=AvroReaderForInsert, active=false]
2023-08-19 21:12:43,495 ERROR [main] o.a.n.c.s.StandardControllerServiceProvider Failed to enable StandardControllerServiceNode[service=AvroReader[id=1654d4a3-0189-1000-7c27-2e120a44cc03], name=AvroReaderForInsert, active=true]
java.util.concurrent.TimeoutException: null
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
at org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:209)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer$FlowControllerComponentScheduler.enableNow(VersionedFlowSynchronizer.java:1022)
at org.apache.nifi.groups.AbstractComponentScheduler.resume(AbstractComponentScheduler.java:67)
at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:402)
at org.apache.nifi.groups.StandardProcessGroupSynchronizer.lambda$synchronize$0(StandardProcessGroupSynchronizer.java:221)
at org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:462)
at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:219)
at org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3832)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:358)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:186)
at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1524)
at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:815)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:538)
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008)
at org.apache.nifi.NiFi.<init>(NiFi.java:170)
at org.apache.nifi.NiFi.<init>(NiFi.java:82)
at org.apache.nifi.NiFi.main(NiFi.java:330)
2023-08-19 21:12:43,511 INFO [main] o.a.n.c.s.StandardProcessScheduler Enabling StandardControllerServiceNode[service=AvroReader[id=12e9105e-1463-1624-1a93-469b94c336e9], name=AvroReaderForInsert, active=false]
2023-08-19 21:13:04,242 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive

after takes long time while throwing above exception it starts
as
please advise on this matter

3 replies

cotopaul
cotopaul
Platinum ⭐️
August 21, 2023

@dulanga, please tell us your NiFi Version, your hardware configurations (OS, RAM and CPU), how many flows you already have on your canvas, your OS limits (open files, running processors, etc), your java version and how you configured bootstrap.conf (mostly the jvm settings and in case you are using an older java version, the args 7-8-9).

Translate
    dulanga
    dulangaAuthor
    Silver ⭐️⭐️
    August 26, 2023

    @cotopaul Please finds details below

    Java Version

    openjdk version "11.0.20" 2023-07-18
    OpenJDK Runtime Environment (build 11.0.20+8-post-Ubuntu-1ubuntu120.04)
    OpenJDK 64-Bit Server VM (build 11.0.20+8-post-Ubuntu-1ubuntu120.04, mixed mode, sharing)


    NIFI Version - 1.16.1
    OS -

    Distributor ID:	Ubuntu
    Description:	Ubuntu 20.04.5 LTS
    Release:	20.04
    Codename:	focal


    Memory

                  total        used        free      shared  buff/cache   available
    Mem:          3.8Gi       1.5Gi       2.1Gi       145Mi       269Mi       2.1Gi
    Swap:         511Mi       511Mi          0B

    CPU

    Architecture:                    x86_64
    CPU op-mode(s):                  32-bit, 64-bit
    Byte Order:                      Little Endian
    Address sizes:                   48 bits physical, 48 bits virtual
    CPU(s):                          2
    On-line CPU(s) list:             0,1
    Thread(s) per core:              1
    Core(s) per socket:              2
    Socket(s):                       1
    NUMA node(s):                    1
    Vendor ID:                       AuthenticAMD
    CPU family:                      23
    Model:                           1
    Model name:                      AMD EPYC 7601 32-Core Processor
    Stepping:                        2
    CPU MHz:                         2199.998
    BogoMIPS:                        4399.99
    Hypervisor vendor:               KVM
    Virtualization type:             full
    L1d cache:                       128 KiB
    L1i cache:                       128 KiB
    L2 cache:                        1 MiB
    L3 cache:                        16 MiB


    -------------------------------------------------------------------------------------------------------------------

    Open file limits - 1024
    ------------------------------------------------------------------------------------------------------
    Maximum number of processes - 15373
    ------------------------------------------------------------------------------------------------------
    bootstrap.conf

    # Java command to use when running NiFi
    java=java

    # Username to use when running NiFi. This value will be ignored on Windows.
    run.as=

    # Preserve shell environment while runnning as "run.as" user
    preserve.environment=false

    # Configure where NiFi's lib and conf directories live
    lib.dir=./lib
    conf.dir=./conf

    # How long to wait after telling NiFi to shutdown before explicitly killing the Process
    graceful.shutdown.seconds=20

    # Disable JSR 199 so that we can use JSP's without running a JDK
    java.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=true

    # JVM memory settings
    java.arg.2=-Xms4096m
    java.arg.3=-Xmx8192m

    # Enable Remote Debugging
    #java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

    java.arg.4=-Djava.net.preferIPv4Stack=true

    # allowRestrictedHeaders is required for Cluster/Node communications to work properly
    java.arg.5=-Dsun.net.http.allowRestrictedHeaders=true
    java.arg.6=-Djava.protocol.handler.pkgs=sun.net.www.protocol

    # The G1GC is known to cause some problems in Java 8 and earlier, but the issues were addressed in Java 9. If using Java 8 or earlier,
    # it is recommended that G1GC not be used, especially in conjunction with the Write Ahead Provenance Repository. However, if using a newer
    # version of Java, it can result in better performance without significant "stop-the-world" delays.
    #java.arg.13=-XX:+UseG1GC

    #Set headless mode by default
    java.arg.14=-Djava.awt.headless=true

    # Root key in hexadecimal format for encrypted sensitive configuration values
    nifi.bootstrap.sensitive.key=

    # Sensitive Property Provider configuration

    # HashiCorp Vault Sensitive Property Providers
    #nifi.bootstrap.protection.hashicorp.vault.conf=./conf/bootstrap-hashicorp-vault.conf

    # AWS Sensitive Property Providers
    #nifi.bootstrap.protection.aws.conf=./conf/bootstrap-aws.conf

    # Azure Key Vault Sensitive Property Providers
    #nifi.bootstrap.protection.azure.keyvault.conf=./conf/bootstrap-azure.conf

    # GCP KMS Sensitive Property Providers
    #nifi.bootstrap.protection.gcp.kms.conf=./conf/bootstrap-gcp.conf

    # Sets the provider of SecureRandom to /dev/urandom to prevent blocking on VMs
    java.arg.15=-Djava.security.egd=file:/dev/urandom

    # Requires JAAS to use only the provided JAAS configuration to authenticate a Subject, without using any "fallback" methods (such as prompting for username/password)
    # Please see https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/single-signon.html, section "EXCEPTIONS TO THE MODEL"
    java.arg.16=-Djavax.security.auth.useSubjectCredsOnly=true

    # Zookeeper 3.5 now includes an Admin Server that starts on port 8080, since NiFi is already using that port disable by default.
    # Please see https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_adminserver_config for configuration options.
    java.arg.17=-Dzookeeper.admin.enableServer=false

    # The following options configure a Java Agent to handle native library loading.
    # It is needed when a custom jar (eg. JDBC driver) has been configured on a component in the flow and this custom jar depends on a native library
    # and tries to load it by its absolute path (java.lang.System.load(String filename) method call).
    # Use this Java Agent only if you get "Native Library ... already loaded in another classloader" errors otherwise!
    #java.arg.18=-javaagent:./lib/aspectj/aspectjweaver-1.9.6.jar
    #java.arg.19=-Daj.weaving.loadersToSkip=sun.misc.Launcher$AppClassLoader,jdk.internal.loader.ClassLoaders$AppClassLoader,org.eclipse.jetty.webapp.WebAppClassLoader,\
    #            org.apache.jasper.servlet.JasperLoader,org.jvnet.hk2.internal.DelegatingClassLoader,org.apache.nifi.nar.NarClassLoader
    # End of Java Agent config for native library loading.

    ###
    # Notification Services for notifying interested parties when NiFi is stopped, started, dies
    ###

    # XML File that contains the definitions of the notification services
    notification.services.file=./conf/bootstrap-notification-services.xml

    # In the case that we are unable to send a notification for an event, how many times should we retry?
    notification.max.attempts=5

    # Comma-separated list of identifiers that are present in the notification.services.file; which services should be used to notify when NiFi is started?
    #nifi.start.notification.services=email-notification

    # Comma-separated list of identifiers that are present in the notification.services.file; which services should be used to notify when NiFi is stopped?
    #nifi.stop.notification.services=email-notification

    # Comma-separated list of identifiers that are present in the notification.services.file; which services should be used to notify when NiFi dies?
    #nifi.dead.notification.services=email-notification

    # The first curator connection issue is logged as ERROR, for example when NiFi cannot connect to one of the Zookeeper nodes.
    # Additional connection issues are logged as DEBUG until the connection is restored.
    java.arg.curator.supress.excessive.logs=-Dcurator-log-only-first-connection-issue-as-error-level=true

    # Port used to listen for communications from NiFi. If this property is missing, empty, or 0, a random ephemeral port is used.
    nifi.bootstrap.listen.port=0







    Translate
    cotopaul
    cotopaul
    Platinum ⭐️
    August 28, 2023

    @dulanga, as far as I can tell from your previous post, you have around 3GB of RAM Memory available on your NiFi node, but you are assigning much more to your JVM.


    So, you have:

    total used free shared buff/cache available
    Mem: 3.8Gi 1.5Gi 2.1Gi 145Mi 269Mi 2.1Gi
    Swap: 511Mi 511Mi 0B

    But you are assigning much more to your JVM:

    # JVM memory settings
    java.arg.2=-Xms4096m
    java.arg.3=-Xmx8192m

     
    Try correcting your config files and assign the correct value for your JVM, in the bootstrap.conf file.
    Here are some best practices: https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999

     

    Translate
    Terms & ConditionsAccessibility statement