Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

setting nifi.python.command=python3 results in Failed to instantiate [org.apache.nifi.controller.FlowController]: Factory method 'flowController' threw exception with message: null

avatar
Frequent Visitor

 

nifi.python.command=python3
nifi.flow.configuration.file=../data/flow.json.gz
nifi.flow.configuration.archive.enabled=false
nifi.flow.configuration.archive.dir=../data/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB
nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=../data/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components
nifi.nar.unpack.uber.jar=false
nifi.state.management.configuration.file=./conf/state-management.xml
nifi.state.management.provider.local=local-provider
nifi.state.management.provider.cluster=zk-provider
nifi.state.management.embedded.zookeeper.start=false
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
nifi.database.directory=./database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
nifi.repository.encryption.protocol.version=
nifi.repository.encryption.key.id=
nifi.repository.encryption.key.provider=
nifi.repository.encryption.key.provider.keystore.location=
nifi.repository.encryption.key.provider.keystore.password=
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=./flowfile_repository
nifi.flowfile.repository.checkpoint.interval=20 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=true
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=50 KB
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=1 days
nifi.content.repository.archive.max.usage.percentage=60%
nifi.content.repository.archive.enabled=false
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=1 day
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
nifi.provenance.repository.indexed.attributes=
nifi.provenance.repository.index.shard.size=500 MB
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
nifi.provenance.repository.buffer.size=100000
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min
nifi.status.repository.questdb.persist.node.days=14
nifi.status.repository.questdb.persist.component.days=3
nifi.status.repository.questdb.persist.location=./status_repository
nifi.remote.input.host=MASKED
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs
nifi.web.http.host=
nifi.web.http.network.interface.default=
nifi.web.https.host=MASKED
nifi.web.https.port=9443
nifi.web.https.network.interface.default=
nifi.web.https.application.protocols=http/1.1
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=MASKED
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.max.access.token.requests.per.second=25
nifi.web.request.timeout=60 secs
nifi.web.request.ip.whitelist=
nifi.web.should.send.server.version=true
nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"
nifi.web.https.ciphersuites.include=
nifi.web.https.ciphersuites.exclude=
nifi.sensitive.props.key=MASKED
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=
nifi.security.keystore=/opt/nifi/data/certs/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=notmyactualpassword
nifi.security.keyPasswd=notmyactualpassword
nifi.security.truststore=/opt/nifi/data/certs/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=notmyactualpassword
nifi.security.user.authorizer=managed-authorizer
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=
nifi.security.user.jws.key.rotation.period=PT1H
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=
nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.additional.scopes=
nifi.security.user.oidc.claim.identifying.user=
nifi.security.user.oidc.fallback.claims.identifying.user=
nifi.security.user.oidc.truststore.strategy=JDK
nifi.security.user.knox.url=
nifi.security.user.knox.publicKey=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.audiences=
nifi.security.user.saml.idp.metadata.url=https://fs.lgt.com/FederationMetadata/2007-06/FederationMetadata.xml
nifi.security.user.saml.sp.entity.id=masked
nifi.security.user.saml.identity.attribute.name=Name ID
nifi.security.user.saml.group.attribute.name=http://schemas.xmlsoap.org/claims/Group
nifi.security.user.saml.request.signing.enabled=false
nifi.security.user.saml.want.assertions.signed=true
nifi.security.user.saml.signature.algorithm=http://www.w3.org/2001/04/xmldsig-more#rsa-sha256
nifi.security.user.saml.authentication.expiration=12 hours
nifi.security.user.saml.single.logout.enabled=false
nifi.security.user.saml.http.client.truststore.strategy=JDK
nifi.security.user.saml.http.client.connect.timeout=30 secs
nifi.security.user.saml.http.client.read.timeout=30 secs
nifi.security.user.saml.message.logging.enabled=true
nifi.listener.bootstrap.port=0
nifi.cluster.protocol.heartbeat.interval=12 sec
nifi.cluster.protocol.heartbeat.missable.max=8
nifi.cluster.protocol.is.secure=true
nifi.cluster.is.node=true
nifi.cluster.node.address=MASKED
nifi.cluster.node.protocol.port=6007
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=30 sec
nifi.cluster.node.read.timeout=60 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=30 sec
nifi.cluster.flow.election.max.candidates=
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=1
nifi.cluster.load.balance.max.thread.count=8​
# Python 
nifi.python.command=python3
# Core Properties #
nifi.flow.configuration.file=../data/flow.json.gz
nifi.flow.configuration.archive.enabled=false
nifi.flow.configuration.archive.dir=../data/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB

nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=../data/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components
nifi.nar.unpack.uber.jar=false

####################
# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties

# H2 Settings
nifi.database.directory=./database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE

# Repository Encryption properties override individual repository implementation properties
nifi.repository.encryption.protocol.version=
nifi.repository.encryption.key.id=
nifi.repository.encryption.key.provider=
nifi.repository.encryption.key.provider.keystore.location=
nifi.repository.encryption.key.provider.keystore.password=

# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=./flowfile_repository
nifi.flowfile.repository.checkpoint.interval=20 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=true

nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=50 KB
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=1 days
nifi.content.repository.archive.max.usage.percentage=60%
nifi.content.repository.archive.enabled=false
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/

# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository

# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=1 day
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable.  Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2


# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000

# Component and Node Status History Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository

# Volatile Status History Repository Properties
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min

# QuestDB Status History Repository Properties
nifi.status.repository.questdb.persist.node.days=14
nifi.status.repository.questdb.persist.component.days=3
nifi.status.repository.questdb.persist.location=./status_repository

# Site to Site properties
nifi.remote.input.host=
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
#############################################

# For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.
# Be aware that changing these properties may affect how your instance can be accessed without any restriction.
# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.

nifi.web.http.host=
nifi.web.http.network.interface.default=
nifi.web.https.host=
nifi.web.https.port=9443
nifi.web.https.network.interface.default=
nifi.web.https.application.protocols=http/1.1
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.max.access.token.requests.per.second=25
nifi.web.request.timeout=60 secs
nifi.web.request.ip.whitelist=
nifi.web.should.send.server.version=true
nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"

# Include or Exclude TLS Cipher Suites for HTTPS
nifi.web.https.ciphersuites.include=
nifi.web.https.ciphersuites.exclude=

# security properties #
nifi.sensitive.props.key=11jba2NSNi50hDUN(
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=

nifi.security.keystore=/opt/nifi/data/certs/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=notmyactualpassword
nifi.security.keyPasswd=notmyactualpassword
nifi.security.truststore=/opt/nifi/data/certs/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=notmyactualpassword

nifi.security.user.authorizer=managed-authorizer
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=
nifi.security.user.jws.key.rotation.period=PT1H
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=

# OpenId Connect SSO Properties #
nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.additional.scopes=
nifi.security.user.oidc.claim.identifying.user=
nifi.security.user.oidc.fallback.claims.identifying.user=
nifi.security.user.oidc.truststore.strategy=JDK

# Apache Knox SSO Properties #
nifi.security.user.knox.url=
nifi.security.user.knox.publicKey=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.audiences=

# SAML Properties #
nifi.security.user.saml.idp.metadata.url=https://fs.lgt.com/FederationMetadata/2007-06/FederationMetadata.xml
nifi.security.user.saml.sp.entity.id=data:devplatform:nifi:saml:sp
nifi.security.user.saml.identity.attribute.name=Name ID
nifi.security.user.saml.group.attribute.name=http://schemas.xmlsoap.org/claims/Group
nifi.security.user.saml.request.signing.enabled=false
nifi.security.user.saml.want.assertions.signed=true
nifi.security.user.saml.signature.algorithm=http://www.w3.org/2001/04/xmldsig-more#rsa-sha256
nifi.security.user.saml.authentication.expiration=12 hours
nifi.security.user.saml.single.logout.enabled=false
nifi.security.user.saml.http.client.truststore.strategy=JDK
nifi.security.user.saml.http.client.connect.timeout=30 secs
nifi.security.user.saml.http.client.read.timeout=30 secs
nifi.security.user.saml.message.logging.enabled=true

# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1@$2
# nifi.security.identity.mapping.transform.dn=NONE
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.security.identity.mapping.value.kerb=$1@$2
# nifi.security.identity.mapping.transform.kerb=UPPER

# Group Mapping Properties #
# These properties allow normalizing group names coming from external sources like LDAP. The following example
# lowercases any group name.
#
# nifi.security.group.mapping.pattern.anygroup=^(.*)$
# nifi.security.group.mapping.value.anygroup=$1
# nifi.security.group.mapping.transform.anygroup=LOWER

# Listener Bootstrap properties #
# This property defines the port used to listen for communications from NiFi Bootstrap. If this property
# is missing, empty, or 0, a random ephemeral port is used.
nifi.listener.bootstrap.port=0

# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=12 sec
nifi.cluster.protocol.heartbeat.missable.max=8
nifi.cluster.protocol.is.secure=true

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=
nifi.cluster.node.protocol.port=6007
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=30 sec
nifi.cluster.node.read.timeout=60 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=30 sec
nifi.cluster.flow.election.max.candidates=

# cluster load balancing properties #
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=1
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=nifi-zookeeper-svc:2181
nifi.zookeeper.connect.timeout=10 secs
nifi.zookeeper.session.timeout=10 secs
nifi.zookeeper.root.node=/nifi
nifi.zookeeper.client.secure=false
nifi.zookeeper.security.keystore=
nifi.zookeeper.security.keystoreType=
nifi.zookeeper.security.keystorePasswd=
nifi.zookeeper.security.truststore=
nifi.zookeeper.security.truststoreType=
nifi.zookeeper.security.truststorePasswd=
nifi.zookeeper.jute.maxbuffer=

# Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management
# Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes
# and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node
# The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal
# and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal
# values configured on the zookeeper server).
nifi.zookeeper.auth.type=
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=

# kerberos #
nifi.kerberos.krb5.file=

# kerberos service principal #
nifi.kerberos.service.principal=
nifi.kerberos.service.keytab.location=

# kerberos spnego principal #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours

# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=

# analytics properties #
nifi.analytics.predict.enabled=false
nifi.analytics.predict.interval=3 mins
nifi.analytics.query.interval=5 mins
nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
nifi.analytics.connection.model.score.name=rSquared
nifi.analytics.connection.model.score.threshold=.90

# runtime monitoring properties
nifi.monitor.long.running.task.schedule=
nifi.monitor.long.running.task.threshold=

# Create automatic diagnostics when stopping/restarting NiFi.

# Enable automatic diagnostic at shutdown.
nifi.diagnostics.on.shutdown.enabled=false

# Include verbose diagnostic information.
nifi.diagnostics.on.shutdown.verbose=false

# The location of the diagnostics folder.
nifi.diagnostics.on.shutdown.directory=./diagnostics

# The maximum number of files permitted in the directory. If the limit is exceeded, the oldest files are deleted.
nifi.diagnostics.on.shutdown.max.filecount=10

# The diagnostics folder's maximum permitted size in bytes. If the limit is exceeded, the oldest files are deleted.
nifi.diagnostics.on.shutdown.max.directory.size=10 MB

# Performance tracking properties
## Specifies what percentage of the time we should track the amount of time processors are using CPU, reading from/writing to content repo, etc.
## This can be useful to understand which components are the most expensive and to understand where system bottlenecks may be occurring.
## The value must be in the range of 0 (inclusive) to 100 (inclusive). A larger value will produce more accurate results, while a smaller value may be
## less expensive to compute.
## Results can be obtained by running "nifi.sh diagnostics <filename>" and then inspecting the produced file.
nifi.performance.tracking.percentage=0

# NAR Provider Properties #
# These properties allow configuring one or more NAR providers. A NAR provider retrieves NARs from an external source
# and copies them to the directory specified by nifi.nar.library.autoload.directory.
#
# Each NAR provider property follows the format:
#  nifi.nar.library.provider.<identifier>.<property-name>
#
# Each NAR provider must have at least one property named "implementation".
#
# Example HDFS NAR Provider:
#   nifi.nar.library.provider.hdfs.implementation=org.apache.nifi.flow.resource.hadoop.HDFSExternalResourceProvider
#   nifi.nar.library.provider.hdfs.resources=/path/to/core-site.xml,/path/to/hdfs-site.xml
#   nifi.nar.library.provider.hdfs.storage.location=hdfs://hdfs-location
#   nifi.nar.library.provider.hdfs.source.directory=/nars
#   [email protected]
#   nifi.nar.library.provider.hdfs.kerberos.keytab=/path/to/nifi.keytab
#   nifi.nar.library.provider.hdfs.kerberos.password=
#
# Example NiFi Registry NAR Provider:
#   nifi.nar.library.provider.nifi-registry.implementation=org.apache.nifi.registry.extension.NiFiRegistryNarProvider
#   nifi.nar.library.provider.nifi-registry.url=http://localhost:18080

 

I am running nifi 2.0.0-M4 on OpenShift. My two node cluster comes up fine but as soon as I try to enable the python-api by setting nifi.python.command=python3 it keeps crashing. Here the relevant log section:

 

mkdir: created directory '/opt/nifi/nifi-current/work'
mkdir: created directory '/opt/nifi/nifi-current/work/python'
updating nifi.cluster.node.address in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.remote.input.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.https.host in /opt/nifi/nifi-current/conf/nifi.properties
updating nifi.web.proxy.host in /opt/nifi/nifi-current/conf/nifi.properties

Java home: /usr/lib/jvm/jre
NiFi home: /opt/nifi/nifi-current

Bootstrap Config File: /opt/nifi/nifi-current/conf/bootstrap.conf

Login Identity Providers Processed [/opt/nifi/nifi-current/./conf/login-identity-providers.xml]

Certificate was added to keystore
Certificate was added to keystore
NiFi running with PID 118.

Java home: /usr/lib/jvm/jre
NiFi home: /opt/nifi/nifi-current

Bootstrap Config File: /opt/nifi/nifi-current/conf/bootstrap.conf

07:28:09,319 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.5.6
07:28:09,320 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2aa5fe93 - No custom configurators were discovered as a service.
07:28:09,320 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2aa5fe93 - Trying to configure with ch.qos.logback.classic.joran.SerializedModelConfigurator
07:28:09,321 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2aa5fe93 - Constructed configurator of type class ch.qos.logback.classic.joran.SerializedModelConfigurator
07:28:09,326 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.scmo]
07:28:09,326 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.scmo]
07:28:09,334 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2aa5fe93 - ch.qos.logback.classic.joran.SerializedModelConfigurator.configure() call lasted 6 milliseconds. ExecutionStatus=INVOKE_NEXT_IF_ANY
07:28:09,334 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2aa5fe93 - Trying to configure with ch.qos.logback.classic.util.DefaultJoranConfigurator
07:28:09,334 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2aa5fe93 - Constructed configurator of type class ch.qos.logback.classic.util.DefaultJoranConfigurator
07:28:09,335 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
07:28:09,339 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/opt/nifi/nifi-current/conf/logback.xml]
07:28:09,458 |-INFO in ch.qos.logback.classic.model.processor.ConfigurationModelHandlerFull - Registering a new ReconfigureOnChangeTask ReconfigureOnChangeTask(born:1722324489456)
07:28:09,459 |-INFO in ch.qos.logback.classic.model.processor.ConfigurationModelHandlerFull - Will scan for changes in [file:/opt/nifi/nifi-current/conf/logback.xml] 
07:28:09,459 |-INFO in ch.qos.logback.classic.model.processor.ConfigurationModelHandlerFull - Setting ReconfigureOnChangeTask scanning period to 30 seconds
07:28:09,469 |-WARN in ch.qos.logback.core.model.processor.ShutdownHookModelHandler - ch.qos.logback.core.hook.DelayingShutdownHook was renamed as ch.qos.logback.core.hook.DefaultShutdownHook
07:28:09,469 |-WARN in ch.qos.logback.core.model.processor.ShutdownHookModelHandler - Please use the new class name
07:28:09,469 |-INFO in ch.qos.logback.core.model.processor.ShutdownHookModelHandler - About to instantiate shutdown hook of type [ch.qos.logback.core.hook.DefaultShutdownHook]
07:28:09,469 |-INFO in ch.qos.logback.core.model.processor.ShutdownHookModelHandler - Registering shutdown hook with JVM runtime.
07:28:09,470 |-INFO in ch.qos.logback.classic.model.processor.LoggerContextListenerModelHandler - Adding LoggerContextListener of type [ch.qos.logback.classic.jul.LevelChangePropagator] to the object stack
07:28:09,482 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating DEBUG level on Logger[ROOT] onto the JUL framework
07:28:09,482 |-INFO in ch.qos.logback.classic.model.processor.LoggerContextListenerModelHandler - Starting LoggerContextListener
07:28:09,484 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [APP_FILE]
07:28:09,484 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
07:28:09,496 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1524126153 - setting totalSizeCap to 500 MB
07:28:09,499 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1524126153 - Archive files will be limited to [100 MB] each.
07:28:09,514 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1524126153 - No compression will be used
07:28:09,515 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1524126153 - Will use the pattern /opt/nifi/nifi-current/logs/nifi-app_%d.%i.log for the active file
07:28:09,544 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@6156496 - The date pattern is 'yyyy-MM-dd' from file name pattern '/opt/nifi/nifi-current/logs/nifi-app_%d.%i.log'.
07:28:09,544 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@6156496 - Roll-over at midnight.
07:28:09,550 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@6156496 - Setting initial period to 2024-07-30T07:27:55.654Z
07:28:09,552 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1524126153 - Cleaning on start up
07:28:09,553 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - first clean up after appender initialization
07:28:09,553 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - Multiple periods, i.e. 32 periods, seem to have elapsed. This is expected at application start.
07:28:09,561 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - No removal attempts were made.
07:28:09,568 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[APP_FILE] - Active log file name: /opt/nifi/nifi-current/logs/nifi-app.log
07:28:09,569 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[APP_FILE] - File property is set to [/opt/nifi/nifi-current/logs/nifi-app.log]
07:28:09,569 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [USER_FILE]
07:28:09,569 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
07:28:09,570 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@63001505 - setting totalSizeCap to 200 MB
07:28:09,570 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@63001505 - Archive files will be limited to [100 MB] each.
07:28:09,570 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@63001505 - No compression will be used
07:28:09,570 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@63001505 - Will use the pattern /opt/nifi/nifi-current/logs/nifi-user_%d.%i.log for the active file
07:28:09,571 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@b62fe6d - The date pattern is 'yyyy-MM-dd' from file name pattern '/opt/nifi/nifi-current/logs/nifi-user_%d.%i.log'.
07:28:09,571 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@b62fe6d - Roll-over at midnight.
07:28:09,571 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@b62fe6d - Setting initial period to 2024-07-30T07:27:54.768Z
07:28:09,572 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@63001505 - Cleaning on start up
07:28:09,572 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - first clean up after appender initialization
07:28:09,572 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - Multiple periods, i.e. 32 periods, seem to have elapsed. This is expected at application start.
07:28:09,573 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[USER_FILE] - Active log file name: /opt/nifi/nifi-current/logs/nifi-user.log
07:28:09,573 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[USER_FILE] - File property is set to [/opt/nifi/nifi-current/logs/nifi-user.log]
07:28:09,573 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [REQUEST_FILE]
07:28:09,573 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
07:28:09,573 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@330084561 - setting totalSizeCap to 200 MB
07:28:09,573 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@330084561 - Archive files will be limited to [100 MB] each.
07:28:09,574 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@330084561 - No compression will be used
07:28:09,574 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@330084561 - Will use the pattern /opt/nifi/nifi-current/logs/nifi-request_%d.%i.log for the active file
07:28:09,574 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@3e3047e6 - The date pattern is 'yyyy-MM-dd' from file name pattern '/opt/nifi/nifi-current/logs/nifi-request_%d.%i.log'.
07:28:09,574 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@3e3047e6 - Roll-over at midnight.
07:28:09,574 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@3e3047e6 - Setting initial period to 2024-07-30T07:02:54.353Z
07:28:09,575 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@330084561 - Cleaning on start up
07:28:09,575 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - first clean up after appender initialization
07:28:09,575 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - Multiple periods, i.e. 32 periods, seem to have elapsed. This is expected at application start.
07:28:09,575 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[REQUEST_FILE] - Active log file name: /opt/nifi/nifi-current/logs/nifi-request.log
07:28:09,575 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[REQUEST_FILE] - File property is set to [/opt/nifi/nifi-current/logs/nifi-request.log]
07:28:09,575 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [BOOTSTRAP_FILE]
07:28:09,575 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
07:28:09,576 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@937773018 - setting totalSizeCap to 100 MB
07:28:09,576 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@937773018 - Archive files will be limited to [50 MB] each.
07:28:09,576 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@937773018 - No compression will be used
07:28:09,576 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@937773018 - Will use the pattern /opt/nifi/nifi-current/logs/nifi-bootstrap_%d.%i.log for the active file
07:28:09,577 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@2b6856dd - The date pattern is 'yyyy-MM-dd' from file name pattern '/opt/nifi/nifi-current/logs/nifi-bootstrap_%d.%i.log'.
07:28:09,577 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@2b6856dd - Roll-over at midnight.
07:28:09,577 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@2b6856dd - Setting initial period to 2024-07-30T07:27:55.654Z
07:28:09,577 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@937773018 - Cleaning on start up
07:28:09,577 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - first clean up after appender initialization
07:28:09,578 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - Multiple periods, i.e. 32 periods, seem to have elapsed. This is expected at application start.
07:28:09,578 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - No removal attempts were made.
07:28:09,578 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[BOOTSTRAP_FILE] - Active log file name: /opt/nifi/nifi-current/logs/nifi-bootstrap.log
07:28:09,578 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[BOOTSTRAP_FILE] - File property is set to [/opt/nifi/nifi-current/logs/nifi-bootstrap.log]
07:28:09,578 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [CONSOLE]
07:28:09,578 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
07:28:09,580 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - No removal attempts were made.
07:28:09,581 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - No removal attempts were made.
07:28:09,582 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi] to INFO
07:28:09,582 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi] onto the JUL framework
07:28:09,582 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.processors] to WARN
07:28:09,582 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[org.apache.nifi.processors] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.processors.standard.LogAttribute] to WARN
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[org.apache.nifi.processors.standard.LogAttribute] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.processors.standard.LogMessage] to WARN
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[org.apache.nifi.processors.standard.LogMessage] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.controller.repository.StandardProcessSession] to WARN
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[org.apache.nifi.controller.repository.StandardProcessSession] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.zookeeper.ClientCnxn] to ERROR
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.apache.zookeeper.ClientCnxn] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.zookeeper.server.NIOServerCnxn] to ERROR
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.apache.zookeeper.server.NIOServerCnxn] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.zookeeper.server.NIOServerCnxnFactory] to ERROR
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.apache.zookeeper.server.NIOServerCnxnFactory] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.zookeeper.server.NettyServerCnxnFactory] to ERROR
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.apache.zookeeper.server.NettyServerCnxnFactory] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.zookeeper.server.quorum] to ERROR
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.apache.zookeeper.server.quorum] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.zookeeper.ZooKeeper] to ERROR
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.apache.zookeeper.ZooKeeper] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.zookeeper.server.PrepRequestProcessor] to ERROR
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.apache.zookeeper.server.PrepRequestProcessor] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.controller.reporting.LogComponentStatuses] to ERROR
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.apache.nifi.controller.reporting.LogComponentStatuses] onto the JUL framework
07:28:09,583 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.calcite.runtime.CalciteException] to OFF
07:28:09,583 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating OFF level on Logger[org.apache.calcite.runtime.CalciteException] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.curator.framework.recipes.leader.LeaderSelector] to OFF
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating OFF level on Logger[org.apache.curator.framework.recipes.leader.LeaderSelector] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.curator.ConnectionState] to OFF
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating OFF level on Logger[org.apache.curator.ConnectionState] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.cluster] to INFO
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.cluster] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.server.JettyServer] to INFO
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.server.JettyServer] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.eclipse.jetty] to INFO
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.eclipse.jetty] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.springframework] to ERROR
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.springframework] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.springframework.security] to INFO
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.springframework.security] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.glassfish.jersey.internal.Errors] to ERROR
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.glassfish.jersey.internal.Errors] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.eclipse.jetty.annotations.AnnotationParser] to ERROR
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.eclipse.jetty.annotations.AnnotationParser] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [net.schmizz.sshj] to WARN
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[net.schmizz.sshj] onto the JUL framework
07:28:09,584 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [com.hierynomus.sshj] to WARN
07:28:09,584 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[com.hierynomus.sshj] onto the JUL framework
07:28:09,585 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [com.hierynomus.smbj] to WARN
07:28:09,585 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[com.hierynomus.smbj] onto the JUL framework
07:28:09,585 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [com.amazonaws.services.kinesis] to WARN
07:28:09,585 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[com.amazonaws.services.kinesis] onto the JUL framework
07:28:09,585 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.atlas] to WARN
07:28:09,585 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[org.apache.atlas] onto the JUL framework
07:28:09,585 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.web.security.requests] to INFO
07:28:09,585 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.web.security.requests] onto the JUL framework
07:28:09,588 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.web.security.requests] to false
07:28:09,589 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [APP_FILE] to Logger[org.apache.nifi.web.security.requests]
07:28:09,589 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [CONSOLE] to Logger[org.apache.nifi.web.security.requests]
07:28:09,589 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.web.security] to INFO
07:28:09,589 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.web.security] onto the JUL framework
07:28:09,589 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.web.security] to false
07:28:09,589 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [USER_FILE] to Logger[org.apache.nifi.web.security]
07:28:09,589 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.web.api.config] to INFO
07:28:09,589 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.web.api.config] onto the JUL framework
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.web.api.config] to false
07:28:09,590 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [USER_FILE] to Logger[org.apache.nifi.web.api.config]
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.authorization] to INFO
07:28:09,590 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.authorization] onto the JUL framework
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.authorization] to false
07:28:09,590 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [USER_FILE] to Logger[org.apache.nifi.authorization]
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.cluster.authorization] to INFO
07:28:09,590 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.cluster.authorization] onto the JUL framework
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.cluster.authorization] to false
07:28:09,590 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [USER_FILE] to Logger[org.apache.nifi.cluster.authorization]
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.web.api.AccessResource] to INFO
07:28:09,590 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.web.api.AccessResource] onto the JUL framework
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.web.api.AccessResource] to false
07:28:09,590 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [USER_FILE] to Logger[org.apache.nifi.web.api.AccessResource]
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.opensaml] to WARN
07:28:09,590 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[org.opensaml] onto the JUL framework
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.opensaml] to false
07:28:09,590 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [USER_FILE] to Logger[org.opensaml]
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.web.server.RequestLog] to INFO
07:28:09,590 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.web.server.RequestLog] onto the JUL framework
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.web.server.RequestLog] to false
07:28:09,590 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [REQUEST_FILE] to Logger[org.apache.nifi.web.server.RequestLog]
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.bootstrap] to INFO
07:28:09,590 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.bootstrap] onto the JUL framework
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.bootstrap] to true
07:28:09,590 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [BOOTSTRAP_FILE] to Logger[org.apache.nifi.bootstrap]
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.bootstrap.Command] to INFO
07:28:09,590 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating INFO level on Logger[org.apache.nifi.bootstrap.Command] onto the JUL framework
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.bootstrap.Command] to true
07:28:09,590 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [BOOTSTRAP_FILE] to Logger[org.apache.nifi.bootstrap.Command]
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.StdOut] to WARN
07:28:09,590 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[org.apache.nifi.StdOut] onto the JUL framework
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.StdOut] to true
07:28:09,590 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [BOOTSTRAP_FILE] to Logger[org.apache.nifi.StdOut]
07:28:09,590 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.nifi.StdErr] to ERROR
07:28:09,591 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating ERROR level on Logger[org.apache.nifi.StdErr] onto the JUL framework
07:28:09,591 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [org.apache.nifi.StdErr] to true
07:28:09,591 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [BOOTSTRAP_FILE] to Logger[org.apache.nifi.StdErr]
07:28:09,591 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to WARN
07:28:09,591 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5c1a8622 - Propagating WARN level on Logger[ROOT] onto the JUL framework
07:28:09,591 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [CONSOLE] to Logger[ROOT]
07:28:09,591 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [APP_FILE] to Logger[ROOT]
07:28:09,591 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@5db45159 - End of configuration.
07:28:09,592 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@6107227e - Registering current configuration as safe fallback point
07:28:09,592 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2aa5fe93 - ch.qos.logback.classic.util.DefaultJoranConfigurator.configure() call lasted 258 milliseconds. ExecutionStatus=DO_NOT_INVOKE_NEXT_IF_ANY

2024-07-30 07:28:09,613 [main] WARN  org.apache.nifi.bootstrap.process.RuntimeValidatorExecutor - -Runtime Configuration [AvailableLocalPorts] validation failed: Local Ports [28231] less than recommended [55000] according to [/proc/sys/net/ipv4/ip_local_port_range]
2024-07-30 07:28:09,615 [main] WARN  org.apache.nifi.bootstrap.process.RuntimeValidatorExecutor - -Runtime Configuration [Swappiness] validation failed: Swappiness [30] more than recommended [0] according to [/proc/sys/vm/swappiness]
2024-07-30 07:28:09,616 [main] WARN  org.apache.nifi.bootstrap.process.RuntimeValidatorExecutor - -Runtime Configuration [SocketTimedWaitDuration] validation failed: TCP Socket Wait [120 seconds] more than recommended [1 seconds] according to [/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_time_wait]
2024-07-30 07:28:09,721 [main] INFO  org.apache.nifi.bootstrap.RunNiFi - -{javaVersion=21.0.4, availableProcessors=4, maxOpenFileDescriptors=1048576, totalPhysicalMemoryMB=20480}
2024-07-30 07:28:09,749 [main] INFO  org.apache.nifi.bootstrap.Command - -Starting Apache NiFi...
2024-07-30 07:28:09,750 [main] INFO  org.apache.nifi.bootstrap.Command - -Working Directory: /opt/nifi/nifi-current
2024-07-30 07:28:09,750 [main] INFO  org.apache.nifi.bootstrap.Command - -Command: /usr/lib/jvm/jre/bin/java -classpath /opt/nifi/nifi-current/./conf:/opt/nifi/nifi-current/./lib/jcl-over-slf4j-2.0.13.jar:/opt/nifi/nifi-current/./lib/jul-to-slf4j-2.0.13.jar:/opt/nifi/nifi-current/./lib/log4j-over-slf4j-2.0.13.jar:/opt/nifi/nifi-current/./lib/logback-classic-1.5.6.jar:/opt/nifi/nifi-current/./lib/logback-core-1.5.6.jar:/opt/nifi/nifi-current/./lib/nifi-api-2.0.0-M4.jar:/opt/nifi/nifi-current/./lib/nifi-framework-api-2.0.0-M4.jar:/opt/nifi/nifi-current/./lib/nifi-nar-utils-2.0.0-M4.jar:/opt/nifi/nifi-current/./lib/nifi-per-process-group-logging-2.0.0-M4.jar:/opt/nifi/nifi-current/./lib/nifi-properties-2.0.0-M4.jar:/opt/nifi/nifi-current/./lib/nifi-property-utils-2.0.0-M4.jar:/opt/nifi/nifi-current/./lib/nifi-python-framework-api-2.0.0-M4.jar:/opt/nifi/nifi-current/./lib/nifi-runtime-2.0.0-M4.jar:/opt/nifi/nifi-current/./lib/nifi-server-api-2.0.0-M4.jar:/opt/nifi/nifi-current/./lib/nifi-stateless-api-2.0.0-M4.jar:/opt/nifi/nifi-current/./lib/slf4j-api-2.0.13.jar:/opt/nifi/nifi-current/./lib/ojdbc8-18.0.0.0.jar -Dorg.apache.jasper.compiler.disablejsr199=true -XX:MaxRAMPercentage=70 -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -XX:+UseG1GC -Dnifi.properties.file.path=/opt/nifi/nifi-current/./conf/nifi.properties -Dnifi.bootstrap.listen.port=41379 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/opt/nifi/nifi-current/logs org.apache.nifi.NiFi 
2024-07-30 07:28:09,762 [main] INFO  org.apache.nifi.bootstrap.Command - -Application Process [157] launched
2024-07-30 07:28:10,173 [NiFi Bootstrap Command Listener] INFO  org.apache.nifi.bootstrap.RunNiFi - -Apache NiFi now running and listening for Bootstrap requests on port 38023
2024-07-30 07:28:20,769 [NiFi logging handler] ERROR org.apache.nifi.StdErr - -WARNING: A restricted method in java.lang.foreign.Linker has been called
2024-07-30 07:28:20,769 [NiFi logging handler] ERROR org.apache.nifi.StdErr - -WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
2024-07-30 07:28:20,770 [NiFi logging handler] ERROR org.apache.nifi.StdErr - -WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
2024-07-30 07:28:20,770 [NiFi logging handler] ERROR org.apache.nifi.StdErr - -
2024-07-30 07:28:21,536 [NiFi logging handler] ERROR org.apache.nifi.StdErr - -Failed to start web server: Error creating bean with name 'frameworkClusterConfiguration': Unsatisfied dependency expressed through method 'setFlowController' parameter 0: Error creating bean with name 'flowController' defined in class path resource [org/apache/nifi/framework/configuration/FlowControllerConfiguration.class]: Failed to instantiate [org.apache.nifi.controller.FlowController]: Factory method 'flowController' threw exception with message: null
2024-07-30 07:28:21,536 [NiFi logging handler] ERROR org.apache.nifi.StdErr - -Shutting down...
2024-07-30 07:28:22,766 [main] INFO  org.apache.nifi.bootstrap.RunNiFi - -NiFi never started. Will not restart NiFi

 

I checked that my java and my python as required:

 

sh-4.4$ python3 --version
Python 3.9.19
sh-4.4$ python3 -m venv
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT]
            [--upgrade-deps]
            ENV_DIR [ENV_DIR ...]
venv: error: the following arguments are required: ENV_DIR
sh-4.4$ java --version
openjdk 21.0.4 2024-07-16 LTS
OpenJDK Runtime Environment (Red_Hat-21.0.4.0.7-1) (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-21.0.4.0.7-1) (build 21.0.4+7-LTS, mixed mode, sharing)

 

What I think is weird that there are no python logs in $NIFI_HOME/logs. Also $NIFI_HOME/work/python is empty as well.  Here is my nifi.properties file:

 

 

1 ACCEPTED SOLUTION

avatar
Frequent Visitor

I asked in the slack channel of apache nifi. So basically it's all in the stacktrace: org.apache.nifi.controller.FlowController.createPythonBridge(FlowController.java:865. The line on the source code: https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M4/nifi-framework-bundle/nifi-framework/nifi-fram....  Here as a snippet at your convenience: 

final File pythonWorkingDirectory = new File(nifiProperties.getProperty(NiFiProperties.PYTHON_WORKING_DIRECTORY));

so because we have a bit of a custom setup we did get all the default python settings as outlined here: https://nifi.apache.org/documentation/nifi-2.0.0-M1/html/administration-guide.html#python_configurat...

After setting all of those: 

 nifi.python.command=python3
 nifi.python.framework.source.directory=./python/framework
 nifi.python.extensions.source.directory.default=./python/extensions
 nifi.python.working.directory=./work/python 
 nifi.python.logs.directory=./logs 
 nifi.python.max.processes.per.extension.type=10 
 nifi.python.max.processes=100

it then started again and I could import the processor

View solution in original post

1 REPLY 1

avatar
Frequent Visitor

I asked in the slack channel of apache nifi. So basically it's all in the stacktrace: org.apache.nifi.controller.FlowController.createPythonBridge(FlowController.java:865. The line on the source code: https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M4/nifi-framework-bundle/nifi-framework/nifi-fram....  Here as a snippet at your convenience: 

final File pythonWorkingDirectory = new File(nifiProperties.getProperty(NiFiProperties.PYTHON_WORKING_DIRECTORY));

so because we have a bit of a custom setup we did get all the default python settings as outlined here: https://nifi.apache.org/documentation/nifi-2.0.0-M1/html/administration-guide.html#python_configurat...

After setting all of those: 

 nifi.python.command=python3
 nifi.python.framework.source.directory=./python/framework
 nifi.python.extensions.source.directory.default=./python/extensions
 nifi.python.working.directory=./work/python 
 nifi.python.logs.directory=./logs 
 nifi.python.max.processes.per.extension.type=10 
 nifi.python.max.processes=100

it then started again and I could import the processor