NiFi Kubernetes cluster - ProtocolException: Failed marshalling 'HEARTBEAT' protocol message

[#1]

Hi everybody!

I am struggling trying to create a NiFi Kubernetes cluster out of cetic helm chart.

After installation, I see a working zookeeper cluster (3 instances) and  a 2 instances stateful set for NiFi. One instance is correctly elected as cluster coordinator, NiFi frontend shows me a 1 member cluster. The second instance's log keeps on popping messages such as:

2022-07-01 11:52:23,839 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:29,313 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:34,792 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message

every five seconds.

I'm attaching the second pod's log and that pod's generated nifi.properties file.

Please help me 😞

nifi.properties:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Core Properties #
nifi.flow.configuration.file=../data/flow.xml.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=../data/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis

nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=../data/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.directory.custom=
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components

####################
# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties

# H2 Settings
nifi.database.directory=../data/database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE

# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.directory=../flowfile_repository
nifi.flowfile.repository.partitions=256
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.always.sync=false

nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000
nifi.swap.in.period=5 sec
nifi.swap.in.threads=1
nifi.swap.out.period=5 sec
nifi.swap.out.threads=4

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=../content_repository
nifi.content.repository.archive.max.retention.period=3 days
nifi.content.repository.archive.max.usage.percentage=85%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=/nifi-content-viewer/

# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key=

# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=../provenance_repository
nifi.provenance.repository.max.storage.time=10 days
nifi.provenance.repository.max.storage.size=8 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536

# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000

# Component Status Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min

# Site to Site properties
nifi.remote.input.host=dashboard-nifi-1405-1.dashboard-nifi-1405-headless.default.svc.cluster.local
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10000
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
nifi.web.war.directory=./lib
nifi.web.proxy.host=dashboard-nifi-1405.default.svc
nifi.web.https.port=8443
nifi.web.http.host=dashboard-nifi-1405-1.dashboard-nifi-1405-headless.default.svc.cluster.local
nifi.web.http.network.interface.default=eth0
nifi.web.https.host=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
# nifi.web.proxy.context.path=

# security properties #
nifi.sensitive.props.key=changeMechangeMe
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=

nifi.security.keystore=./conf/keystore.p12
nifi.security.keystoreType=PKCS12
nifi.security.keystorePasswd=1ff548a931ff221a92f866e5107c0f73
nifi.security.keyPasswd=1ff548a931ff221a92f866e5107c0f73
nifi.security.truststore=./conf/truststore.p12
nifi.security.truststoreType=PKCS12
nifi.security.truststorePasswd=1ca7ca6eb083ef4ecb8bf0983e2f43c9
nifi.security.user.login.identity.provider=single-user-provider
nifi.security.user.authorizer=single-user-authorizer

nifi.security.needClientAuth=

# Apache Knox SSO Properties #
nifi.security.user.knox.url=
nifi.security.user.knox.publicKey=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.audiences=

# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1@$2
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.security.identity.mapping.value.kerb=$1@$2

# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=true

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=dashboard-nifi-1405-1.dashboard-nifi-1405-headless.default.svc.cluster.local
nifi.cluster.node.protocol.port=6007
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=1 mins
nifi.cluster.flow.election.max.candidates=

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=dashboard-nifi-1405-zookeeper:2181
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi

# Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management
# Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes
# and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node
# The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal
# and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal
# values configured on the zookeeper server).
nifi.zookeeper.auth.type=
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=

# kerberos #
nifi.kerberos.krb5.file=

# kerberos service principal #
nifi.kerberos.service.principal=
nifi.kerberos.service.keytab.location=

# kerberos spnego principal #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours

# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=
nifi.web.http.network.interface.lo=lo

--------

app.log from the second node:

2022-07-01 11:51:11,944 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007; will use this address for sending heartbeat messages
2022-07-01 11:51:11,945 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007. Will send Cluster Connection Request to this address
2022-07-01 11:51:12,303 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2022-07-01 11:51:17,309 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007; will use this address for sending heartbeat messages
2022-07-01 11:51:17,310 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007. Will send Cluster Connection Request to this address
2022-07-01 11:51:17,935 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2022-07-01 11:51:22,942 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007; will use this address for sending heartbeat messages
2022-07-01 11:51:22,943 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007. Will send Cluster Connection Request to this address
2022-07-01 11:51:23,442 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2022-07-01 11:51:28,447 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007; will use this address for sending heartbeat messages
2022-07-01 11:51:28,447 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007. Will send Cluster Connection Request to this address
2022-07-01 11:51:28,849 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2022-07-01 11:51:33,853 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007; will use this address for sending heartbeat messages
2022-07-01 11:51:33,854 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007. Will send Cluster Connection Request to this address
2022-07-01 11:51:34,316 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2022-07-01 11:51:39,326 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007; will use this address for sending heartbeat messages
2022-07-01 11:51:39,326 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007. Will send Cluster Connection Request to this address
2022-07-01 11:51:39,853 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2022-07-01 11:51:44,863 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007; will use this address for sending heartbeat messages
2022-07-01 11:51:44,863 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007. Will send Cluster Connection Request to this address
2022-07-01 11:51:45,397 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2022-07-01 11:51:47,203 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive
2022-07-01 11:51:47,204 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Archive cleanup completed for container default; will now allow writing to this container. Bytes used = 57.01 GB, bytes free = 60.86 GB, capacity = 117.87 GB
2022-07-01 11:51:50,405 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007; will use this address for sending heartbeat messages
2022-07-01 11:51:50,406 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at dashboard-nifi-1348-0.dashboard-nifi-1348-headless.default.svc.cluster.local:6007. Will send Cluster Connection Request to this address
2022-07-01 11:51:50,833 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2022-07-01 11:51:55,834 INFO [main] o.a.nifi.controller.StandardFlowService Flow controller will load local dataflow and suspend connection handshake until a cluster connection response is received.
2022-07-01 11:51:55,849 INFO [main] o.apache.nifi.controller.FlowController Cluster State changed from Not Clustered to Clustered
2022-07-01 11:51:55,852 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Primary Node; this node is an active participant in the election.
2022-07-01 11:51:55,853 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election.
2022-07-01 11:51:55,862 INFO [main] o.apache.nifi.controller.FlowController Will no longer send heartbeats
2022-07-01 11:51:55,868 INFO [main] o.a.n.wali.SequentialAccessWriteAheadLog Recovering records from Write-Ahead Log at ../flowfile_repository
2022-07-01 11:51:55,873 INFO [main] o.a.n.wali.SequentialAccessWriteAheadLog No Snapshot File to recover from at ../flowfile_repository. Now recovering records from 0 journal files
2022-07-01 11:51:55,875 INFO [main] o.a.n.wali.SequentialAccessWriteAheadLog Successfully recovered 0 records in 6 milliseconds. Now checkpointing to ensure that Write-Ahead Log is in a consistent state
2022-07-01 11:51:55,916 INFO [main] o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 0 Records and 0 Swap Files in 41 milliseconds (Stop-the-world time = 37 milliseconds), max Transaction ID -1
2022-07-01 11:51:55,919 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Successfully restored 0 FlowFiles and 0 Swap Files
2022-07-01 11:51:55,951 INFO [main] o.apache.nifi.controller.FlowController Performed initial validation of all components in 1 milliseconds
2022-07-01 11:51:55,952 INFO [main] o.apache.nifi.controller.FlowController Starting 0 processors/ports/funnels
2022-07-01 11:51:55,952 INFO [main] o.apache.nifi.controller.FlowController Started 0 Remote Group Ports transmitting
2022-07-01 11:51:56,016 INFO [main] org.apache.nifi.web.server.JettyServer Flow loaded successfully.
2022-07-01 11:51:56,028 INFO [main] org.apache.nifi.nar.NarAutoLoader Starting NAR Auto-Loader for directory ./extensions ...
2022-07-01 11:51:56,030 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2022-07-01 11:51:56,031 INFO [main] org.apache.nifi.web.server.JettyServer https://10.1.2.67:8443/nifi
2022-07-01 11:51:56,031 INFO [main] org.apache.nifi.web.server.JettyServer https://127.0.0.1:8443/nifi
2022-07-01 11:51:56,070 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2022-07-01 11:51:56,071 INFO [main] org.apache.nifi.NiFi Started Application Controller in 94.386 seconds (94386630200 ns)
2022-07-01 11:51:56,436 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:01,904 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:07,311 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:12,773 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:18,360 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:23,839 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:29,313 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:34,792 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:40,163 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:45,210 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2d198533 checkpointed with 2 Records and 0 Swap Files in 10 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID 1
2022-07-01 11:52:45,565 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:47,215 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive
2022-07-01 11:52:47,215 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Archive cleanup completed for container default; will now allow writing to this container. Bytes used = 57.01 GB, bytes free = 60.86 GB, capacity = 117.87 GB
2022-07-01 11:52:50,977 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message
2022-07-01 11:52:56,333 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message

[#2]

Hi @Robybrown,

I was facing with similar issue. You need to enable cert manager and let it generate internal nifi certificates for each node. Tricky part is, that hostname is limited to 64 bytes (64 characters), so cert manager can throw an error, that hostname is to long. This can be overriden by "fullnameOverride" property to make it shorter for example. Next step should be sticky sessions configuration on ingress level.