Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to have 3 nodes for Apache Nifi cluster in Kubernetes?

avatar
Explorer
I am using external zookeeper and I have applied this yaml in order to have Nifi in cluster mode in Kubernetes (see below)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nifi
  labels:
    name: nifi
    app: nifi
  annotations:
    app.kubernetes.io/name: nifi
    app.kubernetes.io/part-of: nifi
spec:
  revisionHistoryLimit: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nifi
  template:
    metadata:
      labels:
        app: nifi
    spec:
      automountServiceAccountToken: false
      enableServiceLinks: false
      restartPolicy: Always
      securityContext:
        runAsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: nifi
        image: XXX
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: nifi
        - containerPort: 8082
          name: cluster
        env:
        - name: "NIFI_SENSITIVE_PROPS_KEY"
          value: "XXX"
        - name: ALLOW_ANONYMOUS_LOGIN
          value: "no"
        - name: SINGLE_USER_CREDENTIALS_USERNAME 
          value: XXX
        - name: SINGLE_USER_CREDENTIALS_PASSWORD 
          value: XXX
        - name: NIFI_WEB_HTTP_HOST
          value: "0.0.0.0"
        - name: NIFI_WEB_HTTP_PORT
          value: "8080"
        # - name: NIFI_WEB_PROXY_HOST
        #   value: 0.0.0.0:8080
        # - name: HOSTNAME
        #   value: "nifi1"
        - name: NIFI_ANALYTICS_PREDICT_ENABLED
          value: "true"
        - name: NIFI_ELECTION_MAX_CANDIDATES
          value: "2"
        - name: NIFI_ELECTION_MAX_WAIT
          value: "1 min"
        - name: NIFI_CLUSTER_IS_NODE
          value: "true"
        - name: NIFI_JVM_HEAP_INIT
          value: "3g"
        - name: NIFI_JVM_HEAP_MAX
          value: "4g"
        - name: NIFI_CLUSTER_NODE_CONNECTION_TIMEOUT
          value: "2 min"
        - name: NIFI_CLUSTER_PROTOCOL_CONNECTION_HANDSHAKE_TIMEOUT
          value: "2 min"        
        - name: NIFI_CLUSTER_NODE_PROTOCOL_MAX_THREADS
          value: "15"
        - name: NIFI_CLUSTER_NODE_PROTOCOL_PORT
          value: "8082"
        - name: NIFI_CLUSTER_NODE_READ_TIMEOUT
          value: "15"
        - name: NIFI_ZK_CONNECT_STRING
          value: "zookeeper:2181"
        - name: NIFI_CLUSTER_NODE_ADDRESS
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        livenessProbe:
          exec:
            command:
              - pgrep
              - java
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 10
          failureThreshold: 3
          successThreshold: 1
        readinessProbe:
          exec:
            command:
              - pgrep
              - java
          initialDelaySeconds: 180
          periodSeconds: 30
          timeoutSeconds: 10
          failureThreshold: 3
          successThreshold: 1
        resources:
          requests:
            cpu: 400m
            memory: 1Gi
          limits:
            cpu: 500m
            memory: 2Gi
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          capabilities:
            drop:
              - ALL

I have one node in my Nifi cluster and node adress is : 0.0.0.0:8080

I would like to have 3 nodes and I think having 0.0.0.0:8080 for nodes is not good. I don't know how to deploy the 3 nodes in Kubernetes and with different node address.

How should I do it and modify my yaml ?

0 REPLIES 0