how do use the NiFi toolkit to create an IP within the SAN as mentioned here:
https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#securing-nifi-with-tls-toolkit:~:text=...
I am using the following script:
bash "${NIFI_HOME_DIR}"/nifi/nifi-toolkit-current/bin/tls-toolkit.sh standalone -O -n "localhost,${NIFI_SAN_DNS}" -C "${cn}" --subjectAlternativeNames "${NIFI_SAN_DNS},IP:${NIFI_SAN_IP}" -o "${NIFI_HOME_DIR}"/key_trust
my output is as follows from my bash script:
tls-toolkit.sh: JAVA_HOME not set; results may vary
The TLS Toolkit is deprecated and targeted for removal in Apache NiFi 2.0.
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine - No nifiPropertiesFile specified, using embedded one.
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Running standalone certificate generation with output directory /var/mnt/NiFi/key_trust
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Generated new CA certificate /var/mnt/NiFi/key_trust/nifi-cert.pem and key /var/mnt/NiFi/key_trust/nifi-key.key
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Using alternate name nifi.server.home with hostname localhost.
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Writing new ssl configuration to /var/mnt/NiFi/key_trust/localhost
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Successfully generated TLS configuration for localhost 1 in /var/mnt/NiFi/key_trust/localhost
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Using alternate name IP:192.168.1.2 with hostname nifi.server.home.
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Writing new ssl configuration to /var/mnt/NiFi/key_trust/nifi.server.home
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Successfully generated TLS configuration for nifi.server.home 1 in /var/mnt/NiFi/key_trust/nifi.server.home
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Generating new client certificate /var/mnt/NiFi/key_trust/CN=sherloq_key_OU=NiFi_C=GB.p12
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - Successfully generated client certificate /var/mnt/NiFi/key_trust/CN=sherloq_key_OU=NiFi_C=GB.p12
[main] INFO org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone - tls-toolkit standalone completed successfully
and when I look at the SAN in the cert all I see is the following:
root@fedora:/var/mnt/NiFi/key_trust# openssl x509 -in nifi-cert.pem -text -noout | grep "DNS"
DNS:localhost
and there is no IP or additional SAN and within chrome I have the following:
my main issue is that we need the SAN IP to be able to talk to NiFi registry when using SSL but I cannot seem to get the communication to work
here is my full build script for NiFi:
#!/bin/bash
function info {
tput bold;
tput setaf 3;
echo $1;
tput sgr0;
}
# Check if the user has root privileges
if [[ $EUID -ne 0 ]]; then
echo "YOU MUST BE ROOT TO RUN THIS SCRIPT."
exit 1
fi
podman rm -f nifi
wait
info "=== Before we begin we need some information ==="
read -r -p "PRESS ENTER WHEN READY"
info "=== LINUX OS DETERMINATION ==="
# shellcheck disable=SC2155
# shellcheck disable=SC2002
export ID=$( cat /etc/os-release | awk -F= '$1=="ID" { print $2 ;}')
# shellcheck disable=SC2155
# shellcheck disable=SC2002
export VARIANT=$( cat /etc/os-release | awk -F= '$1=="VARIANT_ID" { print $2 ;}')
echo "${ID}" "${VARIANT}"
info "=== Setting up NiFI ==="
sleep 1
info "=== Add a Custom Root Directory Location for your Containers ==="
read -r -p "Enter Custom Directory [/var/mnt]: " rootDir
export ROOT_DIR=${rootDir:-/var/mnt}
export NIFI_HOME_DIR=${ROOT_DIR}/NiFi
sleep 1
info "=== Add a Custom CN Key Name ==="
read -r -p "Enter CN Key Name Here [nifi_key]: " keyName
export key_name=${keyName:-nifi_key}
# shellcheck disable=SC2140
export key="${NIFI_HOME_DIR}"/key_trust/"'CN=${key_name:-nifi_key}_OU=NiFi_C=GB.p12'"
export cn="CN=${key_name}, OU=NiFi, C=GB"
info "=== Add a Custom Port for your NiFi Container ==="
read -r -p "Enter Custom Port [default: 8181]: " nifiPort
export NIFI_PORT=${nifiPort:-8181}
# shellcheck disable=SC2155
export NIFI_PORT_CHECK=$(lsof -i -P -n | grep "${NIFI_PORT}")
if [[ "${NIFI_PORT_CHECK}" != "" ]]; then
info "=== PORT IN USE - WOULD YOU LIKE ME TO CHOOSE A PORT FOR YOU? ==="
read -r -p "Please Enter your Response [enter: Y/y or N/n]: " nifiAnswer
export NIFI_ANSWER=${nifiAnswer:-}
fi
if [[ "${NIFI_ANSWER}" =~ (Y|y) ]]; then
while [[ -n "${NIFI_PORT_CHECK}" ]]; do
echo "Port ${NIFI_PORT} is in use. Trying another port..."
NIFI_PORT=$((NIFI_PORT + 1))
NIFI_PORT_CHECK=$(lsof -i -P -n | grep "${NIFI_PORT}")
done
elif [[ "${NIFI_ANSWER}" =~ (N|n) ]]; then
while [[ -n "${NIFI_PORT_CHECK}" ]]; do
echo "Port ${NIFI_PORT} is in use. Trying another port..."
read -r -p "Enter a new port: " newNIFIPort
NIFI_PORT=${newNIFIPort}
NIFI_PORT_CHECK=$(lsof -i -P -n | grep "${NIFI_PORT}")
done
fi
info "=== Add a Custom Port for Protobuf ==="
read -r -p "Enter Custom Port [default: 8282]: " protobufPort
export PROTO_PORT=${protobufPort:-8282}
# shellcheck disable=SC2155
export PROTO_PORT_CHECK=$(lsof -i -P -n | grep "${PROTO_PORT}")
if [[ "${PROTO_PORT_CHECK}" != "" ]]; then
info "=== PORT IN USE - WOULD YOU LIKE ME TO CHOOSE A PORT FOR YOU? ==="
read -r -p "Please Enter your Response [enter: Y/y or N/n]: " protoAnswer
export PROTO_ANSWER=${protoAnswer:-}
fi
if [[ "${PROTO_ANSWER}" =~ (Y|y) ]]; then
while [[ -n "${PROTO_PORT_CHECK}" ]]; do
echo "Port ${PROTO_PORT} is in use. Trying another port..."
PROTO_PORT=$((PROTO_PORT + 1))
PROTO_PORT_CHECK=$(lsof -i -P -n | grep "${PROTO_PORT}")
done
elif [[ "${PROTO_ANSWER}" =~ (N|n) ]]; then
while [[ -n "${PROTO_PORT_CHECK}" ]]; do
echo "Port ${PROTO_PORT} is in use. Trying another port..."
read -r -p "Enter a new port: " newPROTOPort
PROTO_PORT=${newPROTOPort}
PROTO_PORT_CHECK=$(lsof -i -P -n | grep "${PROTO_PORT}")
done
fi
info "=== Add a Custom Port for Web Proxy for NiFi ==="
read -r -p "Enter Custom Port [default: 9191]: " nifiProxy
export NIFI_WEB_PROXY_PORT=${nifiProxy:-9191}
# shellcheck disable=SC2155
export PROXY_PORT_CHECK=$(lsof -i -P -n | grep "${NIFI_WEB_PROXY_PORT}")
if [[ "${PROXY_PORT_CHECK}" != "" ]]; then
info "=== PORT IN USE - WOULD YOU LIKE ME TO CHOOSE A PORT FOR YOU? ==="
read -r -p "Please Enter your Response [enter: Y/y or N/n]: " proxyAnswer
export PROXY_ANSWER=${proxyAnswer:-}
fi
if [[ "${PROXY_ANSWER}" =~ (Y|y) ]]; then
while [[ -n "${PROXY_PORT_CHECK}" ]]; do
echo "Port ${NIFI_WEB_PROXY_PORT} is in use. Trying another port..."
NIFI_WEB_PROXY_PORT=$((NIFI_WEB_PROXY_PORT + 1))
PROXY_PORT_CHECK=$(lsof -i -P -n | grep "${NIFI_WEB_PROXY_PORT}")
done
elif [[ "${PROXY_ANSWER}" =~ (N|n) ]]; then
while [[ -n "${PROXY_PORT_CHECK}" ]]; do
echo "Port ${NIFI_WEB_PROXY_PORT} is in use. Trying another port..."
read -r -p "Enter a new port: " newPROXYPort
NIFI_WEB_PROXY_PORT=${newPROXYPort}
PROXY_PORT_CHECK=$(lsof -i -P -n | grep "${NIFI_WEB_PROXY_PORT}")
done
fi
echo "Port ${NIFI_WEB_PROXY_PORT} is available."
info "=== Please take note of port as you will need to login to NiFi ==="
read -r -p "PRESS ENTER WHEN READY"
# shellcheck disable=SC2155
export NIFI_WEB_PROXY_IP="$(ip route get 8.8.8.8 | grep -oP 'src \K[^ ]+')"
# shellcheck disable=SC2155
export NIFI_SAN_IP="${NIFI_WEB_PROXY_IP}"
# shellcheck disable=SC2155
export NIFI_NET_IP=$(ip ro | awk -v san_ip="${NIFI_SAN_IP}" '$0 ~ san_ip && $2 == "dev" {print $1; exit}')
export NIFI_NET_IP=${NIFI_NET_IP%/*}
export NIFI_SAN_DNS=nifi.server.home
export NIFI_WEB_PROXY_HOST=${NIFI_WEB_PROXY_IP}:${NIFI_WEB_PROXY_PORT}
info "=== INFLUX SNAPSHOT VERSION ==="
read -r -p "Enter Snapshot Version ..or.. ENTER TO ACCEPT DEFAULT [default:1.28.0-SNAPSHOT]: " snapshotVersion
export influx_version=${snapshotVersion:-1.28.0-SNAPSHOT}
info "=== NIFI VERSION ==="
read -r -p "Enter NIFI Version ..or.. ENTER TO ACCEPT CURRENT KNOWN WORKING [current known working:1.25.0]: " nifiVersion
export nifi_version=${nifiVersion:-1.25.0}
# shellcheck disable=SC2155
export work_dir=$(pwd)
if [[ ! -d ${NIFI_HOME_DIR} ]]; then
mkdir -p "${NIFI_HOME_DIR}"
elif [[ -d ${NIFI_HOME_DIR} ]]; then
rm -rf "${NIFI_HOME_DIR}"; mkdir "${NIFI_HOME_DIR}"
fi
mkdir -p "${NIFI_HOME_DIR}"/UID
touch "${NIFI_HOME_DIR}"/UID/admin
sleep 1
# shellcheck disable=SC2027
echo """${cn}""" > "${NIFI_HOME_DIR}"/UID/admin
podman pull docker.io/apache/nifi:"${nifi_version}"
podman run -d \
--name nifi_cp \
-p "${NIFI_PORT}":8080 \
nifi:"${nifi_version}"
podman run -d \
--name protobuf \
-p "${PROTOBUF_PORT}":8080 \
whiver/nifi-protobuf:latest
## influx ##
if [[ ! -e ${work_dir}/nifi-influx-database-nar-${influx_version}.nar ]]; then
wget https://github.com/influxdata/nifi-influxdb-bundle/releases/download/v"${influx_version}"/nifi-influx-database-nar-"${influx_version}".nar
fi
## apache iotdb ##
wget https://repo1.maven.org/maven2/org/apache/nifi/nifi-iotdb-nar/"${nifi_version}"/nifi-iotdb-nar-"${nifi_version}".nar
podman cp nifi_cp:/opt/nifi "${NIFI_HOME_DIR}"
podman cp protobuf:/opt/nifi/nifi-1.4.0/lib/nifi-protobuf-processor-0.2.0.nar "${NIFI_HOME_DIR}"/nifi/nifi-current/lib/
# shellcheck disable=SC2086
mkdir ${NIFI_HOME_DIR}/nifi/certs && mkdir "${NIFI_HOME_DIR}"/nifi-registry && mkdir "${NIFI_HOME_DIR}"/nifi-registry/certs/
mv "${work_dir}"/nifi-influx-database-nar-"${influx_version}".nar "${NIFI_HOME_DIR}"/nifi/nifi-current/lib/.
mv "${work_dir}"/nifi-iotdb-nar-"${nifi_version}".nar "${NIFI_HOME_DIR}"/nifi/nifi-current/lib/.
podman stop nifi_cp && podman stop protobuf
wait
podman rm -f nifi_cp && podman rm -f protobuf
wait
cd "${NIFI_HOME_DIR}"/nifi/nifi-toolkit-current/bin/ || return
#bash "${NIFI_HOME_DIR}"/nifi/nifi-toolkit-current/bin/tls-toolkit.sh standalone -O -n "${NIFI_SAN_IP}" -C "${cn}" -o "${NIFI_HOME_DIR}"/key_trust
# shellcheck disable=SC2027
bash "${NIFI_HOME_DIR}"/nifi/nifi-toolkit-current/bin/tls-toolkit.sh standalone -O -n "localhost,${NIFI_SAN_DNS}" -C "${cn}" --subjectAlternativeNames "${NIFI_SAN_DNS},IP:192.168.1.2" -o "${NIFI_HOME_DIR}"/key_trust
mv "${NIFI_HOME_DIR}"/key_trust/localhost/*.jks "${NIFI_HOME_DIR}"/nifi/certs/ && chown 1000:1000 -R "${NIFI_HOME_DIR}" && cp "${NIFI_HOME_DIR}"/nifi/certs/* "${NIFI_HOME_DIR}"/nifi-registry/certs/
# shellcheck disable=SC2002
KEYSTORE_PASSWORD="$(cat "${NIFI_HOME_DIR}"/key_trust/localhost/nifi.properties | grep nifi.security.keystorePasswd)"
KEYSTORE_PASSWORD="$(echo -n "${KEYSTORE_PASSWORD//nifi.security.keystorePasswd=/""}")"
# shellcheck disable=SC2002
TRUSTSTORE_PASSWORD="$(cat "${NIFI_HOME_DIR}"/key_trust/localhost/nifi.properties | grep nifi.security.truststorePasswd)"
TRUSTSTORE_PASSWORD="$(echo -n "${TRUSTSTORE_PASSWORD//nifi.security.truststorePasswd=/""}")"
USER_ID="$(cat "${NIFI_HOME_DIR}"/UID/admin)"
## nifi config optimisation ##
sed -i 's/java.arg.2=-Xms512m/java.arg.2=-Xms10g/g' "${NIFI_HOME_DIR}"/nifi/nifi-current/conf/bootstrap.conf
sed -i 's/java.arg.3=-Xmx512m/java.arg.3=-Xmx10g/g' "${NIFI_HOME_DIR}"/nifi/nifi-current/conf/bootstrap.conf
##rm /home/NiFi/nifi/nifi-current/conf/authorizers.xml
podman run -d \
--name nifi \
--security-opt label=disable \
--privileged \
-v "${NIFI_HOME_DIR}"/nifi/certs:/opt/certs \
-v "${NIFI_HOME_DIR}"/nifi/nifi-current/lib:/opt/nifi/nifi-current/lib \
-p "${NIFI_WEB_PROXY_PORT}":8443 \
-e NIFI_WEB_PROXY_HOST="${NIFI_WEB_PROXY_HOST}" \
-e AUTH=tls \
-e KEYSTORE_PATH=/opt/certs/keystore.jks \
-e KEYSTORE_TYPE=JKS \
-e KEYSTORE_PASSWORD="$KEYSTORE_PASSWORD" \
-e TRUSTSTORE_PATH=/opt/certs/truststore.jks \
-e TRUSTSTORE_TYPE=JKS \
-e TRUSTSTORE_PATH=/opt/certs/truststore.jks \
-e TRUSTSTORE_PASSWORD="$TRUSTSTORE_PASSWORD" \
-e INITIAL_ADMIN_IDENTITY="$USER_ID" \
nifi:"${nifi_version}"
## remove for new approach ##
#-v "${NIFI_HOME_DIR}"/nifi/nifi-current/lib/nifi-protobuf-processor-0.2.0.nar:/opt/nifi/nifi-current/lib/nifi-protobuf-processor-0.2.0.nar \
#-v "${NIFI_HOME_DIR}"/nifi/nifi-current/lib/nifi-influx-database-nar-"${influx_version}".nar:/opt/nifi/nifi-current/lib/nifi-influx-database-nar-"${influx_version}".nar \
if [[ ! -e /etc/systemd/system/nifi.service ]]; then
cat << EOF > /etc/systemd/system/nifi.service
[Unit]
Description=NiFi Container
After=firewalld.service
[Service]
Restart=always
ExecStart=/usr/bin/podman start -a nifi
ExecStop=/usr/bin/podman stop -t 2 nifi
[Install]
WantedBy=multi-user.target
EOF
systemctl enable nifi.service && systemctl start nifi.service
fi
info "=== DO YOU INTEND TO ACCESS NIFI FROM A REMOTE HOST? ==="
read -r -p "Please Enter your Response [enter: Y/y or N/n]: " hostAnswer
export HOST_ANSWER=${hostAnswer:-}
if [[ "${HOST_ANSWER}" =~ (Y|y) ]]; then
info "=== Set up Remote Host Details for NIFI Key ==="
sleep 1
info "=== NOTE: YOU NEED SSH ENABLED ON REMOTE HOST ==="
sleep 2
info "=== Add a SCP User ==="
read -r -p "Enter Remote User Name [insert user here]: " userName
export scp_user=${userName:-}
info "=== Add Remote User Password ===="
read -r -s -p "Enter Your User Password [insert password here]: " userPasswd
echo
echo "Re-enter Your User Password: "
read -r -s -p "" userPasswd2
# shellcheck disable=SC2053
while [[ $userPasswd != $userPasswd2 ]]; do
echo "Passwords do not match."
echo "Please try again."
read -r -s -p "Enter Your User Password: " userPasswd
echo
read -r -s -p "Please Re-Enter Your User Password: " userPasswd2
echo
done
echo "Passwords Match Hooray."
export ssh_pwd=${userPasswd:-}
info "=== Add Remote Host IP Address ==="
read -r -p "Enter Your IP Here [192.168.1.1]: " remoteIpAddress
export scp_ip=${remoteIpAddress:-192.168.1.1}
info "=== Add Remote Host SSH Port ==="
read -r -p "Enter Your Port Here [Default: 22]: " remotePort
export scp_port=${remotePort:-22}
info "=== Add Remote SCP Directory ==="
sleep 1
info "=== NOTE: TAKE THIS OPPORTUNITY TO SETUP A FOLDER ON REMOTE HOST ==="
read -r -p "PRESS ENTER WHEN READY"
read -r -p "Enter Remote Directory [example:/Users/${scp_user}/Desktop/p12]: " remoteDir
export remote_dir=${remoteDir:-/Users/${scp_user}/Desktop/p12}
sshpass -p "${ssh_pwd}" scp -P "${scp_port}" -o StrictHostKeyChecking=no "${key}" "${scp_user}"@"${scp_ip}":"${remote_dir}"
elif [[ "${HOST_ANSWER}" =~ (N|n) ]]; then
info "=== Add Local Directory for NiFI p12 key and pwd ==="
sleep 1
info "=== NOTE: TAKE THIS OPPORTUNITY TO SETUP A FOLDER ON YOUR HOST ==="
read -r -p "PRESS ENTER WHEN READY"
# shellcheck disable=SC2155
export local_user=$(who | awk 'NR==1 {print$1}')
read -r -p "Enter Local Directory [example:/var/home/${local_user}/Documents/p12]: " localDir
export local_dir=${localDir:-/var/home/${local_user}/Documents/p12}
cp "${NIFI_HOME_DIR}"/key_trust/CN* "${local_dir}"/.
chown 1000:1000 "${local_dir}"/*
fi
info "=== DETERMINING FIREWALL STATE ==="
if [[ "${ID}" = "fedora" ]]; then
# shellcheck disable=SC2155
export FIREWALL_STATE=$(firewall-cmd --state)
echo "${VARIANT}" "${ID}" "${FIREWALL_STATE}"
elif [[ "${ID}" == "debian" || "${ID}" == "ubuntu" ]]; then
# shellcheck disable=SC2155
export FIREWALL_STATE=$(ufw status | awk -F: '$1=="Status" { print $2;}')
echo "${VARIANT}" "${ID}" "${FIREWALL_STATE}"
fi
if [[ "${ID}" = "fedora" ]]; then
# shellcheck disable=SC2155
export ACTIVE_ZONE=$(firewall-cmd --get-active-zones | awk 'NR==1' | grep -o '^[^(]*')
echo "ACTIVE ZONE=""${ACTIVE_ZONE}"
fi
if [[ "${ID}" == "fedora" ]] && [[ "${FIREWALL_STATE}" == "running" ]]; then
echo "== FIREWALL IS RUNNING - APPLYING INFLUX PORTS TO FIREWALL =="
firewall-cmd --zone="${ACTIVE_ZONE}" --permanent --add-port="${NIFI_WEB_PROXY_PORT}"/tcp
firewall-cmd --reload
elif [[ "${ID}" == "debian" || "${ID}" == "ubuntu" ]] && [[ "${FIREWALL_STATE}" == "active" ]]; then
echo "== FIREWALL IS ACTIVE - APPLYING INFLUX PORTS TO FIREWALL =="
ufw allow "${NIFI_WEB_PROXY_PORT}"/tcp
ufw disable && ufw enable
fi
sleep 40
podman exec nifi sed -i 's/NIFI_ANALYTICS_PREDICT_ENABLED:-false/NIFI_ANALYTICS_PREDICT_ENABLED:-true/g' /opt/nifi/scripts/start.sh
podman exec nifi sed -i 's/NIFI_ANALYTICS_PREDICT_INTERVAL:-3 mins/NIFI_ANALYTICS_PREDICT_INTERVAL:-60 mins/g' /opt/nifi/scripts/start.sh
podman stop nifi
wait
podman start nifi
# shellcheck disable=SC2155
export CN_PASSWD=$(cat "${NIFI_HOME_DIR}"/key_trust/CN*.password)
echo "CN PASSWORD: ${CN_PASSWD}"
Announcements
Community Announcements
Community Announcements
Product Announcements
What's New @ Cloudera
Community Announcements
