CFM - NiFi is not starting due to "ERROR o.a.n.b.Command: The sensitive.key file /var/run/cloudera-scm-agent/process/6145-nifi-NIFI_NODE/sensitive.key already exists. That shouldn't have been. Aborting"

[#1]

Hi

I try to install a fresh CFM instalation in a CM&CDH 6.2.0 cluster.

I install NiFi CA and NiFi Registry and NiFi (Parcel distribution with csd) in the cluster.

NiFi CA and NiFi Registry works fine, both services are started and looks healthy but i can't start NiFi, I install in 2 nodes to made a cluster but i can't start, always the same message:

19/10/31 11:58:12 ERROR o.a.n.b.Command: The sensitive.key file /var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE/sensitive.key already exists. That shouldn't have been. Aborting.
Thu Oct 31 11:58:16 CET 2019
JAVA_HOME=/usr/java/jdk1.8.0_121
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/nifi_nifi-NIFI_NODE-6ef7e4e859120410b823067891375e2e_pid45532.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE
CMF_CONF_DIR=
Thu Oct 31 11:58:16 CET 2019: WARNING: this command is restarted by Cloudera Manager, this mean that initial start is failed.
Complete environment is:
Thu Oct 31 11:58:16 CET 2019: CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive-hcatalog TOMCAT_HOME=/usr/lib/bigtop-tomcat CM_CSD_SCRIPT=scripts/control.sh COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh CDH_PIG_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/pig CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/solr PARCELS_ROOT=/opt/cloudera/parcels FLUME_CLASSPATH=/opt/cloudera/parcels/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373/lib/hadoop/lib/* NIFI_REGISTRY_DIST=/opt/cloudera/parcels/CFM-1.0.1.0/REGISTRY CLOUDERA_MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive CGROUP_ROOT_DEVICES=/sys/fs/cgroup/devices SCM_DEFINES_SCRIPTS=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/meta/cdh_env.sh:/opt/cloudera/parcels/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373/meta/gplextras_env.sh:/opt/cloudera/parcels/CFM-1.0.1.0/meta/nifi_bundle_env.sh CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/avro MR2_CLASSPATH=/opt/cloudera/parcels/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373/lib/hadoop/lib/* NIFI_PID_DIR=/var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE CLOUDERA_ORACLE_CONNECTOR_JAR=/usr/share/java/oracle-connector-java.jar CDH_VERSION=6 HIVE_DEFAULT_XML=/etc/hive/conf.dist/hive-default.xml SEARCH_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/search MGMT_HOME=/opt/cloudera/cm CGROUP_GROUP_BLKIO= PARCEL_DIRNAMES=CDH-6.2.0-1.cdh6.2.0.p0.967373:GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373:CFM-1.0.1.0 CERT_OUTPUT_DIRECTORY=/var/lib/nifi/cert JSVC_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/bigtop-utils CDH_SQOOP_HOME=/usr/lib/sqoop JAVA_LIBRARY_PATH=/opt/cloudera/parcels/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373/lib/hadoop/lib/native LD_LIBRARY_PATH=/opt/cloudera/parcels/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373/lib/impala/lib ZK_QUORUM=puemaster1.pue.es:2181,pueworker1.pue.es:2181,pueworker2.pue.es:2181 nifi_service_principal=nifi/pueworker5.pue.es@PUE.ES CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/impala BOOTSTRAP_CONF_DIR=/var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE CDH_MR2_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-mapreduce NIFI_LOG_DIR=/var/log/nifi CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-httpfs CSD_NIFI_CONF=/var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE CM_STATUS_CODES=STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE CDH_HUE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue CDH_MR1_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-0.20-mapreduce CGROUP_ROOT_MEMORY=/sys/fs/cgroup/memory SPARK_LIBRARY_PATH=/opt/cloudera/parcels/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373/lib/hadoop/lib/native CMF_AGENT_ARGS= PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin CGROUP_GROUP_MEMORY= CDH_KAFKA_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kafka CONF_DIR=/var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/sentry CMF_SERVER_ROOT=/opt/cloudera/cm CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/parquet SUPERVISOR_GROUP_NAME=6146-nifi-NIFI_NODE PWD=/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE CDH_SPARK_CLASSPATH=/opt/cloudera/parcels/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373/lib/spark-netlib/lib/* JAVA_HOME=/usr/java/jdk1.8.0_121 ENABLE_TLS=true HADOOP_CLASSPATH=/opt/cloudera/parcels/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373/lib/hadoop/lib/* NIFI_COMMONS=/opt/cloudera/parcels/CFM-1.0.1.0 NIFI_WORKING_DIRECTORY=/var/lib/nifi SCRIPTS_DIR=/var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE/scripts LANG=en_US.UTF-8 nifi_spnego_principal=HTTP/pueworker5.pue.es@PUE.ES CGROUP_ROOT_CPUACCT=/sys/fs/cgroup/cpu,cpuacct CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop CGROUP_ROOT_BLKIO=/sys/fs/cgroup/blkio CGROUP_ROOT_CPU=/sys/fs/cgroup/cpu,cpuacct CDH_SQOOP2_HOME=/usr/lib/sqoop2 NIFI_CONF_DIR=/var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE CDH_CRUNCH_HOME=/usr/lib/crunch CDH_LLAMA_HOME=/usr/lib/llama/ CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark CLOUDERA_ROOT=/opt/cloudera CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/bin/hadoop CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-hdfs CLOUDERA_DIR=/opt/cloudera/cm SUPERVISOR_ENABLED=1 HOME=/var/lib/nifi SHLVL=1 CA_GENERATE_CERTS=true CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie KEYTRUSTEE_SERVER_HOME=/usr/lib/keytrustee-server CDH_KUDU_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kudu CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase-solr NIFI_ENV_PATH=/var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE/nifi-env.sh KEYTRUSTEE_KP_HOME=/usr/share/keytrustee-keyprovider CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/flume-ng CDH_KMS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-kms CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase CDH_YARN_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-yarn CUSTOM_JAVA_HOME= CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/zookeeper SUPERVISOR_PROCESS_NAME=6146-nifi-NIFI_NODE SUPERVISOR_SERVER_URL=unix:///run/cloudera-scm-agent/supervisor/supervisord.sock CGROUP_GROUP_CPU= CMF_PACKAGE_DIR=/opt/cloudera/cm-agent/service ORACLE_HOME=/usr/share/oracle/instantclient CGROUP_GROUP_DEVICES=system.slice/cloudera-scm-agent.service WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/etc/hive-webhcat/conf.dist/webhcat-default.xml CLOUDERA_POSTGRESQL_JDBC_JAR=/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar CGROUP_GROUP_CPUACCT= CSD_JAVA_OPTS=-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/nifi_nifi-NIFI_NODE-6ef7e4e859120410b823067891375e2e_pid45532.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh NIFI_TOOLKIT_DIST=/opt/cloudera/parcels/CFM-1.0.1.0/TOOLKIT HADOOP_CREDSTORE_PASSWORD=********** CSD_HOST=pueworker5.pue.es HBASE_CLASSPATH=/opt/cloudera/parcels/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373/lib/hadoop/lib/* NIFI_DIST=/opt/cloudera/parcels/CFM-1.0.1.0/NIFI _=/usr/bin/env
Connecting to puemaster1.pue.es:2181,pueworker1.pue.es:2181,pueworker2.pue.es:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[cluster, controller, brokers, zookeeper, isr_change_notification, admin, log_dir_event_notification, ngdata, nifi, controller_epoch, aliases.json, solr, clusterstate.json, solr2, consumers, hive_zookeeper_namespace_hive, latest_producer_id_block, config, hbase, sentry]
Sourcing /var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE/nifi-env.sh...

Java home: /usr/java/jdk1.8.0_121
NiFi home: /opt/cloudera/parcels/CFM-1.0.1.0/NIFI

Bootstrap Config File: /var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE/bootstrap.conf

19/10/31 11:58:17 ERROR o.a.n.b.Command: The sensitive.key file /var/run/cloudera-scm-agent/process/6146-nifi-NIFI_NODE/sensitive.key already exists. That shouldn't have been. Aborting.

I search on google and find a Jira in Apache with the related issue, but is old (2017) and the author said that is not solve and he couldn't work on the "bug"

The Jira: https://issues.apache.org/jira/browse/NIFI-4150

Any idea to fix this? like a i said is a fresh instalation of CFM 1.0.1.0.

Regards.

[#2]

Finally, i remove NiFi service and try to configure step by step... i was tried to configure tls and ldap authentication in one step before start and generate certs from NiFi CA, now the cluster are working but i only have access to a single web ui, the other one (i install in 2 nodes) doesn't work.

[#3]

@Sergiete 

The nifi-app.log should tell you why the second node failed to start if NiFi got beyond the bootstrap process.  If not, the nifi bootstrap.log will tell you why it failed to start.

Matt

[#4]

@Sergiete 

The sensitive.key file is created during NiFi startup and is removed once startup completes successfully.  The fact that it still exists when you are trying to start NiFi, tells me that some previous startup attempt failed after sensitive.key was created, but before startup completed.  

You can safely remove this sensitive.key file from your NiFi nodes and start your NiFi service again.  If NiFi fails to start and you see the sensitive.key was created and not removed again, look through your NiFi logs to see why it failed.  It will be for a different reason since you had manually removed the sensitive.key before that startup.

I have not seen this condition occur on any of my CFM installs yet, but have heard of this happening before.  What I do not have is logs to determine what is happening in those cases.

Matt

[#5 · Accepted solution]

Hi @MattWho 

I understand, but like i said was a fresh instalation and NiFi never works. I try to remove the file from the /run/cloudera-scm-agent/proccess/NIFI_FOLDER but not works and i can't find the file in other location on the node.

Maybe was my fault, i try to start directly with TLS and never works, maybe the first start can't create the cert from the NiFi CA (because i saw in the role log a message with missing keystore file) and is for that the NiFi can't start fine and then create the previous sensitive.key file.

Finally i remove the service, start un-secure, force to NiFi CA to recreate the certs and then, activate TLS/SSL and is when NiFi starts fine and is working.

Now i have another problem, i install NiFi in two workers nodes but only one web ui works, the other web ui shows this message:

Secure Connection Failed

An error occurred during a connection to HOSTNAME:8443. Certificate key usage inadequate for attempted operation. Error code: SEC_ERROR_INADEQUATE_KEY_USAGE

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.

Any way @MattWho thanks for your help, i'll keep working on the NiFi web ui issue and try to configure ldap authentication, in this moment i only get the user name validation, i try to integrate the gruop validation.

Best Regards.