I have Nifi adminstered by HDF 3.4.1.1, running on a single node. The nifi-app.log has stopped updating ls /var/log/nifi2 nifi-app_2019-07-07_10.0.log nifi-app_2019-07-07_15.0.log nifi-app_2019-07-07_20.0.log nifi-setup.log nifi-user_2019-06-20.log nifi-user_2019-06-28.log nifi-app_2019-07-07_11.0.log nifi-app_2019-07-07_16.0.log nifi-app_2019-07-07_21.0.log nifi-user_2019-06-07.log nifi-user_2019-06-24.log nifi-user_2019-07-03.log nifi-app_2019-07-07_12.0.log nifi-app_2019-07-07_17.0.log nifi-app.log nifi-user_2019-06-10.log nifi-user_2019-06-25.log nifi-user_2019-07-04.log nifi-app_2019-07-07_13.0.log nifi-app_2019-07-07_18.0.log nifi-bootstrap_2019-06-27.log nifi-user_2019-06-11.log nifi-user_2019-06-26.log nifi-user_2019-07-05.log nifi-app_2019-07-07_14.0.log nifi-app_2019-07-07_19.0.log nifi-bootstrap.log nifi-user_2019-06-19.log nifi-user_2019-06-27.log nifi-user.log There are nifi flows actively running since 07-07. How can I get logs flowing again and debug this issue? The last entry in the nifi-app.log is: 2019-07-07 22:59:59,544 WARN [Timer-Driven Process Thread-14] o.a.n.controller.tasks.ConnectableTask Administratively Yielding ListS3 ... View more

I have a reproducible bug where nifi service won't start. Beginning from a clean HDF install, I change the JVM memory settings in Ambari UI - nifi-bootstrap-env. (I have 32G of memory on my host). # JVM memory settings java.arg.2=-Xms{ {nifi_initial_mem}} java.arg.3=-Xmx{ {nifi_max_mem}} to # JVM memory settings java.arg.2=-Xms12228m java.arg.3=-Xmx12228m Then the restart services is successful with no errors, but the service doesn't actually start (in Ambari UI summary it shows the component as stopped). Even if I revert the configs back, I get the same bug - Nifi won't start. stderr: /var/lib/ambari-agent/data/errors-225.txt None stdout: /var/lib/ambari-agent/data/output-225.txt 2019-06-03 14:34:05,743 - Stack Feature Version Info: Cluster Stack=3.4, Command Stack=None, Command Version=3.4.1.1-4 -> 3.4.1.1-4 2019-06-03 14:34:05,863 - Stack Feature Version Info: Cluster Stack=3.4, Command Stack=None, Command Version=3.4.1.1-4 -> 3.4.1.1-4 2019-06-03 14:34:05,866 - Group['nifiregistry'] {} 2019-06-03 14:34:05,867 - Group['hadoop'] {} 2019-06-03 14:34:05,867 - Group['nifi'] {} 2019-06-03 14:34:05,867 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-06-03 14:34:05,868 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-06-03 14:34:05,869 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2019-06-03 14:34:05,869 - User['nifiregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifiregistry'], 'uid': None} 2019-06-03 14:34:05,870 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifi'], 'uid': None} 2019-06-03 14:34:05,870 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2019-06-03 14:34:05,871 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2019-06-03 14:34:05,875 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2019-06-03 14:34:05,888 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} 2019-06-03 14:34:05,891 - Skipping Execute[('setenforce', '0')] due to not_if 2019-06-03 14:34:05,891 - Testing the JVM's JCE policy to see it if supports an unlimited key length. 2019-06-03 14:34:06,068 - The unlimited key JCE policy is required, and appears to have been installed. 2019-06-03 14:34:06,329 - Stack Feature Version Info: Cluster Stack=3.4, Command Stack=None, Command Version=3.4.1.1-4 -> 3.4.1.1-4 2019-06-03 14:34:06,392 - File['/usr/hdf/current/nifi/bin/nifi-env.sh'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0755} 2019-06-03 14:34:06,393 - Execute['export JAVA_HOME=/usr/jdk64/jdk1.8.0_112;/usr/hdf/current/nifi/bin/nifi.sh stop >> /var/log/nifi/nifi-setup.log'] {'user': 'nifi'} 2019-06-03 14:34:09,855 - Pid file /var/run/nifi/nifi.pid is empty or does not exist 2019-06-03 14:34:09,856 - File['/var/lib/ambari-agent/tmp/run_ca.sh'] {'owner': 'nifi', 'content': StaticFile('run_ca.sh'), 'group': 'nifi', 'mode': 0755} 2019-06-03 14:34:09,856 - Changing owner of package files 2019-06-03 14:34:09,856 - Directory['/usr/hdf/current/nifi-toolkit'] {'group': 'nifi', 'cd_access': 'a', 'recursion_follow_links': True, 'create_parents': False, 'recursive_ownership': True, 'owner': 'nifi', 'mode': 0755} 2019-06-03 14:34:09,862 - Directory['/var/run/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:09,863 - Directory['/var/lib/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:09,995 - Directory['/var/lib/nifi/database_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:09,995 - Directory['/var/lib/nifi/flowfile_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:09,996 - Directory['/var/lib/nifi/provenance_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:09,997 - Directory['/usr/hdf/current/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:09,998 - Directory['/var/lib/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:09,998 - Directory['/var/lib/nifi/state/local'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:09,999 - Directory['/usr/hdf/current/nifi/lib'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:10,002 - Directory['/var/lib/nifi/content_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:10,029 - Directory['/var/lib/nifi/content_repository'] {'owner': 'nifi', 'group': 'nifi', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'} 2019-06-03 14:34:10,057 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'} 2019-06-03 14:34:10,060 - File['/etc/security/limits.d/nifi.conf'] {'content': Template('nifi.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} 2019-06-03 14:34:10,061 - PropertiesFile['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'group': 'nifi', 'mode': 0600, 'properties': ...} 2019-06-03 14:34:10,065 - Generating properties file: /usr/hdf/current/nifi/conf/nifi.properties 2019-06-03 14:34:10,065 - File['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600, 'encoding': 'UTF-8'} 2019-06-03 14:34:10,163 - Writing File['/usr/hdf/current/nifi/conf/nifi.properties'] because contents don't match 2019-06-03 14:34:10,168 - File['/usr/hdf/current/nifi/conf/bootstrap.conf'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600} 2019-06-03 14:34:10,169 - Writing File['/usr/hdf/current/nifi/conf/bootstrap.conf'] because contents don't match 2019-06-03 14:34:10,174 - File['/usr/hdf/current/nifi/conf/logback.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400} 2019-06-03 14:34:10,177 - File['/usr/hdf/current/nifi/conf/state-management.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400} 2019-06-03 14:34:10,186 - File['/usr/hdf/current/nifi/conf/authorizers.xml'] {'owner': 'nifi', 'content': ..., 'group': 'nifi', 'mode': 0600} 2019-06-03 14:34:10,191 - File['/usr/hdf/current/nifi/conf/login-identity-providers.xml'] {'owner': 'nifi', 'content': '<loginIdentityProviders>\n \n \n \n \n \n \n\n </loginIdentityProviders>', 'group': 'nifi', 'mode': 0600} 2019-06-03 14:34:10,194 - File['/usr/hdf/current/nifi/bin/nifi-env.sh'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0755} 2019-06-03 14:34:10,197 - File['/usr/hdf/current/nifi/conf/bootstrap-notification-services.xml'] {'owner': 'nifi', 'content': '<services>\n \n \n\n \n\n \n </services>', 'group': 'nifi', 'mode': 0400} 2019-06-03 14:34:10,197 - Using repository toolkit script: /usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh 2019-06-03 14:34:10,197 - File['/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh'] {'mode': 0755} 2019-06-03 14:34:10,197 - Encrypting NiFi sensitive configuration properties 2019-06-03 14:34:10,201 - Execute[(u'/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh', '-v', '-b', u'/usr/hdf/current/nifi/conf/bootstrap.conf', '-n', u'/usr/hdf/current/nifi/conf/nifi.properties', '-f', u'/var/lib/nifi/conf/flow.xml.gz', '-s', [PROTECTED], '-a', u'/usr/hdf/current/nifi/conf/authorizers.xml', '-m', '-e', [PROTECTED], '-p', [PROTECTED])] {'environment': {'JAVA_OPTS': u'-Xms128m -Xmx256m', 'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_112'}, 'logoutput': False, 'user': 'nifi'} 2019-06-03 14:34:12,719 - ModifyPropertiesFile['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'properties': {'#nifi.security.ambari.hash.kspwd': '32b68561e3f9aae8cf3871dc16db6e717759449332926e0cfed0005b5bea282a', '#nifi.security.ambari.hash.kpwd': '32b68561e3f9aae8cf3871dc16db6e717759449332926e0cfed0005b5bea282a', '#nifi.security.ambari.hash.tspwd': '32b68561e3f9aae8cf3871dc16db6e717759449332926e0cfed0005b5bea282a'}} 2019-06-03 14:34:12,719 - Modifying existing properties file: /usr/hdf/current/nifi/conf/nifi.properties 2019-06-03 14:34:12,722 - File['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'content': ..., 'group': None, 'mode': None, 'encoding': 'utf-8'} 2019-06-03 14:34:12,722 - Writing File['/usr/hdf/current/nifi/conf/nifi.properties'] because contents don't match 2019-06-03 14:34:12,725 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'} 2019-06-03 14:34:12,726 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-nifi.json 2019-06-03 14:34:12,726 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-nifi.json'] {'content': Template('input.config-nifi.json.j2'), 'mode': 0644} 2019-06-03 14:34:12,726 - Ranger admin not installed 2019-06-03 14:34:12,726 - Execute['export JAVA_HOME=/usr/jdk64/jdk1.8.0_112;/usr/hdf/current/nifi/bin/nifi.sh start >> /var/log/nifi/nifi-setup.log'] {'user': 'nifi'} 2019-06-03 14:34:15,744 - Execute['sleep 5'] {} 2019-06-03 14:34:20,748 - Trying to register NIFI Registry. This can take up to several minutes. Please, wait... 2019-06-03 14:34:20,748 - Trying to retrieve NIFI clients... 2019-06-03 14:34:20,748 - Using repository toolkit script: /usr/hdf/current/nifi-toolkit/bin/cli.sh 2019-06-03 14:34:20,748 - Using repository toolkit script: /usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh 2019-06-03 14:36:24,932 - Unable to create or update registry client. Please review NiFi to setup the registry manually if necessary. 2019-06-03 14:36:24,932 - Pid files for current script are not defined Command completed successfully! /var/log/nifi/nifi-setup.log 2019-06-03 14:34:06,849 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not currently running Java home: /usr/jdk64/jdk1.8.0_112 NiFi home: /usr/hdf/current/nifi Bootstrap Config File: /usr/hdf/current/nifi/conf/bootstrap.conf ... View more