NiFi Toolkit: Can't start NiFi with a ~75MB gzipped flow.xml

[#1]

Hi, 

There is an issue when having a lot of (17.000 +)  components in NiFi's canvas resulting in a very large template.

When the flow.xml.gz in /var/lib/conf/flow.xml.gz starts to get large we need to increase maximum heap size for the toolkit JVM (Ambari > NiFi > Advanced nifi-toolkit-env > Toolkit java options) in order for NiFi to start. Normally this solves the issue setting it to -Xms1024m -Xmx7168m or any reasonably high maximum value.

The issue is when flow.xml.gz starts to exceed ~70MB, NiFi can't be started even with large amounts of memory (we've tried up to 70GB).

IDK if this is a memory problem (e.g. we need more RAM) or if we've hit a wall of some kind.

Error:

ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: 
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.lang.StringCoding.encode(StringCoding.java:350)
	at java.lang.String.getBytes(String.java:941)
	at org.apache.commons.io.IOUtils.write(IOUtils.java:2025)
	at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
	at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6$_closure30.doCall(ConfigEncryptionTool.groovy:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at groovy.lang.Closure.call(Closure.java:414)
	at groovy.lang.Closure.call(Closure.java:430)
	at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
	at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
	at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6.doCall(ConfigEncryptionTool.groovy:869)
Requested array size exceeds VM limit

After this point NiFi can't start, automatically deletes / creates a new flow.xml.gz and starts with an empty canvas. 

Full log:

Spoiler

stderr:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in <module>
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 143, in start
self.configure(env, is_starting = True)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 111, in configure
self.write_configurations(params, is_starting)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 247, in write_configurations
support_encrypt_authorizers=params.stack_support_encrypt_authorizers
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util_common.py", line 574, in encrypt_sensitive_properties
Execute(encrypt_config_command, user=nifi_user, logoutput=False, environment=environment)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -a /usr/hdf/current/nifi/conf/authorizers.xml -m -e '[PROTECTED]' -p '[PROTECTED]'' returned 255. 
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of login-identity-providers.xml
2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source login-identity-providers.xml and destination login-identity-providers.xml are identical [/usr/hdf/current/nifi/conf/login-identity-providers.xml] so the original will be overwritten
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml
2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz
2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: /usr/hdf/current/nifi/conf/bootstrap.conf
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 161 properties from /usr/hdf/current/nifi/conf/nifi.properties
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 161 properties from /usr/hdf/current/nifi/conf/nifi.properties
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 3 protected properties of 5 sensitive properties (75%)
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 158 properties
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded login identity providers content (14 lines)
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in login-identity-providers.xml
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in login-identity-providers.xml
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded authorizers content (30 lines)
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in authorizers.xml
2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in authorizers.xml
2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Decrypted and re-encrypted 56 elements for flow.xml.gz
2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keyPasswd with aes/gcm/256 -> l3Bk1iJuJf+Y5yyM||Y7jSBqByRUZPmJa2jxf0x2j2s+2efYYoVxp89PeRmCrXbf7M14wUrjKfq4z85RgKTkh/K49m+cZr2fo
2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keyPasswd.protected
2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keystorePasswd with aes/gcm/256 -> qK4aq6CsT54K5a+f||XuhbPv7Dn456Gd0UwEnFb8hMn1np9EhsaZgT8cFCeXGRgRmxJ34VAsOZrc9AmzWen73DYU+3cCQ2YB0
2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keystorePasswd.protected
2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.truststorePasswd with aes/gcm/256 -> CtBw7uibRzIGE0+8||q/aQRM/4uVzMLPga0R/noUj5MpruYl/z5xM62/CUUtskgfyv
2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.truststorePasswd.protected
2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.sensitive.props.key with aes/gcm/256 -> NwNIA9WqScpilGSV||L34l7Lsx5NyEmWjqoGTtaGWDhZ8eo34ADw
2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.sensitive.props.key.protected
2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Final result: 162 keys including 4 protected keys
2020/01/29 14:55:13 ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain:
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.lang.StringCoding.encode(StringCoding.java:350)
at java.lang.String.getBytes(String.java:941)
at org.apache.commons.io.IOUtils.write(IOUtils.java:2025)
at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6$_closure30.doCall(ConfigEncryptionTool.groovy:870)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at groovy.lang.Closure.call(Closure.java:414)
at groovy.lang.Closure.call(Closure.java:430)
at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6.doCall(ConfigEncryptionTool.groovy:869)
Requested array size exceeds VM limit

usage: org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain [-h] [options]

This tool enables easy encryption and decryption of configuration files for NiFi and its sub-projects. Unprotected files can be input to this tool to be
protected by a key in a manner that is understood by NiFi. Protected files, along with a key, can be input to this tool to be unprotected, for troubleshooting
or automation purposes.

-h,--help Show usage information (this message)
--nifiRegistry Specifies to target NiFi Registry. When this flag is not included, NiFi is the target.

When targeting NiFi:
-h,--help Show usage information (this message)
-v,--verbose Sets verbose mode (default false)
-n,--niFiProperties <file> The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)
-o,--outputNiFiProperties <file> The destination nifi.properties file containing protected config values (will not modify input nifi.properties)
-l,--loginIdentityProviders <file> The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is
specified)
-i,--outputLoginIdentityProviders <file> The destination login-identity-providers.xml file containing protected config values (will not modify input
login-identity-providers.xml)
-a,--authorizers <file> The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)
-u,--outputAuthorizers <file> The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
-f,--flowXml <file> The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)
-g,--outputFlowXml <file> The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)
-b,--bootstrapConf <file> The bootstrap.conf file to persist master key
-k,--key <keyhex> The raw hexadecimal key to use to encrypt the sensitive properties
-e,--oldKey <keyhex> The old raw hexadecimal key to use during key migration
-p,--password <password> The password from which to derive the key to use to encrypt the sensitive properties
-w,--oldPassword <password> The old password from which to derive the key during migration
-r,--useRawKey If provided, the secure console will prompt for the raw key value in hexadecimal form
-m,--migrate If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with
a new key
-x,--encryptFlowXmlOnly If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or
login-identity-providers.xml files will not be modified
-s,--propsKey <password|keyhex> The password or key to use to encrypt the sensitive processor properties in flow.xml.gz
-A,--newFlowAlgorithm <algorithm> The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz
-P,--newFlowProvider <algorithm> The security provider to use to encrypt the sensitive processor properties in flow.xml.gz
-c,--translateCli Translates the nifi.properties file to a format suitable for the NiFi CLI tool

When targeting NiFi Registry using the --nifiRegistry flag:
-h,--help Show usage information (this message)
-v,--verbose Sets verbose mode (default false)
-p,--password <password> Protect the files using a password-derived key. If an argument is not provided to this flag, interactive mode will
be triggered to prompt the user to enter the password.
-k,--key <keyhex> Protect the files using a raw hexadecimal key. If an argument is not provided to this flag, interactive mode will be
triggered to prompt the user to enter the key.
--oldPassword <password> If the input files are already protected using a password-derived key, this specifies the old password so that the
files can be unprotected before re-protecting.
--oldKey <keyhex> If the input files are already protected using a key, this specifies the raw hexadecimal key so that the files can
be unprotected before re-protecting.
-b,--bootstrapConf <file> The bootstrap.conf file containing no master key or an existing master key. If a new password or key is specified
(using -p or -k) and no output bootstrap.conf file is specified, then this file will be overwritten to persist the
new master key.
-B,--outputBootstrapConf <file> The destination bootstrap.conf file to persist master key. If specified, the input bootstrap.conf will not be
modified.
-r,--nifiRegistryProperties <file> The nifi-registry.properties file containing unprotected config values, overwritten if no output file specified.
-R,--outputNifiRegistryProperties <file> The destination nifi-registry.properties file containing protected config values.
-a,--authorizersXml <file> The authorizers.xml file containing unprotected config values, overwritten if no output file specified.
-A,--outputAuthorizersXml <file> The destination authorizers.xml file containing protected config values.
-i,--identityProvidersXml <file> The identity-providers.xml file containing unprotected config values, overwritten if no output file specified.
-I,--outputIdentityProvidersXml <file> The destination identity-providers.xml file containing protected config values.
--decrypt Can be used with -r to decrypt a previously encrypted NiFi Registry Properties file. Decrypted content is printed to
STDOUT.
stdout:
2020-01-29 14:54:59,928 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-01-29 14:54:59,944 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2020-01-29 14:55:00,197 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-01-29 14:55:00,202 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2020-01-29 14:55:00,203 - Group['livy'] {}
2020-01-29 14:55:00,204 - Group['spark'] {}
2020-01-29 14:55:00,204 - Group['nifiregistry'] {}
2020-01-29 14:55:00,204 - Group['hdfs'] {}
2020-01-29 14:55:00,204 - Group['zeppelin'] {}
2020-01-29 14:55:00,204 - Group['hadoop'] {}
2020-01-29 14:55:00,204 - Group['nifi'] {}
2020-01-29 14:55:00,205 - Group['users'] {}
2020-01-29 14:55:00,205 - Group['knox'] {}
2020-01-29 14:55:00,205 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-01-29 14:55:00,206 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-01-29 14:55:00,207 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-01-29 14:55:00,208 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-01-29 14:55:00,208 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-01-29 14:55:00,209 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2020-01-29 14:55:00,210 - User['nifiregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifiregistry'], 'uid': None}
2020-01-29 14:55:00,211 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['zeppelin', 'hadoop'], 'uid': None}
2020-01-29 14:55:00,212 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifi'], 'uid': None}
2020-01-29 14:55:00,213 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-01-29 14:55:00,213 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['livy', 'hadoop'], 'uid': None}
2020-01-29 14:55:00,214 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['spark', 'hadoop'], 'uid': None}
2020-01-29 14:55:00,215 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2020-01-29 14:55:00,216 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}
2020-01-29 14:55:00,216 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-01-29 14:55:00,217 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-01-29 14:55:00,218 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'knox'], 'uid': None}
2020-01-29 14:55:00,218 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2020-01-29 14:55:00,220 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2020-01-29 14:55:00,225 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2020-01-29 14:55:00,226 - Group['hdfs'] {}
2020-01-29 14:55:00,226 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}
2020-01-29 14:55:00,226 - FS Type: HDFS
2020-01-29 14:55:00,226 - Directory['/etc/hadoop'] {'mode': 0755}
2020-01-29 14:55:00,240 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}
2020-01-29 14:55:00,240 - Writing File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] because contents don't match
2020-01-29 14:55:00,241 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2020-01-29 14:55:00,256 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2020-01-29 14:55:00,263 - Skipping Execute[('setenforce', '0')] due to not_if
2020-01-29 14:55:00,264 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2020-01-29 14:55:00,266 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2020-01-29 14:55:00,266 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2020-01-29 14:55:00,267 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2020-01-29 14:55:00,270 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
2020-01-29 14:55:00,271 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'}
2020-01-29 14:55:00,277 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2020-01-29 14:55:00,286 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2020-01-29 14:55:00,286 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2020-01-29 14:55:00,287 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2020-01-29 14:55:00,290 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2020-01-29 14:55:00,294 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2020-01-29 14:55:00,297 - Skipping unlimited key JCE policy check and setup since the Java VM is not managed by Ambari
2020-01-29 14:55:00,304 - Skipping stack-select on NIFI because it does not exist in the stack-select package structure.
2020-01-29 14:55:00,647 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2020-01-29 14:55:00,704 - File['/var/lib/ambari-agent/tmp/run_ca.sh'] {'owner': 'nifi', 'content': StaticFile('run_ca.sh'), 'group': 'nifi', 'mode': 0755}
2020-01-29 14:55:00,706 - Changing owner of package files
2020-01-29 14:55:00,706 - Directory['/usr/hdf/current/nifi-toolkit'] {'group': 'nifi', 'cd_access': 'a', 'recursion_follow_links': True, 'create_parents': False, 'recursive_ownership': True, 'owner': 'nifi', 'mode': 0755}
2020-01-29 14:55:00,710 - Directory['/var/run/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,711 - Directory['/var/lib/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,807 - Directory['/var/lib/nifi/database_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,807 - Directory['/hadoopfs/fs1/nifi/flowfile_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,808 - Directory['/hadoopfs/fs1/nifi/provenance_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,810 - Directory['/usr/hdf/current/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,812 - Directory['/var/lib/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,812 - Directory['/var/lib/nifi/state/local'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,814 - Directory['/usr/hdf/current/nifi/lib'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,817 - Directory['/hadoopfs/fs1/nifi/content_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,896 - Directory['/hadoopfs/fs1/nifi/content_repository'] {'owner': 'nifi', 'group': 'nifi', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}
2020-01-29 14:55:00,962 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2020-01-29 14:55:00,966 - File['/etc/security/limits.d/nifi.conf'] {'content': Template('nifi.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2020-01-29 14:55:00,968 - PropertiesFile['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'group': 'nifi', 'mode': 0600, 'properties': ...}
2020-01-29 14:55:00,973 - Generating properties file: /usr/hdf/current/nifi/conf/nifi.properties
2020-01-29 14:55:00,973 - File['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600, 'encoding': 'UTF-8'}
2020-01-29 14:55:01,070 - Writing File['/usr/hdf/current/nifi/conf/nifi.properties'] because contents don't match
2020-01-29 14:55:01,074 - File['/usr/hdf/current/nifi/conf/bootstrap.conf'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600}
2020-01-29 14:55:01,075 - Writing File['/usr/hdf/current/nifi/conf/bootstrap.conf'] because contents don't match
2020-01-29 14:55:01,078 - File['/usr/hdf/current/nifi/conf/logback.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}
2020-01-29 14:55:01,081 - File['/usr/hdf/current/nifi/conf/state-management.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}
2020-01-29 14:55:01,088 - File['/usr/hdf/current/nifi/conf/authorizers.xml'] {'owner': 'nifi', 'content': ..., 'group': 'nifi', 'mode': 0600}
2020-01-29 14:55:01,093 - File['/usr/hdf/current/nifi/conf/login-identity-providers.xml'] {'owner': 'nifi', 'content': '<loginIdentityProviders>\n \n \n \n \n <provider>\n <identifier>kerberos-provider</identifier>\n <class>org.apache.nifi.kerberos.KerberosProvider</class>\n <property name="Default Realm">DHUB-FP.COM</property>\n <property name="Authentication Expiration">12 hours</property>\n </provider>\n \n\n </loginIdentityProviders>', 'group': 'nifi', 'mode': 0600}
2020-01-29 14:55:01,095 - File['/usr/hdf/current/nifi/bin/nifi-env.sh'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0755}
2020-01-29 14:55:01,097 - File['/usr/hdf/current/nifi/conf/bootstrap-notification-services.xml'] {'owner': 'nifi', 'content': '<services>\n \n \n\n \n\n \n </services>', 'group': 'nifi', 'mode': 0400}
2020-01-29 14:55:01,099 - File['/usr/hdf/current/nifi/conf/nifi_jaas.conf'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}
2020-01-29 14:55:01,099 - Using repository toolkit script: /usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh
2020-01-29 14:55:01,099 - File['/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh'] {'mode': 0755}
2020-01-29 14:55:01,099 - Encrypting NiFi sensitive configuration properties
2020-01-29 14:55:01,166 - Execute[('/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh', '-v', '-b', u'/usr/hdf/current/nifi/conf/bootstrap.conf', '-n', u'/usr/hdf/current/nifi/conf/nifi.properties', '-f', u'/var/lib/nifi/conf/flow.xml.gz', '-s', [PROTECTED], '-l', u'/usr/hdf/current/nifi/conf/login-identity-providers.xml', '-a', u'/usr/hdf/current/nifi/conf/authorizers.xml', '-m', '-e', [PROTECTED], '-p', [PROTECTED])] {'environment': {'JAVA_OPTS': u'-Xms71680m -Xmx71680m', 'JAVA_HOME': u'/usr/lib/jvm/java'}, 'logoutput': False, 'user': 'nifi'}
2020-01-29 14:55:14,348 - Skipping stack-select on NIFI because it does not exist in the stack-select package structure.

Command failed after 1 tries

NiFi version: 1.9.0

Ambari Version: 2.7.3.0

Java Version:

openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

OS version: centos-release-7-6.1810.2.el7.centos.x86_64

Thank you in advance for any tips in the right direction. 

[#2 · Accepted solution]

@DavidR 

That is one extremely large flow.xml.gz.  I would guess that once uncompressed it falls in the ~1.5 GB size range.  My guess is that you have a large number of templates loaded in your NiFi.

The Encrypt-Config toolkit serializes the flow.xml.gz in memory in order to re-encrypt the sensitive properties.
The "Requested array size exceeds VM limit" is telling you that the action taken has created an array that exceeds the max JVM array size of 2^31-1 elements.  Increasing the JVM heap size will not change this max supportable array size.

NiFi then ends up with no flow.xml.gz because as a result of the failed encrypt config toolkiit operation, no re-encrypted flow.xml.gz was produced as output.   So on next restart attempt of NiFi service, it starts fine and creates a blank flow.xml.gz since one did not exist.

This is documented in this bug: https://issues.apache.org/jira/browse/NIFI-6999

Workaround for this issue:
1. Start your NiFi nodes from command line rather then via Ambari.  ( <path to nifi>/bin/nifi.sh start )
2. Once NiFi cluster is back up and running, download and then delete templates stored in NiFi.   Access the global menu (upper right corner) --> Templates.  This opens a list of all stored NiFi templates.  One by one download (optional) and delete each template.  Removing stored templates does not affect anything instantiated to the canvas.

These actions should greatly reduce the size of your flow.xml.gz.  Then try restarting NiFi via Ambari again.

Hope this helps,

Matt

[#3 · Accepted solution]

@DavidR 

That is one extremely large flow.xml.gz.  I would guess that once uncompressed it falls in the ~1.5 GB size range.  My guess is that you have a large number of templates loaded in your NiFi.

The Encrypt-Config toolkit serializes the flow.xml.gz in memory in order to re-encrypt the sensitive properties.
The "Requested array size exceeds VM limit" is telling you that the action taken has created an array that exceeds the max JVM array size of 2^31-1 elements.  Increasing the JVM heap size will not change this max supportable array size.

NiFi then ends up with no flow.xml.gz because as a result of the failed encrypt config toolkiit operation, no re-encrypted flow.xml.gz was produced as output.   So on next restart attempt of NiFi service, it starts fine and creates a blank flow.xml.gz since one did not exist.

This is documented in this bug: https://issues.apache.org/jira/browse/NIFI-6999

Workaround for this issue:
1. Start your NiFi nodes from command line rather then via Ambari.  ( <path to nifi>/bin/nifi.sh start )
2. Once NiFi cluster is back up and running, download and then delete templates stored in NiFi.   Access the global menu (upper right corner) --> Templates.  This opens a list of all stored NiFi templates.  One by one download (optional) and delete each template.  Removing stored templates does not affect anything instantiated to the canvas.

These actions should greatly reduce the size of your flow.xml.gz.  Then try restarting NiFi via Ambari again.

Hope this helps,

Matt

[#4]

Hi @MattWho 

Thank you so much for the clarification, pointing us in the right direction and for the proposed workaround. We're going to try that and come to you with feedback. Your contribution to this community is invaluable. 

BR,

David

[#5]

@MattWho 

Sorry for the delay.

Workaround for this issue:
1. Start your NiFi nodes from command line rather then via Ambari.  ( <path to nifi>/bin/nifi.sh start )

When I try to start this flow in a separate testing cluster using <path to nifi>/bin/nifi.sh start I get:

ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.IllegalArgumentException: There was an issue decrypting protected properties
java.lang.IllegalArgumentException: There was an issue decrypting protected properties
at org.apache.nifi.NiFi.initializeProperties(NiFi.java:337)
at org.apache.nifi.NiFi.convertArgumentsToValidatedNiFiProperties(NiFi.java:305)
at org.apache.nifi.NiFi.main(NiFi.java:296)
Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException: The provider factory cannot generate providers without a key
at org.apache.nifi.properties.AESSensitivePropertyProviderFactory.getProvider(AESSensitivePropertyProviderFactory.java:40)
at org.apache.nifi.properties.NiFiPropertiesLoader.getSensitivePropertyProvider(NiFiPropertiesLoader.java:190)
at org.apache.nifi.properties.NiFiPropertiesLoader.load(NiFiPropertiesLoader.java:249)
at org.apache.nifi.properties.NiFiPropertiesLoader.load(NiFiPropertiesLoader.java:266)
at org.apache.nifi.properties.NiFiPropertiesLoader.loadDefault(NiFiPropertiesLoader.java:173)
at org.apache.nifi.properties.NiFiPropertiesLoader.get(NiFiPropertiesLoader.java:284)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.NiFi.initializeProperties(NiFi.java:332)
... 2 common frames omitted

Is this caused by trying to launch the flow. in another environment or am I supposed to configure something before trying to launch nifi this way?

Thanks in advance once again,

David Resende

[#6 · Accepted solution]

@DavidR 

NiFi requires that a sensitive properties key has been configured.  The Sensitive properties key is used to encrypt all passwords written to the flow.xml.gz file.

The flow.xml.gz is not portable between different installations of NiFi unless at least 1 of the 2 following is true:

1. Both NiFi installations are using the exact same sensitive properties key in the nif.properties file:

nifi.sensitive.props.key=

All the sensitive properties (passwords) entered in to components configured in your dataflows contained within the flow.xml.gz are encrypted.  NiFi will fail to start if it cannot decrypt these sensitive properties as the flow.xml.gz is uncompressed and loaded in to memory.

2. The  flow.xml.gz contains no encrypted sensitive properties.  If you do not know the sensitive properties key used on NiFi where the flow.xml.gz was obtained, you can uncompress the flow.xml.gz yourself and remove all occurrences of "enc{.*}".  For example:

<value>enc{2fecf4bbb0456fd10c088f73aab2b3c3e92b532afa46eb042f0b0a14b06b1b60}</value>

would be replaced with:

<value></value>

Then gzip the flow.xml again. This removes all passwords from your flow.xml.gz will allowing it to be loaded by any new NiFi installation.  You will then need to go through your components and set the passwords again which will then get encrypted to the flow.xml.gz using the sensitive props key set on that NiFi.

Hope this helps,

Matt

[#7]

@MattWho Thank you once again for the information posted here. It solved our problem. 

I was able to run the flow in another cluster using nifi.sh instead of running it with ambari (and the encrypt tool it uses).

copying nifi.sensitive.props.key in /etc/nifi/conf/nifi.properties from source cluster did the trick. 

BR,

David