NiFi Toolkit: Can't start NiFi with a ~75MB gzipped flow.xml

Question

Hi, There is an issue when having a lot of (17.000 +) components in NiFi's canvas resulting in a very large template. When the flow.xml.gz in /var/lib/conf/flow.xml.gz starts to get large we need to increase maximum heap size for the toolkit JVM (Ambari > NiFi > Advanced nifi-toolkit-env > Toolkit java options) in order for NiFi to start. Normally this solves the issue setting it to -Xms1024m -Xmx7168m or any reasonably high maximum value. The issue is when flow.xml.gz starts to exceed ~70MB, NiFi can't be started even with large amounts of memory (we've tried up to 70GB). IDK if this is a memory problem (e.g. we need more RAM) or if we've hit a wall of some kind. Error: ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.lang.StringCoding.encode(StringCoding.java:350) at java.lang.String.getBytes(String.java:941) at org.apache.commons.io.IOUtils.write(IOUtils.java:2025) at org.apache.commons.io.IOUtils$write$0.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141) at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6$_closure30.doCall(ConfigEncryptionTool.groovy:870) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at groovy.lang.Closure.call(Closure.java:414) at groovy.lang.Closure.call(Closure.java:430) at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622) at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54) at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6.doCall(ConfigEncryptionTool.groovy:869)Requested array size exceeds VM limit After this point NiFi can't start, automatically deletes / creates a new flow.xml.gz and starts with an empty canvas. Full log: Spoiler (Highlight to read) stderr:Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in Master().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 143, in startself.configure(env, is_starting = True)File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 111, in configureself.write_configurations(params, is_starting)File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 247, in write_configurationssupport_encrypt_authorizers=params.stack_support_encrypt_authorizersFile "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util_common.py", line 574, in encrypt_sensitive_propertiesExecute(encrypt_config_command, user=nifi_user, logoutput=False, environment=environment)File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__self.env.run()File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in runself.run_action(resource, action)File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_actionprovider_action()File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_runreturns=self.resource.returns)File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in innerresult = function(command, **kwargs)File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_calltries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapperresult = _call(command, **kwargs_copy)File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _callraise ExecutionFailed(err_msg, code, out, err)resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -a /usr/hdf/current/nifi/conf/authorizers.xml -m -e '[PROTECTED]' -p '[PROTECTED]'' returned 255. 2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of login-identity-providers.xml2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source login-identity-providers.xml and destination login-identity-providers.xml are identical [/usr/hdf/current/nifi/conf/login-identity-providers.xml] so the original will be overwritten2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: /usr/hdf/current/nifi/conf/bootstrap.conf2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 161 properties from /usr/hdf/current/nifi/conf/nifi.properties2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 161 properties from /usr/hdf/current/nifi/conf/nifi.properties2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 3 protected properties of 5 sensitive properties (75%)2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 158 properties2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded login identity providers content (14 lines)2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in login-identity-providers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in login-identity-providers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded authorizers content (30 lines)2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in authorizers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in authorizers.xml2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Decrypted and re-encrypted 56 elements for flow.xml.gz2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keyPasswd with aes/gcm/256 -> l3Bk1iJuJf+Y5yyM||Y7jSBqByRUZPmJa2jxf0x2j2s+2efYYoVxp89PeRmCrXbf7M14wUrjKfq4z85RgKTkh/K49m+cZr2fo2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keyPasswd.protected2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keystorePasswd with aes/gcm/256 -> qK4aq6CsT54K5a+f||XuhbPv7Dn456Gd0UwEnFb8hMn1np9EhsaZgT8cFCeXGRgRmxJ34VAsOZrc9AmzWen73DYU+3cCQ2YB02020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keystorePasswd.protected2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.truststorePasswd with aes/gcm/256 -> CtBw7uibRzIGE0+8||q/aQRM/4uVzMLPga0R/noUj5MpruYl/z5xM62/CUUtskgfyv2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.truststorePasswd.protected2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.sensitive.props.key with aes/gcm/256 -> NwNIA9WqScpilGSV||L34l7Lsx5NyEmWjqoGTtaGWDhZ8eo34ADw2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.sensitive.props.key.protected2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Final result: 162 keys including 4 protected keys2020/01/29 14:55:13 ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain:java.lang.OutOfMemoryError: Requested array size exceeds VM limitat java.lang.StringCoding.encode(StringCoding.java:350)at java.lang.String.getBytes(String.java:941)at org.apache.commons.io.IOUtils.write(IOUtils.java:2025)at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6$_closure30.doCall(ConfigEncryptionTool.groovy:870)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)at groovy.lang.Closure.call(Closure.java:414)at groovy.lang.Closure.call(Closure.java:430)at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6.doCall(ConfigEncryptionTool.groovy:869)Requested array size exceeds VM limit usage: org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain [-h] [options] This tool enables easy encryption and decryption of configuration files for NiFi and its sub-projects. Unprotected files can be input to this tool to beprotected by a key in a manner that is understood by NiFi. Protected files, along with a key, can be input to this tool to be unprotected, for troubleshootingor automation purposes. -h,--help Show usage information (this message)--nifiRegistry Specifies to target NiFi Registry. When this flag is not included, NiFi is the target. When targeting NiFi:-h,--help Show usage information (this message)-v,--verbose Sets verbose mode (default false)-n,--niFiProperties The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)-o,--outputNiFiProperties The destination nifi.properties file containing protected config values (will not modify input nifi.properties)-l,--loginIdentityProviders The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i isspecified)-i,--outputLoginIdentityProviders The destination login-identity-providers.xml file containing protected config values (will not modify inputlogin-identity-providers.xml)-a,--authorizers The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)-u,--outputAuthorizers The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)-f,--flowXml The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)-g,--outputFlowXml The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)-b,--bootstrapConf The bootstrap.conf file to persist master key-k,--key The raw hexadecimal key to use to encrypt the sensitive properties-e,--oldKey The old raw hexadecimal key to use during key migration-p,--password The password from which to derive the key to use to encrypt the sensitive properties-w,--oldPassword The old password from which to derive the key during migration-r,--useRawKey If provided, the secure console will prompt for the raw key value in hexadecimal form-m,--migrate If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted witha new key-x,--encryptFlowXmlOnly If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/orlogin-identity-providers.xml files will not be modified-s,--propsKey The password or key to use to encrypt the sensitive processor properties in flow.xml.gz-A,--newFlowAlgorithm The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz-P,--newFlowProvider The security provider to use to encrypt the sensitive processor properties in flow.xml.gz-c,--translateCli Translates the nifi.properties file to a format suitable for the NiFi CLI tool When targeting NiFi Registry using the --nifiRegistry flag:-h,--help Show usage information (this message)-v,--verbose Sets verbose mode (default false)-p,--password Protect the files using a password-derived key. If an argument is not provided to this flag, interactive mode willbe triggered to prompt the user to enter the password.-k,--key Protect the files using a raw hexadecimal key. If an argument is not provided to this flag, interactive mode will betriggered to prompt the user to enter the key.--oldPassword If the input files are already protected using a password-derived key, this specifies the old password so that thefiles can be unprotected before re-protecting.--oldKey If the input files are already protected using a key, this specifies the raw hexadecimal key so that the files canbe unprotected before re-protecting.-b,--bootstrapConf The bootstrap.conf file containing no master key or an existing master key. If a new password or key is specified(using -p or -k) and no output bootstrap.conf file is specified, then this file will be overwritten to persist thenew master key.-B,--outputBootstrapConf The destination bootstrap.conf file to persist master key. If specified, the input bootstrap.conf will not bemodified.-r,--nifiRegistryProperties The nifi-registry.properties file containing unprotected config values, overwritten if no output file specified.-R,--outputNifiRegistryProperties The destination nifi-registry.properties file containing protected config values.-a,--authorizersXml The authorizers.xml file containing unprotected config values, overwritten if no output file specified.-A,--outputAuthorizersXml The destination authorizers.xml file containing protected config values.-i,--identityProvidersXml The identity-providers.xml file containing unprotected config values, overwritten if no output file specified.-I,--outputIdentityProvidersXml The destination identity-providers.xml file containing protected config values.--decrypt Can be used with -r to decrypt a previously encrypted NiFi Registry Properties file. Decrypted content is printed toSTDOUT.stdout:2020-01-29 14:54:59,928 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-782020-01-29 14:54:59,944 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf2020-01-29 14:55:00,197 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-782020-01-29 14:55:00,202 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf2020-01-29 14:55:00,203 - Group['livy'] {}2020-01-29 14:55:00,204 - Group['spark'] {}2020-01-29 14:55:00,204 - Group['nifiregistry'] {}2020-01-29 14:55:00,204 - Group['hdfs'] {}2020-01-29 14:55:00,204 - Group['zeppelin'] {}2020-01-29 14:55:00,204 - Group['hadoop'] {}2020-01-29 14:55:00,204 - Group['nifi'] {}2020-01-29 14:55:00,205 - Group['users'] {}2020-01-29 14:55:00,205 - Group['knox'] {}2020-01-29 14:55:00,205 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,206 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,207 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,208 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,208 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,209 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}2020-01-29 14:55:00,210 - User['nifiregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifiregistry'], 'uid': None}2020-01-29 14:55:00,211 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['zeppelin', 'hadoop'], 'uid': None}2020-01-29 14:55:00,212 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifi'], 'uid': None}2020-01-29 14:55:00,213 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,213 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['livy', 'hadoop'], 'uid': None}2020-01-29 14:55:00,214 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['spark', 'hadoop'], 'uid': None}2020-01-29 14:55:00,215 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}2020-01-29 14:55:00,216 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}2020-01-29 14:55:00,216 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,217 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,218 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'knox'], 'uid': None}2020-01-29 14:55:00,218 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}2020-01-29 14:55:00,220 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}2020-01-29 14:55:00,225 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if2020-01-29 14:55:00,226 - Group['hdfs'] {}2020-01-29 14:55:00,226 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}2020-01-29 14:55:00,226 - FS Type: HDFS2020-01-29 14:55:00,226 - Directory['/etc/hadoop'] {'mode': 0755}2020-01-29 14:55:00,240 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}2020-01-29 14:55:00,240 - Writing File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] because contents don't match2020-01-29 14:55:00,241 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}2020-01-29 14:55:00,256 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}2020-01-29 14:55:00,263 - Skipping Execute[('setenforce', '0')] due to not_if2020-01-29 14:55:00,264 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}2020-01-29 14:55:00,266 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}2020-01-29 14:55:00,266 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}2020-01-29 14:55:00,267 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}2020-01-29 14:55:00,270 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}2020-01-29 14:55:00,271 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'}2020-01-29 14:55:00,277 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}2020-01-29 14:55:00,286 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}2020-01-29 14:55:00,286 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}2020-01-29 14:55:00,287 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}2020-01-29 14:55:00,290 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}2020-01-29 14:55:00,294 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}2020-01-29 14:55:00,297 - Skipping unlimited key JCE policy check and setup since the Java VM is not managed by Ambari2020-01-29 14:55:00,304 - Skipping stack-select on NIFI because it does not exist in the stack-select package structure.2020-01-29 14:55:00,647 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-782020-01-29 14:55:00,704 - File['/var/lib/ambari-agent/tmp/run_ca.sh'] {'owner': 'nifi', 'content': StaticFile('run_ca.sh'), 'group': 'nifi', 'mode': 0755}2020-01-29 14:55:00,706 - Changing owner of package files2020-01-29 14:55:00,706 - Directory['/usr/hdf/current/nifi-toolkit'] {'group': 'nifi', 'cd_access': 'a', 'recursion_follow_links': True, 'create_parents': False, 'recursive_ownership': True, 'owner': 'nifi', 'mode': 0755}2020-01-29 14:55:00,710 - Directory['/var/run/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,711 - Directory['/var/lib/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,807 - Directory['/var/lib/nifi/database_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,807 - Directory['/hadoopfs/fs1/nifi/flowfile_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,808 - Directory['/hadoopfs/fs1/nifi/provenance_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,810 - Directory['/usr/hdf/current/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,812 - Directory['/var/lib/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,812 - Directory['/var/lib/nifi/state/local'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,814 - Directory['/usr/hdf/current/nifi/lib'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,817 - Directory['/hadoopfs/fs1/nifi/content_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,896 - Directory['/hadoopfs/fs1/nifi/content_repository'] {'owner': 'nifi', 'group': 'nifi', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,962 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}2020-01-29 14:55:00,966 - File['/etc/security/limits.d/nifi.conf'] {'content': Template('nifi.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}2020-01-29 14:55:00,968 - PropertiesFile['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'group': 'nifi', 'mode': 0600, 'properties': ...}2020-01-29 14:55:00,973 - Generating properties file: /usr/hdf/current/nifi/conf/nifi.properties2020-01-29 14:55:00,973 - File['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600, 'encoding': 'UTF-8'}2020-01-29 14:55:01,070 - Writing File['/usr/hdf/current/nifi/conf/nifi.properties'] because contents don't match2020-01-29 14:55:01,074 - File['/usr/hdf/current/nifi/conf/bootstrap.conf'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600}2020-01-29 14:55:01,075 - Writing File['/usr/hdf/current/nifi/conf/bootstrap.conf'] because contents don't match2020-01-29 14:55:01,078 - File['/usr/hdf/current/nifi/conf/logback.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}2020-01-29 14:55:01,081 - File['/usr/hdf/current/nifi/conf/state-management.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}2020-01-29 14:55:01,088 - File['/usr/hdf/current/nifi/conf/authorizers.xml'] {'owner': 'nifi', 'content': ..., 'group': 'nifi', 'mode': 0600}2020-01-29 14:55:01,093 - File['/usr/hdf/current/nifi/conf/login-identity-providers.xml'] {'owner': 'nifi', 'content': ' kerberos-provider org.apache.nifi.kerberos.KerberosProvider DHUB-FP.COM 12 hours ', 'group': 'nifi', 'mode': 0600}2020-01-29 14:55:01,095 - File['/usr/hdf/current/nifi/bin/nifi-env.sh'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0755}2020-01-29 14:55:01,097 - File['/usr/hdf/current/nifi/conf/bootstrap-notification-services.xml'] {'owner': 'nifi', 'content': ' ', 'group': 'nifi', 'mode': 0400}2020-01-29 14:55:01,099 - File['/usr/hdf/current/nifi/conf/nifi_jaas.conf'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}2020-01-29 14:55:01,099 - Using repository toolkit script: /usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh2020-01-29 14:55:01,099 - File['/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh'] {'mode': 0755}2020-01-29 14:55:01,099 - Encrypting NiFi sensitive configuration properties2020-01-29 14:55:01,166 - Execute[('/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh', '-v', '-b', u'/usr/hdf/current/nifi/conf/bootstrap.conf', '-n', u'/usr/hdf/current/nifi/conf/nifi.properties', '-f', u'/var/lib/nifi/conf/flow.xml.gz', '-s', [PROTECTED], '-l', u'/usr/hdf/current/nifi/conf/login-identity-providers.xml', '-a', u'/usr/hdf/current/nifi/conf/authorizers.xml', '-m', '-e', [PROTECTED], '-p', [PROTECTED])] {'environment': {'JAVA_OPTS': u'-Xms71680m -Xmx71680m', 'JAVA_HOME': u'/usr/lib/jvm/java'}, 'logoutput': False, 'user': 'nifi'}2020-01-29 14:55:14,348 - Skipping stack-select on NIFI because it does not exist in the stack-select package structure. Command failed after 1 tries stderr:Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in Master().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 143, in startself.configure(env, is_starting = True)File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 111, in configureself.write_configurations(params, is_starting)File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 247, in write_configurationssupport_encrypt_authorizers=params.stack_support_encrypt_authorizersFile "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util_common.py", line 574, in encrypt_sensitive_propertiesExecute(encrypt_config_command, user=nifi_user, logoutput=False, environment=environment)File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__self.env.run()File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in runself.run_action(resource, action)File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_actionprovider_action()File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_runreturns=self.resource.returns)File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in innerresult = function(command, **kwargs)File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_calltries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapperresult = _call(command, **kwargs_copy)File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _callraise ExecutionFailed(err_msg, code, out, err)resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -a /usr/hdf/current/nifi/conf/authorizers.xml -m -e '[PROTECTED]' -p '[PROTECTED]'' returned 255. 2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of login-identity-providers.xml2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source login-identity-providers.xml and destination login-identity-providers.xml are identical [/usr/hdf/current/nifi/conf/login-identity-providers.xml] so the original will be overwritten2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz2020/01/29 14:55:02 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: /usr/hdf/current/nifi/conf/bootstrap.conf2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 161 properties from /usr/hdf/current/nifi/conf/nifi.properties2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 161 properties from /usr/hdf/current/nifi/conf/nifi.properties2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 3 protected properties of 5 sensitive properties (75%)2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 158 properties2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded login identity providers content (14 lines)2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in login-identity-providers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in login-identity-providers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded authorizers content (30 lines)2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in authorizers.xml2020/01/29 14:55:02 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in authorizers.xml2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Decrypted and re-encrypted 56 elements for flow.xml.gz2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keyPasswd with aes/gcm/256 -> l3Bk1iJuJf+Y5yyM||Y7jSBqByRUZPmJa2jxf0x2j2s+2efYYoVxp89PeRmCrXbf7M14wUrjKfq4z85RgKTkh/K49m+cZr2fo2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keyPasswd.protected2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.keystorePasswd with aes/gcm/256 -> qK4aq6CsT54K5a+f||XuhbPv7Dn456Gd0UwEnFb8hMn1np9EhsaZgT8cFCeXGRgRmxJ34VAsOZrc9AmzWen73DYU+3cCQ2YB02020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.keystorePasswd.protected2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.security.truststorePasswd with aes/gcm/256 -> CtBw7uibRzIGE0+8||q/aQRM/4uVzMLPga0R/noUj5MpruYl/z5xM62/CUUtskgfyv2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.security.truststorePasswd.protected2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Protected nifi.sensitive.props.key with aes/gcm/256 -> NwNIA9WqScpilGSV||L34l7Lsx5NyEmWjqoGTtaGWDhZ8eo34ADw2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated protection key nifi.sensitive.props.key.protected2020/01/29 14:55:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Final result: 162 keys including 4 protected keys2020/01/29 14:55:13 ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain:java.lang.OutOfMemoryError: Requested array size exceeds VM limitat java.lang.StringCoding.encode(StringCoding.java:350)at java.lang.String.getBytes(String.java:941)at org.apache.commons.io.IOUtils.write(IOUtils.java:2025)at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6$_closure30.doCall(ConfigEncryptionTool.groovy:870)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)at groovy.lang.Closure.call(Closure.java:414)at groovy.lang.Closure.call(Closure.java:430)at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6.doCall(ConfigEncryptionTool.groovy:869)Requested array size exceeds VM limit usage: org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain [-h] [options] This tool enables easy encryption and decryption of configuration files for NiFi and its sub-projects. Unprotected files can be input to this tool to beprotected by a key in a manner that is understood by NiFi. Protected files, along with a key, can be input to this tool to be unprotected, for troubleshootingor automation purposes. -h,--help Show usage information (this message)--nifiRegistry Specifies to target NiFi Registry. When this flag is not included, NiFi is the target. When targeting NiFi:-h,--help Show usage information (this message)-v,--verbose Sets verbose mode (default false)-n,--niFiProperties The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)-o,--outputNiFiProperties The destination nifi.properties file containing protected config values (will not modify input nifi.properties)-l,--loginIdentityProviders The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i isspecified)-i,--outputLoginIdentityProviders The destination login-identity-providers.xml file containing protected config values (will not modify inputlogin-identity-providers.xml)-a,--authorizers The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)-u,--outputAuthorizers The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)-f,--flowXml The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)-g,--outputFlowXml The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)-b,--bootstrapConf The bootstrap.conf file to persist master key-k,--key The raw hexadecimal key to use to encrypt the sensitive properties-e,--oldKey The old raw hexadecimal key to use during key migration-p,--password The password from which to derive the key to use to encrypt the sensitive properties-w,--oldPassword The old password from which to derive the key during migration-r,--useRawKey If provided, the secure console will prompt for the raw key value in hexadecimal form-m,--migrate If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted witha new key-x,--encryptFlowXmlOnly If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/orlogin-identity-providers.xml files will not be modified-s,--propsKey The password or key to use to encrypt the sensitive processor properties in flow.xml.gz-A,--newFlowAlgorithm The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz-P,--newFlowProvider The security provider to use to encrypt the sensitive processor properties in flow.xml.gz-c,--translateCli Translates the nifi.properties file to a format suitable for the NiFi CLI tool When targeting NiFi Registry using the --nifiRegistry flag:-h,--help Show usage information (this message)-v,--verbose Sets verbose mode (default false)-p,--password Protect the files using a password-derived key. If an argument is not provided to this flag, interactive mode willbe triggered to prompt the user to enter the password.-k,--key Protect the files using a raw hexadecimal key. If an argument is not provided to this flag, interactive mode will betriggered to prompt the user to enter the key.--oldPassword If the input files are already protected using a password-derived key, this specifies the old password so that thefiles can be unprotected before re-protecting.--oldKey If the input files are already protected using a key, this specifies the raw hexadecimal key so that the files canbe unprotected before re-protecting.-b,--bootstrapConf The bootstrap.conf file containing no master key or an existing master key. If a new password or key is specified(using -p or -k) and no output bootstrap.conf file is specified, then this file will be overwritten to persist thenew master key.-B,--outputBootstrapConf The destination bootstrap.conf file to persist master key. If specified, the input bootstrap.conf will not bemodified.-r,--nifiRegistryProperties The nifi-registry.properties file containing unprotected config values, overwritten if no output file specified.-R,--outputNifiRegistryProperties The destination nifi-registry.properties file containing protected config values.-a,--authorizersXml The authorizers.xml file containing unprotected config values, overwritten if no output file specified.-A,--outputAuthorizersXml The destination authorizers.xml file containing protected config values.-i,--identityProvidersXml The identity-providers.xml file containing unprotected config values, overwritten if no output file specified.-I,--outputIdentityProvidersXml The destination identity-providers.xml file containing protected config values.--decrypt Can be used with -r to decrypt a previously encrypted NiFi Registry Properties file. Decrypted content is printed toSTDOUT.stdout:2020-01-29 14:54:59,928 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-782020-01-29 14:54:59,944 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf2020-01-29 14:55:00,197 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-782020-01-29 14:55:00,202 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf2020-01-29 14:55:00,203 - Group['livy'] {}2020-01-29 14:55:00,204 - Group['spark'] {}2020-01-29 14:55:00,204 - Group['nifiregistry'] {}2020-01-29 14:55:00,204 - Group['hdfs'] {}2020-01-29 14:55:00,204 - Group['zeppelin'] {}2020-01-29 14:55:00,204 - Group['hadoop'] {}2020-01-29 14:55:00,204 - Group['nifi'] {}2020-01-29 14:55:00,205 - Group['users'] {}2020-01-29 14:55:00,205 - Group['knox'] {}2020-01-29 14:55:00,205 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,206 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,207 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,208 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,208 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,209 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}2020-01-29 14:55:00,210 - User['nifiregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifiregistry'], 'uid': None}2020-01-29 14:55:00,211 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['zeppelin', 'hadoop'], 'uid': None}2020-01-29 14:55:00,212 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifi'], 'uid': None}2020-01-29 14:55:00,213 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,213 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['livy', 'hadoop'], 'uid': None}2020-01-29 14:55:00,214 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['spark', 'hadoop'], 'uid': None}2020-01-29 14:55:00,215 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}2020-01-29 14:55:00,216 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}2020-01-29 14:55:00,216 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,217 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}2020-01-29 14:55:00,218 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'knox'], 'uid': None}2020-01-29 14:55:00,218 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}2020-01-29 14:55:00,220 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}2020-01-29 14:55:00,225 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if2020-01-29 14:55:00,226 - Group['hdfs'] {}2020-01-29 14:55:00,226 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}2020-01-29 14:55:00,226 - FS Type: HDFS2020-01-29 14:55:00,226 - Directory['/etc/hadoop'] {'mode': 0755}2020-01-29 14:55:00,240 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}2020-01-29 14:55:00,240 - Writing File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-env.sh'] because contents don't match2020-01-29 14:55:00,241 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}2020-01-29 14:55:00,256 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}2020-01-29 14:55:00,263 - Skipping Execute[('setenforce', '0')] due to not_if2020-01-29 14:55:00,264 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}2020-01-29 14:55:00,266 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}2020-01-29 14:55:00,266 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}2020-01-29 14:55:00,267 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}2020-01-29 14:55:00,270 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}2020-01-29 14:55:00,271 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'}2020-01-29 14:55:00,277 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}2020-01-29 14:55:00,286 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}2020-01-29 14:55:00,286 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}2020-01-29 14:55:00,287 - File['/usr/hdp/3.1.0.0-78/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}2020-01-29 14:55:00,290 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}2020-01-29 14:55:00,294 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}2020-01-29 14:55:00,297 - Skipping unlimited key JCE policy check and setup since the Java VM is not managed by Ambari2020-01-29 14:55:00,304 - Skipping stack-select on NIFI because it does not exist in the stack-select package structure.2020-01-29 14:55:00,647 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-782020-01-29 14:55:00,704 - File['/var/lib/ambari-agent/tmp/run_ca.sh'] {'owner': 'nifi', 'content': StaticFile('run_ca.sh'), 'group': 'nifi', 'mode': 0755}2020-01-29 14:55:00,706 - Changing owner of package files2020-01-29 14:55:00,706 - Directory['/usr/hdf/current/nifi-toolkit'] {'group': 'nifi', 'cd_access': 'a', 'recursion_follow_links': True, 'create_parents': False, 'recursive_ownership': True, 'owner': 'nifi', 'mode': 0755}2020-01-29 14:55:00,710 - Directory['/var/run/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,711 - Directory['/var/lib/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,807 - Directory['/var/lib/nifi/database_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,807 - Directory['/hadoopfs/fs1/nifi/flowfile_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,808 - Directory['/hadoopfs/fs1/nifi/provenance_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,810 - Directory['/usr/hdf/current/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,812 - Directory['/var/lib/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,812 - Directory['/var/lib/nifi/state/local'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,814 - Directory['/usr/hdf/current/nifi/lib'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,817 - Directory['/hadoopfs/fs1/nifi/content_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,896 - Directory['/hadoopfs/fs1/nifi/content_repository'] {'owner': 'nifi', 'group': 'nifi', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}2020-01-29 14:55:00,962 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}2020-01-29 14:55:00,966 - File['/etc/security/limits.d/nifi.conf'] {'content': Template('nifi.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}2020-01-29 14:55:00,968 - PropertiesFile['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'group': 'nifi', 'mode': 0600, 'properties': ...}2020-01-29 14:55:00,973 - Generating properties file: /usr/hdf/current/nifi/conf/nifi.properties2020-01-29 14:55:00,973 - File['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600, 'encoding': 'UTF-8'}2020-01-29 14:55:01,070 - Writing File['/usr/hdf/current/nifi/conf/nifi.properties'] because contents don't match2020-01-29 14:55:01,074 - File['/usr/hdf/current/nifi/conf/bootstrap.conf'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600}2020-01-29 14:55:01,075 - Writing File['/usr/hdf/current/nifi/conf/bootstrap.conf'] because contents don't match2020-01-29 14:55:01,078 - File['/usr/hdf/current/nifi/conf/logback.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}2020-01-29 14:55:01,081 - File['/usr/hdf/current/nifi/conf/state-management.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}2020-01-29 14:55:01,088 - File['/usr/hdf/current/nifi/conf/authorizers.xml'] {'owner': 'nifi', 'content': ..., 'group': 'nifi', 'mode': 0600}2020-01-29 14:55:01,093 - File['/usr/hdf/current/nifi/conf/login-identity-providers.xml'] {'owner': 'nifi', 'content': ' kerberos-provider org.apache.nifi.kerberos.KerberosProvider DHUB-FP.COM 12 hours ', 'group': 'nifi', 'mode': 0600}2020-01-29 14:55:01,095 - File['/usr/hdf/current/nifi/bin/nifi-env.sh'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0755}2020-01-29 14:55:01,097 - File['/usr/hdf/current/nifi/conf/bootstrap-notification-services.xml'] {'owner': 'nifi', 'content': ' ', 'group': 'nifi', 'mode': 0400}2020-01-29 14:55:01,099 - File['/usr/hdf/current/nifi/conf/nifi_jaas.conf'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}2020-01-29 14:55:01,099 - Using repository toolkit script: /usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh2020-01-29 14:55:01,099 - File['/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh'] {'mode': 0755}2020-01-29 14:55:01,099 - Encrypting NiFi sensitive configuration properties2020-01-29 14:55:01,166 - Execute[('/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh', '-v', '-b', u'/usr/hdf/current/nifi/conf/bootstrap.conf', '-n', u'/usr/hdf/current/nifi/conf/nifi.properties', '-f', u'/var/lib/nifi/conf/flow.xml.gz', '-s', [PROTECTED], '-l', u'/usr/hdf/current/nifi/conf/login-identity-providers.xml', '-a', u'/usr/hdf/current/nifi/conf/authorizers.xml', '-m', '-e', [PROTECTED], '-p', [PROTECTED])] {'environment': {'JAVA_OPTS': u'-Xms71680m -Xmx71680m', 'JAVA_HOME': u'/usr/lib/jvm/java'}, 'logoutput': False, 'user': 'nifi'}2020-01-29 14:55:14,348 - Skipping stack-select on NIFI because it does not exist in the stack-select package structure. Command failed after 1 tries NiFi version: 1.9.0 Ambari Version: 2.7.3.0 Java Version: openjdk version "1.8.0_191"OpenJDK Runtime Environment (build 1.8.0_191-b12)OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) OS version: centos-release-7-6.1810.2.el7.centos.x86_64 Thank you in advance for any tips in the right direction.

MattWho · Accepted Answer

@DavidR

That is one extremely large flow.xml.gz. I would guess that once uncompressed it falls in the ~1.5 GB size range. My guess is that you have a large number of templates loaded in your NiFi.

The Encrypt-Config toolkit serializes the flow.xml.gz in memory in order to re-encrypt the sensitive properties.
The "Requested array size exceeds VM limit" is telling you that the action taken has created an array that exceeds the max JVM array size of 2^31-1 elements. Increasing the JVM heap size will not change this max supportable array size.

NiFi then ends up with no flow.xml.gz because as a result of the failed encrypt config toolkiit operation, no re-encrypted flow.xml.gz was produced as output. So on next restart attempt of NiFi service, it starts fine and creates a blank flow.xml.gz since one did not exist.

This is documented in this bug: https://issues.apache.org/jira/browse/NIFI-6999

Workaround for this issue:
1. Start your NiFi nodes from command line rather then via Ambari. ( <path to nifi>/bin/nifi.sh start )
2. Once NiFi cluster is back up and running, download and then delete templates stored in NiFi. Access the global menu (upper right corner) --> Templates. This opens a list of all stored NiFi templates. One by one download (optional) and delete each template. Removing stored templates does not affect anything instantiated to the canvas.

These actions should greatly reduce the size of your flow.xml.gz. Then try restarting NiFi via Ambari again.

Hope this helps,

Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.