NiFi won't start throwing "java.lang.OutOfMemoryError: Java heap space" after HDF upgrade to 3.0.2

[#1]

Issue:

When you upgrade HDF to 3.0.2, you might experience NiFi service unable to start via Ambari due to the OOM exception in the below stack trace

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 309, in <module>
    Master().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 177, in start
    self.configure(env, is_starting = True)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 120, in locking_configure
    original_configure(obj, *args, **kw)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 150, in configure
    params.nifi_flow_config_dir, params.nifi_sensitive_props_key, is_starting)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in encrypt_sensitive_properties
    Execute(encrypt_config_script_prefix, user=nifi_user,logoutput=False)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.2.0-76/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -p '[PROTECTED]'' returned 1. 2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of login-identity-providers.xml
2018/04/27 17:28:26 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source login-identity-providers.xml and destination login-identity-providers.xml are identical [/usr/hdf/current/nifi/conf/login-identity-providers.xml] so the original will be overwritten
2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/04/27 17:28:26 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten
2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz
2018/04/27 17:28:26 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten
2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool:        bootstrap.conf:               /usr/hdf/current/nifi/conf/bootstrap.conf
2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  nifi.properties:              /usr/hdf/current/nifi/conf/nifi.properties
2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties:              /usr/hdf/current/nifi/conf/nifi.properties
2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml
2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml
2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  flow.xml.gz:                                  /var/lib/nifi/conf/flow.xml.gz
2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz:                                  /var/lib/nifi/conf/flow.xml.gz
2018/04/27 17:28:26 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 133 properties from /usr/hdf/current/nifi/conf/nifi.properties
2018/04/27 17:28:27 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 133 properties from /usr/hdf/current/nifi/conf/nifi.properties
2018/04/27 17:28:27 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 133 properties
2018/04/27 17:28:27 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded LoginIdentityProviders content (104 lines)
2018/04/27 17:28:27 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in login-identity-providers.xml
2018/04/27 17:28:27 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in login-identity-providers.xml
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
        at java.lang.StringBuilder.append(StringBuilder.java:190)
        at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:143)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2370)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2348)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2325)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2273)
        at org.apache.commons.io.IOUtils.toString(IOUtils.java:1041)
        at org.apache.commons.io.IOUtils$toString.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
        at org.apache.nifi.properties.ConfigEncryptionTool$_loadFlowXml_closure2$_closure19.doCall(ConfigEncryptionTool.groovy:488)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
        at groovy.lang.Closure.call(Closure.java:426)
        at groovy.lang.Closure.call(Closure.java:442)
        at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
        at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)

Root cause: By default, the max heap configured in the encrypt-config.sh file is not sufficient to run the service via Ambari. If you are upgrading HDF to versions above & including 3.0.2 it requires a max heap of 1G to be set in-order for the NiFi service to start successfully from Ambari

Resolution:

1. ssh into the Ambari server node

2. find / -name encrypt-config.sh

3. Open the 'encrypt-config.sh' file from the correct HDF version using 'vi'. For ex. in this scenario the version is 3.0.2, so you'll find the script under the below location

/var/lib/ambari-server/resources/mpacks/hdf-ambari-mpack-3.0.2.0-76/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.2.0-76/bin/encrypt-config.sh

4. Increase the max heap to '1024m' (-Xmx1024m) as shown below and 'wq!' the change

umask 0077
   "${JAVA}" -cp "${CLASSPATH}" -Xms128m -Xmx1024m org.apache.nifi.properties.ConfigEncryptionTool "$@"
   return $?
}
 
 
init
run "$@"

5. Restart ambari-server and ambari-agent

6. Now restart NiFi service from Ambari UI and it should work like a charm

[#2]

Heads up folks!

The above responses are valid for previous version of HDF. For Ambari 2.6.1 with NIFI

1.5.0.3.1.1.0-35 or better the location changed to inside the Ambari interface. Please find

Ambari GUI --> NIFI ---> Configs ----> Advanced nifi-toolkit-env ---> NIFI Toolkit Java Options

I've found restarting the ambari-server and ambari-agents is no longer necessary as ambari-server pushes these options to the toolkit at runtime, overiding the configurations in the script and the configuration file.