NiFi crashed and unable start again with error " Java Heap Spare"

[#1]

I am using NiFi to put data to hdfs and hive, at some point needed to restart NiFi but it is not starting again with " Java Heap Space" error. Can any body help me to resolve this issue as NiFi configurations are OK and it was running fine. Below is the output of log:

**********************************************************************************************************************************************

Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 982, in restart
    self.status(env)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 156, in status
    check_process_status(status_params.nifi_node_pid_file)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status
    raise ComponentIsNotRunning()
ComponentIsNotRunning

The above exception was the cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 278, in <module>
    Master().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 142, in start
    self.configure(env, is_starting = True)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 110, in configure
    self.write_configurations(params, is_starting)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 227, in write_configurations
    params.stack_support_encrypt_authorizers, params.stack_version_buildnum)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util.py", line 393, in encrypt_sensitive_properties
    Execute(encrypt_config_command, user=nifi_user, logoutput=False, environment=environment)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -a /usr/hdf/current/nifi/conf/authorizers.xml -p '[PROTECTED]'' returned 255. 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/11/15 13:48:31 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml
2018/11/15 13:48:31 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz
2018/11/15 13:48:31 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool:        bootstrap.conf:               /usr/hdf/current/nifi/conf/bootstrap.conf
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  nifi.properties:              /usr/hdf/current/nifi/conf/nifi.properties
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties:              /usr/hdf/current/nifi/conf/nifi.properties
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  login-identity-providers.xml: null
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  authorizers.xml:              /usr/hdf/current/nifi/conf/authorizers.xml
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml:              /usr/hdf/current/nifi/conf/authorizers.xml
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  flow.xml.gz:                  /var/lib/nifi/conf/flow.xml.gz
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz:                  /var/lib/nifi/conf/flow.xml.gz
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 154 properties from /usr/hdf/current/nifi/conf/nifi.properties
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 154 properties from /usr/hdf/current/nifi/conf/nifi.properties
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 1 protected properties of 5 sensitive properties (25%)
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 153 properties
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded authorizers content (74 lines)
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in authorizers.xml
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in authorizers.xml
2018/11/15 13:48:32 ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: 
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
	at java.lang.StringBuilder.append(StringBuilder.java:190)
	at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:142)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2538)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2516)
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:2493)
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:2441)
	at org.apache.commons.io.IOUtils.toString(IOUtils.java:1084)
	at org.apache.commons.io.IOUtils$toString.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
	at org.apache.nifi.properties.ConfigEncryptionTool$_loadFlowXml_closure3$_closure29.doCall(ConfigEncryptionTool.groovy:666)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at groovy.lang.Closure.call(Closure.java:414)
	at groovy.lang.Closure.call(Closure.java:430)
	at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
	at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
Java heap space

usage: org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain [-h] [options]

This tool enables easy encryption and decryption of configuration files for NiFi and its sub-projects. Unprotected files can be input to this tool to be
protected by a key in a manner that is understood by NiFi. Protected files, along with a key, can be input to this tool to be unprotected, for troubleshooting
or automation purposes.

 -h,--help           Show usage information (this message)
    --nifiRegistry   Specifies to target NiFi Registry. When this flag is not included, NiFi is the target.

When targeting NiFi:
 -h,--help                                  Show usage information (this message)
 -v,--verbose                               Sets verbose mode (default false)
 -n,--niFiProperties <file>                 The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)
 -o,--outputNiFiProperties <file>           The destination nifi.properties file containing protected config values (will not modify input nifi.properties)
 -l,--loginIdentityProviders <file>         The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is
                                            specified)
 -i,--outputLoginIdentityProviders <file>   The destination login-identity-providers.xml file containing protected config values (will not modify input
                                            login-identity-providers.xml)
 -a,--authorizers <file>                    The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)
 -u,--outputAuthorizers <file>              The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
 -f,--flowXml <file>                        The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)
 -g,--outputFlowXml <file>                  The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)
 -b,--bootstrapConf <file>                  The bootstrap.conf file to persist master key
 -k,--key <keyhex>                          The raw hexadecimal key to use to encrypt the sensitive properties
 -e,--oldKey <keyhex>                       The old raw hexadecimal key to use during key migration
 -p,--password <password>                   The password from which to derive the key to use to encrypt the sensitive properties
 -w,--oldPassword <password>                The old password from which to derive the key during migration
 -r,--useRawKey                             If provided, the secure console will prompt for the raw key value in hexadecimal form
 -m,--migrate                               If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with
                                            a new key
 -x,--encryptFlowXmlOnly                    If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or
                                            login-identity-providers.xml files will not be modified
 -s,--propsKey <password|keyhex>            The password or key to use to encrypt the sensitive processor properties in flow.xml.gz
 -A,--newFlowAlgorithm <algorithm>          The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz
 -P,--newFlowProvider <algorithm>           The security provider to use to encrypt the sensitive processor properties in flow.xml.gz
 -c,--translateCli                          Translates the nifi.properties file to a format suitable for the NiFi CLI tool

When targeting NiFi Registry using the --nifiRegistry flag:
 -h,--help                                  Show usage information (this message)
 -v,--verbose                               Sets verbose mode (default false)
 -p,--password <password>                   Protect the files using a password-derived key. If an argument is not provided to this flag, interactive mode will
                                            be triggered to prompt the user to enter the password.
 -k,--key <keyhex>                          Protect the files using a raw hexadecimal key. If an argument is not provided to this flag, interactive mode will be
                                            triggered to prompt the user to enter the key.
    --oldPassword <password>                If the input files are already protected using a password-derived key, this specifies the old password so that the
                                            files can be unprotected before re-protecting.
    --oldKey <keyhex>                       If the input files are already protected using a key, this specifies the raw hexadecimal key so that the files can
                                            be unprotected before re-protecting.
 -b,--bootstrapConf <file>                  The bootstrap.conf file containing no master key or an existing master key. If a new password or key is specified
                                            (using -p or -k) and no output bootstrap.conf file is specified, then this file will be overwritten to persist the
                                            new master key.
 -B,--outputBootstrapConf <file>            The destination bootstrap.conf file to persist master key. If specified, the input bootstrap.conf will not be
                                            modified.
 -r,--nifiRegistryProperties <file>         The nifi-registry.properties file containing unprotected config values, overwritten if no output file specified.
 -R,--outputNifiRegistryProperties <file>   The destination nifi-registry.properties file containing protected config values.
 -a,--authorizersXml <file>                 The authorizers.xml file containing unprotected config values, overwritten if no output file specified.
 -A,--outputAuthorizersXml <file>           The destination authorizers.xml file containing protected config values.
 -i,--identityProvidersXml <file>           The identity-providers.xml file containing unprotected config values, overwritten if no output file specified.
 -I,--outputIdentityProvidersXml <file>     The destination identity-providers.xml file containing protected config values.
    --decrypt                               Can be used with -r to decrypt a previously encrypted NiFi Registry Properties file. Decrypted content is printed to
                                            STDOUT.

[#2 · Accepted solution]

@Naeem Ullah Khan

I see that you are getting OOM error:

2018/11/15 13:48:32 ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: 
java.lang.OutOfMemoryError: Java heap space

Based on your HDF version please change the Heap Settings inside the toolkit as following inside file "/var/lib/ambari-agent/tmp/nifi-toolkit-xxxx.x-x/bin/encrypt-config.sh"

"${JAVA}" -cp "${CLASSPATH}" ${JAVA_OPTS:--Xms512m -Xmx1024m} org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain "$@

.

See: https://community.hortonworks.com/content/supportkb/198288/unable-to-upgrade-from-hdf-300-to-hdf-311...

In Later version of Nifi you should be able to make this change from Ambari UIl

[#3]

Dear @Jay Kumar SenSharma, Thank you very much for your reply which perfectly worked in my case, my NiFi is up and running now. This is great help indeed.