NiFi crashed and unable start again with error " Java Heap Spare"

naeem_khan — Thu, 15 Nov 2018 17:10:48 GMT

I am using NiFi to put data to hdfs and hive, at some point needed to restart NiFi but it is not starting again with " Java Heap Space" error. Can any body help me to resolve this issue as NiFi configurations are OK and it was running fine. Below is the output of log:

**********************************************************************************************************************************************

Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 982, in restart
    self.status(env)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 156, in status
    check_process_status(status_params.nifi_node_pid_file)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status
    raise ComponentIsNotRunning()
ComponentIsNotRunning

The above exception was the cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 278, in <module>
    Master().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 142, in start
    self.configure(env, is_starting = True)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 110, in configure
    self.write_configurations(params, is_starting)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 227, in write_configurations
    params.stack_support_encrypt_authorizers, params.stack_version_buildnum)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util.py", line 393, in encrypt_sensitive_properties
    Execute(encrypt_config_command, user=nifi_user, logoutput=False, environment=environment)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -a /usr/hdf/current/nifi/conf/authorizers.xml -p '[PROTECTED]'' returned 255. 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2018/11/15 13:48:31 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml
2018/11/15 13:48:31 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz
2018/11/15 13:48:31 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool:        bootstrap.conf:               /usr/hdf/current/nifi/conf/bootstrap.conf
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  nifi.properties:              /usr/hdf/current/nifi/conf/nifi.properties
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties:              /usr/hdf/current/nifi/conf/nifi.properties
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  login-identity-providers.xml: null
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  authorizers.xml:              /usr/hdf/current/nifi/conf/authorizers.xml
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml:              /usr/hdf/current/nifi/conf/authorizers.xml
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  flow.xml.gz:                  /var/lib/nifi/conf/flow.xml.gz
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz:                  /var/lib/nifi/conf/flow.xml.gz
2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 154 properties from /usr/hdf/current/nifi/conf/nifi.properties
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 154 properties from /usr/hdf/current/nifi/conf/nifi.properties
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 1 protected properties of 5 sensitive properties (25%)
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 153 properties
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded authorizers content (74 lines)
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in authorizers.xml
2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in authorizers.xml
2018/11/15 13:48:32 ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: 
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
	at java.lang.StringBuilder.append(StringBuilder.java:190)
	at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:142)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2538)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2516)
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:2493)
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:2441)
	at org.apache.commons.io.IOUtils.toString(IOUtils.java:1084)
	at org.apache.commons.io.IOUtils$toString.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
	at org.apache.nifi.properties.ConfigEncryptionTool$_loadFlowXml_closure3$_closure29.doCall(ConfigEncryptionTool.groovy:666)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at groovy.lang.Closure.call(Closure.java:414)
	at groovy.lang.Closure.call(Closure.java:430)
	at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
	at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
Java heap space

usage: org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain [-h] [options]

This tool enables easy encryption and decryption of configuration files for NiFi and its sub-projects. Unprotected files can be input to this tool to be
protected by a key in a manner that is understood by NiFi. Protected files, along with a key, can be input to this tool to be unprotected, for troubleshooting
or automation purposes.

 -h,--help           Show usage information (this message)
    --nifiRegistry   Specifies to target NiFi Registry. When this flag is not included, NiFi is the target.

When targeting NiFi:
 -h,--help                                  Show usage information (this message)
 -v,--verbose                               Sets verbose mode (default false)
 -n,--niFiProperties <file>                 The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)
 -o,--outputNiFiProperties <file>           The destination nifi.properties file containing protected config values (will not modify input nifi.properties)
 -l,--loginIdentityProviders <file>         The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is
                                            specified)
 -i,--outputLoginIdentityProviders <file>   The destination login-identity-providers.xml file containing protected config values (will not modify input
                                            login-identity-providers.xml)
 -a,--authorizers <file>                    The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)
 -u,--outputAuthorizers <file>              The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
 -f,--flowXml <file>                        The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)
 -g,--outputFlowXml <file>                  The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)
 -b,--bootstrapConf <file>                  The bootstrap.conf file to persist master key
 -k,--key <keyhex>                          The raw hexadecimal key to use to encrypt the sensitive properties
 -e,--oldKey <keyhex>                       The old raw hexadecimal key to use during key migration
 -p,--password <password>                   The password from which to derive the key to use to encrypt the sensitive properties
 -w,--oldPassword <password>                The old password from which to derive the key during migration
 -r,--useRawKey                             If provided, the secure console will prompt for the raw key value in hexadecimal form
 -m,--migrate                               If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with
                                            a new key
 -x,--encryptFlowXmlOnly                    If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or
                                            login-identity-providers.xml files will not be modified
 -s,--propsKey <password|keyhex>            The password or key to use to encrypt the sensitive processor properties in flow.xml.gz
 -A,--newFlowAlgorithm <algorithm>          The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz
 -P,--newFlowProvider <algorithm>           The security provider to use to encrypt the sensitive processor properties in flow.xml.gz
 -c,--translateCli                          Translates the nifi.properties file to a format suitable for the NiFi CLI tool

When targeting NiFi Registry using the --nifiRegistry flag:
 -h,--help                                  Show usage information (this message)
 -v,--verbose                               Sets verbose mode (default false)
 -p,--password <password>                   Protect the files using a password-derived key. If an argument is not provided to this flag, interactive mode will
                                            be triggered to prompt the user to enter the password.
 -k,--key <keyhex>                          Protect the files using a raw hexadecimal key. If an argument is not provided to this flag, interactive mode will be
                                            triggered to prompt the user to enter the key.
    --oldPassword <password>                If the input files are already protected using a password-derived key, this specifies the old password so that the
                                            files can be unprotected before re-protecting.
    --oldKey <keyhex>                       If the input files are already protected using a key, this specifies the raw hexadecimal key so that the files can
                                            be unprotected before re-protecting.
 -b,--bootstrapConf <file>                  The bootstrap.conf file containing no master key or an existing master key. If a new password or key is specified
                                            (using -p or -k) and no output bootstrap.conf file is specified, then this file will be overwritten to persist the
                                            new master key.
 -B,--outputBootstrapConf <file>            The destination bootstrap.conf file to persist master key. If specified, the input bootstrap.conf will not be
                                            modified.
 -r,--nifiRegistryProperties <file>         The nifi-registry.properties file containing unprotected config values, overwritten if no output file specified.
 -R,--outputNifiRegistryProperties <file>   The destination nifi-registry.properties file containing protected config values.
 -a,--authorizersXml <file>                 The authorizers.xml file containing unprotected config values, overwritten if no output file specified.
 -A,--outputAuthorizersXml <file>           The destination authorizers.xml file containing protected config values.
 -i,--identityProvidersXml <file>           The identity-providers.xml file containing unprotected config values, overwritten if no output file specified.
 -I,--outputIdentityProvidersXml <file>     The destination identity-providers.xml file containing protected config values.
    --decrypt                               Can be used with -r to decrypt a previously encrypted NiFi Registry Properties file. Decrypted content is printed to
                                            STDOUT.

Re: NiFi crashed and unable start again with error " Java Heap Spare"

jsensharma — Thu, 15 Nov 2018 18:10:31 GMT

@Naeem Ullah Khan

I see that you are getting OOM error:

2018/11/15 13:48:32 ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: 
java.lang.OutOfMemoryError: Java heap space

Based on your HDF version please change the Heap Settings inside the toolkit as following inside file "/var/lib/ambari-agent/tmp/nifi-toolkit-xxxx.x-x/bin/encrypt-config.sh"

"${JAVA}" -cp "${CLASSPATH}" ${JAVA_OPTS:--Xms512m -Xmx1024m} org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain "$@

See: https://community.hortonworks.com/content/supportkb/198288/unable-to-upgrade-from-hdf-300-to-hdf-311-as-encry.html

In Later version of Nifi you should be able to make this change from Ambari UIl

Re: NiFi crashed and unable start again with error " Java Heap Spare"

naeem_khan — Fri, 16 Nov 2018 18:01:52 GMT

Dear @Jay Kumar SenSharma, Thank you very much for your reply which perfectly worked in my case, my NiFi is up and running now. This is great help indeed.

question NiFi crashed and unable start again with error " Java Heap Spare" in Archives of Support Questions (Read Only)

NiFi crashed and unable start again with error " Java Heap Spare"

Re: NiFi crashed and unable start again with error " Java Heap Spare"

Re: NiFi crashed and unable start again with error " Java Heap Spare"