I am using NiFi to put data to hdfs and hive, at some point needed to restart NiFi but it is not starting again with " Java Heap Space" error. Can any body help me to resolve this issue as NiFi configurations are OK and it was running fine. Below is the output of log: ********************************************************************************************************************************************** Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 982, in restart self.status(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 156, in status check_process_status(status_params.nifi_node_pid_file) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status raise ComponentIsNotRunning() ComponentIsNotRunning The above exception was the cause of the following exception: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 278, in <module> Master().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 142, in start self.configure(env, is_starting = True) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 110, in configure self.write_configurations(params, is_starting) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 227, in write_configurations params.stack_support_encrypt_authorizers, params.stack_version_buildnum) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util.py", line 393, in encrypt_sensitive_properties Execute(encrypt_config_command, user=nifi_user, logoutput=False, environment=environment) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run returns=self.resource.returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -a /usr/hdf/current/nifi/conf/authorizers.xml -p '[PROTECTED]'' returned 255. 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties 2018/11/15 13:48:31 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml 2018/11/15 13:48:31 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz 2018/11/15 13:48:31 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: /usr/hdf/current/nifi/conf/bootstrap.conf 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz 2018/11/15 13:48:31 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 154 properties from /usr/hdf/current/nifi/conf/nifi.properties 2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 154 properties from /usr/hdf/current/nifi/conf/nifi.properties 2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 1 protected properties of 5 sensitive properties (25%) 2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 153 properties 2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded authorizers content (74 lines) 2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in authorizers.xml 2018/11/15 13:48:32 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in authorizers.xml 2018/11/15 13:48:32 ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596) at java.lang.StringBuilder.append(StringBuilder.java:190) at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:142) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2538) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2516) at org.apache.commons.io.IOUtils.copy(IOUtils.java:2493) at org.apache.commons.io.IOUtils.copy(IOUtils.java:2441) at org.apache.commons.io.IOUtils.toString(IOUtils.java:1084) at org.apache.commons.io.IOUtils$toString.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133) at org.apache.nifi.properties.ConfigEncryptionTool$_loadFlowXml_closure3$_closure29.doCall(ConfigEncryptionTool.groovy:666) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at groovy.lang.Closure.call(Closure.java:414) at groovy.lang.Closure.call(Closure.java:430) at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622) at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54) Java heap space usage: org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain [-h] [options] This tool enables easy encryption and decryption of configuration files for NiFi and its sub-projects. Unprotected files can be input to this tool to be protected by a key in a manner that is understood by NiFi. Protected files, along with a key, can be input to this tool to be unprotected, for troubleshooting or automation purposes. -h,--help Show usage information (this message) --nifiRegistry Specifies to target NiFi Registry. When this flag is not included, NiFi is the target. When targeting NiFi: -h,--help Show usage information (this message) -v,--verbose Sets verbose mode (default false) -n,--niFiProperties <file> The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified) -o,--outputNiFiProperties <file> The destination nifi.properties file containing protected config values (will not modify input nifi.properties) -l,--loginIdentityProviders <file> The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is specified) -i,--outputLoginIdentityProviders <file> The destination login-identity-providers.xml file containing protected config values (will not modify input login-identity-providers.xml) -a,--authorizers <file> The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified) -u,--outputAuthorizers <file> The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml) -f,--flowXml <file> The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified) -g,--outputFlowXml <file> The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz) -b,--bootstrapConf <file> The bootstrap.conf file to persist master key -k,--key <keyhex> The raw hexadecimal key to use to encrypt the sensitive properties -e,--oldKey <keyhex> The old raw hexadecimal key to use during key migration -p,--password <password> The password from which to derive the key to use to encrypt the sensitive properties -w,--oldPassword <password> The old password from which to derive the key during migration -r,--useRawKey If provided, the secure console will prompt for the raw key value in hexadecimal form -m,--migrate If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with a new key -x,--encryptFlowXmlOnly If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or login-identity-providers.xml files will not be modified -s,--propsKey <password|keyhex> The password or key to use to encrypt the sensitive processor properties in flow.xml.gz -A,--newFlowAlgorithm <algorithm> The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz -P,--newFlowProvider <algorithm> The security provider to use to encrypt the sensitive processor properties in flow.xml.gz -c,--translateCli Translates the nifi.properties file to a format suitable for the NiFi CLI tool When targeting NiFi Registry using the --nifiRegistry flag: -h,--help Show usage information (this message) -v,--verbose Sets verbose mode (default false) -p,--password <password> Protect the files using a password-derived key. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password. -k,--key <keyhex> Protect the files using a raw hexadecimal key. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key. --oldPassword <password> If the input files are already protected using a password-derived key, this specifies the old password so that the files can be unprotected before re-protecting. --oldKey <keyhex> If the input files are already protected using a key, this specifies the raw hexadecimal key so that the files can be unprotected before re-protecting. -b,--bootstrapConf <file> The bootstrap.conf file containing no master key or an existing master key. If a new password or key is specified (using -p or -k) and no output bootstrap.conf file is specified, then this file will be overwritten to persist the new master key. -B,--outputBootstrapConf <file> The destination bootstrap.conf file to persist master key. If specified, the input bootstrap.conf will not be modified. -r,--nifiRegistryProperties <file> The nifi-registry.properties file containing unprotected config values, overwritten if no output file specified. -R,--outputNifiRegistryProperties <file> The destination nifi-registry.properties file containing protected config values. -a,--authorizersXml <file> The authorizers.xml file containing unprotected config values, overwritten if no output file specified. -A,--outputAuthorizersXml <file> The destination authorizers.xml file containing protected config values. -i,--identityProvidersXml <file> The identity-providers.xml file containing unprotected config values, overwritten if no output file specified. -I,--outputIdentityProvidersXml <file> The destination identity-providers.xml file containing protected config values. --decrypt Can be used with -r to decrypt a previously encrypted NiFi Registry Properties file. Decrypted content is printed to STDOUT. ... View more

I am working to clean CSV files and stuck due to additional coma's in headers. Can any body guide me how to replace header line with customer header. ... View more

I am using NiFi replacetext processor to delete some strings in csv file but somehow its not working. Below is the search string: (,Minutes.*\n|CellID|=|"|LogicRNCID|CGI|CellIndex|GCELL:LABEL) below is snapshot: Can anyone tell how to mention multiple string in search box and how to mention delete these in replacement value. ... View more

I have build a process on NiFi for windows where it lists SFTP from a linux machine, Fetchsftp for listed file and putsftp to another linux server. This process is running fine and all files are received on target server. Now I wan to run this process as a cron job on a linux machine independent of NiFi which is running on windows machine in browser. Can any body guide me how to configure linux machine to do this process as a job. ... View more