Created 03-29-2018 08:53 PM
I'm new to the Hortonworks environment and I'm using the HDF 3.1 Sandbox in Virtualbox. Since last night my flow.xml.gz won't load after a reboot. I haven't changed any variables from my perspective.
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 231, in <module> Master().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 978, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 152, in start self.configure(env, is_starting = True) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 120, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 122, in configure params.stack_support_encrypt_authorizers) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util.py", line 448, in encrypt_sensitive_properties Execute(encrypt_config_command, user=nifi_user, logoutput=False, environment=environment) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.0.0-564/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -a /usr/hdf/current/nifi/conf/authorizers.xml -p '[PROTECTED]'' returned 9. The place it is looking is: /var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.0.0-564/classpath:/var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.0.0-564/lib/* 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml 2018/03/29 09:44:18 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties 2018/03/29 09:44:18 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz 2018/03/29 09:44:18 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: /usr/hdf/current/nifi/conf/bootstrap.conf 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz 2018/03/29 09:44:18 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 145 properties from /usr/hdf/current/nifi/conf/nifi.properties 2018/03/29 09:44:19 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 145 properties from /usr/hdf/current/nifi/conf/nifi.properties 2018/03/29 09:44:19 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 145 properties 2018/03/29 09:44:19 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded authorizers content (74 lines) 2018/03/29 09:44:19 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in authorizers.xml 2018/03/29 09:44:19 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No unencrypted password property elements found in authorizers.xml Cannot load flow.xml.gz
Created 04-02-2018 11:29 PM
How much memory are you allocating to VirtualBox? It should be no less than 8GB, but more is always better. I found that 10-11 GB works best.
Created 04-03-2018 06:56 AM
Thank you for the response, I've normally allocated 8GB, but I've booted up the machine with 12GB now. Still the same error log. I wonder what the returned 9 means in this part of the log.
resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.0.0-564/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -a /usr/hdf/current/nifi/conf/authorizers.xml -p '[PROTECTED]'' returned 9.
Created 04-03-2018 08:14 AM
Matt posted an answer in HCC, You have encountered the following:
https://issues.apache.org/jira/browse/NIFI-4363
You can get around this issue by editing the following script on every NiFi node in your cluster:
/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh
Near the bottom of the script you will find this line:
"${JAVA}"-cp "${CLASSPATH}"-Xms128m-Xmx256m org.apache.nifi.properties.ConfigEncryptionTool"$@"
Edit it so that it now says:
"${JAVA}"-cp "${CLASSPATH}"-Xms512m-Xmx1024m org.apache.nifi.properties.ConfigEncryptionTool"$@"
Then go back to Ambari and start your NiFi service.
Hope that helps
Created 05-07-2018 11:56 AM
I've ran into the same issue again after reverting to an older snapshot, I've updated the scripts and the config in Ambari (since the edits in the scripts didn't do anything and JAVA_OPTS needed a memory increase) but it didn't work out. Any suggestions are appreciated.
Maybe the flow.xml is corrupt?
Execute[('/var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.0.0-564/bin/encrypt-config.sh', '-v', '-b', u'/usr/hdf/current/nifi/conf/bootstrap.conf', '-n', u'/usr/hdf/current/nifi/conf/nifi.properties', '-f', u'/var/lib/nifi/conf/flow.xml.gz', '-s', [PROTECTED], '-a', u'/usr/hdf/current/nifi/conf/authorizers.xml', '-p', [PROTECTED])] {'environment': {'JAVA_OPTS': u'-Xms512m -Xmx1024m', 'JAVA_HOME': u'/usr/java/default'}, 'logoutput': False, 'user': 'nifi'} Command failed after 1 tries
Created 05-17-2018 01:54 PM
Heads up folks!
The above responses are valid for previous version of HDF. For Ambari 2.6.1 with NIFI
1.5.0.3.1.1.0-35 or better the location changed to inside the Ambari interface. Please find
Ambari GUI --> NIFI ---> Configs ----> Advanced nifi-toolkit-env ---> NIFI Toolkit Java Options