Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDF Nifi not starting

avatar
Contributor

Hi Team,

We are seeing some weird error in HDF 3.0.1.0-43 NIFI it not starting using ambari but it starting without any issues in commandline.

It something failing in the encryption, kindly guide me to resolve this thread.

resource_management.core.exceptions.ExecutionFailed: Execution of 'JAVA_HOME=/usr/jdk64/jdk1.8.0_60 /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /data/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -p '[PROTECTED]'' returned 1. 
stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 309, in <module>
    Master().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 850, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 177, in start
    self.configure(env, is_starting = True)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 119, in locking_configure
    original_configure(obj, *args, **kw)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 150, in configure
    params.nifi_flow_config_dir, params.nifi_sensitive_props_key, is_starting)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in encrypt_sensitive_properties
    Execute(encrypt_config_script_prefix, user=nifi_user,logoutput=False)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'JAVA_HOME=/usr/jdk64/jdk1.8.0_60 /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /data/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -p '[PROTECTED]'' returned 1. 
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of login-identity-providers.xml
2017/10/12 23:54:22 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source login-identity-providers.xml and destination login-identity-providers.xml are identical [/usr/hdf/current/nifi/conf/login-identity-providers.xml] so the original will be overwritten
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2017/10/12 23:54:22 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz
2017/10/12 23:54:22 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/data/nifi/conf/flow.xml.gz] so the original will be overwritten
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool:        bootstrap.conf:               /usr/hdf/current/nifi/conf/bootstrap.conf
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  nifi.properties:              /usr/hdf/current/nifi/conf/nifi.properties
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties:              /usr/hdf/current/nifi/conf/nifi.properties
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src)  flow.xml.gz: /data/nifi/conf/flow.xml.gz
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: /data/nifi/conf/flow.xml.gz
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 128 properties from /usr/hdf/current/nifi/conf/nifi.properties
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 128 properties from /usr/hdf/current/nifi/conf/nifi.properties
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 128 properties
2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded LoginIdentityProviders content (108 lines)
2017/10/12 23:54:22 WARN [main] org.apache.nifi.properties.AESSensitivePropertyProvider: JCE Unlimited Strength Cryptography Jurisdiction policies are not available, so the max key length is 128 bits
2017/10/12 23:54:22 WARN [main] org.apache.nifi.properties.AESSensitivePropertyProvider: JCE Unlimited Strength Cryptography Jurisdiction policies are not available, so the max key length is 128 bits
2017/10/12 23:54:23 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in login-identity-providers.xml
2017/10/12 23:54:23 WARN [main] org.apache.nifi.properties.AESSensitivePropertyProvider: JCE Unlimited Strength Cryptography Jurisdiction policies are not available, so the max key length is 128 bits
2017/10/12 23:54:23 WARN [main] org.apache.nifi.properties.AESSensitivePropertyProvider: JCE Unlimited Strength Cryptography Jurisdiction policies are not available, so the max key length is 128 bits
2017/10/12 23:54:23 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Attempting to encrypt property
2017/10/12 23:54:23 INFO [main] org.apache.nifi.properties.AESSensitivePropertyProvider: AES Sensitive Property Provider encrypted a sensitive value successfully
2017/10/12 23:54:23 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated XML content: <?xml version="1.0" encoding="UTF-8"?><loginIdentityProviders>
  <provider>
    <identifier>ldap-provider</identifier>
    <class>org.apache.nifi.ldap.LdapProvider</class>
    <property name="Identity Strategy">USE_DN</property>
    <property name="Authentication Strategy">SIMPLE</property>
    <property name="Manager DN">svc.haasdev.001@zone1.scbdev.net</property>
    <property name="Manager Password" encryption="aes/gcm/128">cYAgEgKZAsfFiVjR||jB6aUU2eBkjpqUx2q7DvmE3AclRJZPdkKvQ4</property>
    <property name="TLS - Keystore"/>
    <property name="TLS - Keystore Password"/>
    <property name="TLS - Keystore Type"/>
    <property name="TLS - Truststore"/>
    <property name="TLS - Truststore Password"/>
    <property name="TLS - Truststore Type"/>
    <property name="TLS - Client Auth"/>
    <property name="TLS - Protocol">TLS</property>
    <property name="TLS - Shutdown Gracefully"/>
    <property name="Referral Strategy">FOLLOW</property>
    <property name="Connect Timeout">10 secs</property>
    <property name="Read Timeout">10 secs</property>
    <property name="Url">ldap://HKWVADIDM05.zone1.scbdev.net:389</property>
    <property name="User Search Base">OU=ITSC,dc=zone1,dc=scbdev,dc=net</property>
    <property name="Identity Strategy">USE_USERNAME</property>
    <property name="User Search Filter">sAMAccountName={0}</property>
    <property name="Authentication Expiration">12 hours</property>
  </provider>
</loginIdentityProviders>
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:143)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2370)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2348)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2325)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2273)
at org.apache.commons.io.IOUtils.toString(IOUtils.java:1041)
at org.apache.commons.io.IOUtils$toString.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
at org.apache.nifi.properties.ConfigEncryptionTool$_loadFlowXml_closure2$_closure19.doCall(ConfigEncryptionTool.groovy:488)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
at groovy.lang.Closure.call(Closure.java:426)
at groovy.lang.Closure.call(Closure.java:442)
at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
 stdout:
2017-10-12 23:53:50,252 - Group['ranger'] {}
2017-10-12 23:53:50,253 - Group['hadoop'] {}
2017-10-12 23:53:50,253 - Group['nifi'] {}
2017-10-12 23:53:50,253 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-10-12 23:53:50,255 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-10-12 23:53:50,255 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-10-12 23:53:50,256 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2017-10-12 23:53:50,257 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-10-12 23:53:50,257 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger']}
2017-10-12 23:53:50,258 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'nifi']}
2017-10-12 23:53:50,259 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-10-12 23:53:50,260 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-10-12 23:53:50,265 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2017-10-12 23:53:50,278 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2017-10-12 23:53:50,284 - Skipping Execute[('setenforce', '0')] due to not_if
2017-10-12 23:53:50,501 - Stack Feature Version Info: stack_version=3.0, version=3.0.1.0-43, current_cluster_version=3.0.1.0-43 -> 3.0.1.0-43
2017-10-12 23:53:50,546 - File['/usr/hdf/current/nifi/bin/nifi-env.sh'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0755}
2017-10-12 23:53:50,548 - Execute['export JAVA_HOME=/usr/jdk64/jdk1.8.0_60;/usr/hdf/current/nifi/bin/nifi.sh stop >> /data/nifi/log/nifi-setup.log'] {'user': 'nifi'}
2017-10-12 23:54:20,757 - Pid file /var/run/nifi/nifi.pid is empty or does not exist
2017-10-12 23:54:20,759 - Directory['/var/run/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,759 - Directory['/data/nifi/log'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,763 - Directory['/data/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,936 - Directory['/data/nifi/database_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,937 - Directory['/data/nifi/flowfile_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,947 - Directory['/data/nifi/provenance_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,957 - Directory['/usr/hdf/current/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,958 - Directory['/data/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,958 - Directory['/data/nifi/state/local'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,959 - Directory['/usr/hdf/current/nifi/lib'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,961 - Directory['{{nifi_content_repo_dir_default}}'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-10-12 23:54:20,962 - Directory['/data/nifi/content_repository'] {'owner': 'nifi', 'group': 'nifi', 'create_parents': True, 'recursive_ownership': True}
2017-10-12 23:54:21,027 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2017-10-12 23:54:21,030 - File['/etc/security/limits.d/nifi.conf'] {'content': Template('nifi.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2017-10-12 23:54:21,174 - PropertiesFile['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'group': 'nifi', 'mode': 0600, 'properties': ...}
2017-10-12 23:54:21,178 - Generating properties file: /usr/hdf/current/nifi/conf/nifi.properties
2017-10-12 23:54:21,178 - File['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600}
2017-10-12 23:54:21,258 - Writing File['/usr/hdf/current/nifi/conf/nifi.properties'] because contents don't match
2017-10-12 23:54:21,261 - File['/usr/hdf/current/nifi/conf/bootstrap.conf'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600}
2017-10-12 23:54:21,265 - File['/usr/hdf/current/nifi/conf/logback.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}
2017-10-12 23:54:21,268 - File['/usr/hdf/current/nifi/conf/state-management.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}
2017-10-12 23:54:21,273 - File['/usr/hdf/current/nifi/conf/authorizers.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}
2017-10-12 23:54:21,277 - File['/usr/hdf/current/nifi/conf/login-identity-providers.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600}
2017-10-12 23:54:21,278 - File['/usr/hdf/current/nifi/bin/nifi-env.sh'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0755}
2017-10-12 23:54:21,280 - File['/usr/hdf/current/nifi/conf/bootstrap-notification-services.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400}
2017-10-12 23:54:21,281 - Encrypting NiFi sensitive configuration properties
2017-10-12 23:54:21,281 - File['/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh'] {'mode': 0755}
2017-10-12 23:54:21,429 - Execute[(u'JAVA_HOME=/usr/jdk64/jdk1.8.0_60', '/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh', '-v', '-b', u'/usr/hdf/current/nifi/conf/bootstrap.conf', '-n', u'/usr/hdf/current/nifi/conf/nifi.properties', '-f', u'/data/nifi/conf/flow.xml.gz', '-s', [PROTECTED], '-l', u'/usr/hdf/current/nifi/conf/login-identity-providers.xml', '-p', [PROTECTED])] {'logoutput': False, 'user': 'nifi'}
Command failed after 1 tries
1 ACCEPTED SOLUTION

avatar
Super Mentor

@Anishkumar Valsalam

You have encountered the following:

https://issues.apache.org/jira/browse/NIFI-4363

You can get around this issue by editing the following script on every NiFi node in your cluster:

/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh 

Near the bottom of the script you will find this line:

"${JAVA}" -cp "${CLASSPATH}" -Xms128m -Xmx256m org.apache.nifi.properties.ConfigEncryptionTool "$@"

Edit it so that it now says:

"${JAVA}" -cp "${CLASSPATH}" -Xms512m -Xmx1024m org.apache.nifi.properties.ConfigEncryptionTool "$@"

Then go back to Ambari and start your NiFi service.

Thank you,

Matt

View solution in original post

5 REPLIES 5

avatar
Contributor

@Matt Clarke Bro you have any idea on this error?

avatar
Super Mentor

@Anishkumar Valsalam

You have encountered the following:

https://issues.apache.org/jira/browse/NIFI-4363

You can get around this issue by editing the following script on every NiFi node in your cluster:

/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh 

Near the bottom of the script you will find this line:

"${JAVA}" -cp "${CLASSPATH}" -Xms128m -Xmx256m org.apache.nifi.properties.ConfigEncryptionTool "$@"

Edit it so that it now says:

"${JAVA}" -cp "${CLASSPATH}" -Xms512m -Xmx1024m org.apache.nifi.properties.ConfigEncryptionTool "$@"

Then go back to Ambari and start your NiFi service.

Thank you,

Matt

avatar
Super Mentor

@Anishkumar Valsalam

If the answer provided solved your issue, please take a moment to click "Accept" below the answer to close out this thread. This will assist other who are searching for a solution to the same problem.

Thank you,

Matt

avatar
Contributor
@Matt Clarke

Thanks Bro you are my saviour 🙂

avatar
Explorer

Thanks Matt and Anish. It helped me a lot today 🙂