Hi Team, I am trying to configure the RPG in my cluster environemt (3nodes) how to distribute the data between the 3 nodes. This the below processor i am trying with RPG but it not processing at all attached the details FYR. Kindly help me to close this thread. ... View more

Hi Team, We are seeing some weird error in HDF 3.0.1.0-43 NIFI it not starting using ambari but it starting without any issues in commandline. It something failing in the encryption, kindly guide me to resolve this thread. resource_management.core.exceptions.ExecutionFailed: Execution of 'JAVA_HOME=/usr/jdk64/jdk1.8.0_60 /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /data/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -p '[PROTECTED]'' returned 1. stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 309, in <module> Master().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 850, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 177, in start self.configure(env, is_starting = True) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 119, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 150, in configure params.nifi_flow_config_dir, params.nifi_sensitive_props_key, is_starting) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in encrypt_sensitive_properties Execute(encrypt_config_script_prefix, user=nifi_user,logoutput=False) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'JAVA_HOME=/usr/jdk64/jdk1.8.0_60 /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /data/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -p '[PROTECTED]'' returned 1. 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of login-identity-providers.xml 2017/10/12 23:54:22 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source login-identity-providers.xml and destination login-identity-providers.xml are identical [/usr/hdf/current/nifi/conf/login-identity-providers.xml] so the original will be overwritten 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties 2017/10/12 23:54:22 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz 2017/10/12 23:54:22 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/data/nifi/conf/flow.xml.gz] so the original will be overwritten 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: /usr/hdf/current/nifi/conf/bootstrap.conf 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: /usr/hdf/current/nifi/conf/login-identity-providers.xml 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: /data/nifi/conf/flow.xml.gz 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: /data/nifi/conf/flow.xml.gz 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 128 properties from /usr/hdf/current/nifi/conf/nifi.properties 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 128 properties from /usr/hdf/current/nifi/conf/nifi.properties 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 128 properties 2017/10/12 23:54:22 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded LoginIdentityProviders content (108 lines) 2017/10/12 23:54:22 WARN [main] org.apache.nifi.properties.AESSensitivePropertyProvider: JCE Unlimited Strength Cryptography Jurisdiction policies are not available, so the max key length is 128 bits 2017/10/12 23:54:22 WARN [main] org.apache.nifi.properties.AESSensitivePropertyProvider: JCE Unlimited Strength Cryptography Jurisdiction policies are not available, so the max key length is 128 bits 2017/10/12 23:54:23 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: No encrypted password property elements found in login-identity-providers.xml 2017/10/12 23:54:23 WARN [main] org.apache.nifi.properties.AESSensitivePropertyProvider: JCE Unlimited Strength Cryptography Jurisdiction policies are not available, so the max key length is 128 bits 2017/10/12 23:54:23 WARN [main] org.apache.nifi.properties.AESSensitivePropertyProvider: JCE Unlimited Strength Cryptography Jurisdiction policies are not available, so the max key length is 128 bits 2017/10/12 23:54:23 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Attempting to encrypt property 2017/10/12 23:54:23 INFO [main] org.apache.nifi.properties.AESSensitivePropertyProvider: AES Sensitive Property Provider encrypted a sensitive value successfully 2017/10/12 23:54:23 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Updated XML content: <?xml version="1.0" encoding="UTF-8"?><loginIdentityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Identity Strategy">USE_DN</property> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">svc.haasdev.001@zone1.scbdev.net</property> <property name="Manager Password" encryption="aes/gcm/128">cYAgEgKZAsfFiVjR||jB6aUU2eBkjpqUx2q7DvmE3AclRJZPdkKvQ4</property> <property name="TLS - Keystore"/> <property name="TLS - Keystore Password"/> <property name="TLS - Keystore Type"/> <property name="TLS - Truststore"/> <property name="TLS - Truststore Password"/> <property name="TLS - Truststore Type"/> <property name="TLS - Client Auth"/> <property name="TLS - Protocol">TLS</property> <property name="TLS - Shutdown Gracefully"/> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://HKWVADIDM05.zone1.scbdev.net:389</property> <property name="User Search Base">OU=ITSC,dc=zone1,dc=scbdev,dc=net</property> <property name="Identity Strategy">USE_USERNAME</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Authentication Expiration">12 hours</property> </provider> </loginIdentityProviders> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569) at java.lang.StringBuilder.append(StringBuilder.java:190) at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:143) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2370) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2348) at org.apache.commons.io.IOUtils.copy(IOUtils.java:2325) at org.apache.commons.io.IOUtils.copy(IOUtils.java:2273) at org.apache.commons.io.IOUtils.toString(IOUtils.java:1041) at org.apache.commons.io.IOUtils$toString.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133) at org.apache.nifi.properties.ConfigEncryptionTool$_loadFlowXml_closure2$_closure19.doCall(ConfigEncryptionTool.groovy:488) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) at groovy.lang.Closure.call(Closure.java:426) at groovy.lang.Closure.call(Closure.java:442) at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622) at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) stdout: 2017-10-12 23:53:50,252 - Group['ranger'] {} 2017-10-12 23:53:50,253 - Group['hadoop'] {} 2017-10-12 23:53:50,253 - Group['nifi'] {} 2017-10-12 23:53:50,253 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2017-10-12 23:53:50,255 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2017-10-12 23:53:50,255 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2017-10-12 23:53:50,256 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2017-10-12 23:53:50,257 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2017-10-12 23:53:50,257 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger']} 2017-10-12 23:53:50,258 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'nifi']} 2017-10-12 23:53:50,259 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2017-10-12 23:53:50,260 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2017-10-12 23:53:50,265 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if 2017-10-12 23:53:50,278 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} 2017-10-12 23:53:50,284 - Skipping Execute[('setenforce', '0')] due to not_if 2017-10-12 23:53:50,501 - Stack Feature Version Info: stack_version=3.0, version=3.0.1.0-43, current_cluster_version=3.0.1.0-43 -> 3.0.1.0-43 2017-10-12 23:53:50,546 - File['/usr/hdf/current/nifi/bin/nifi-env.sh'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0755} 2017-10-12 23:53:50,548 - Execute['export JAVA_HOME=/usr/jdk64/jdk1.8.0_60;/usr/hdf/current/nifi/bin/nifi.sh stop >> /data/nifi/log/nifi-setup.log'] {'user': 'nifi'} 2017-10-12 23:54:20,757 - Pid file /var/run/nifi/nifi.pid is empty or does not exist 2017-10-12 23:54:20,759 - Directory['/var/run/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,759 - Directory['/data/nifi/log'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,763 - Directory['/data/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,936 - Directory['/data/nifi/database_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,937 - Directory['/data/nifi/flowfile_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,947 - Directory['/data/nifi/provenance_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,957 - Directory['/usr/hdf/current/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,958 - Directory['/data/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,958 - Directory['/data/nifi/state/local'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,959 - Directory['/usr/hdf/current/nifi/lib'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,961 - Directory['{{nifi_content_repo_dir_default}}'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-10-12 23:54:20,962 - Directory['/data/nifi/content_repository'] {'owner': 'nifi', 'group': 'nifi', 'create_parents': True, 'recursive_ownership': True} 2017-10-12 23:54:21,027 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'} 2017-10-12 23:54:21,030 - File['/etc/security/limits.d/nifi.conf'] {'content': Template('nifi.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} 2017-10-12 23:54:21,174 - PropertiesFile['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'group': 'nifi', 'mode': 0600, 'properties': ...} 2017-10-12 23:54:21,178 - Generating properties file: /usr/hdf/current/nifi/conf/nifi.properties 2017-10-12 23:54:21,178 - File['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600} 2017-10-12 23:54:21,258 - Writing File['/usr/hdf/current/nifi/conf/nifi.properties'] because contents don't match 2017-10-12 23:54:21,261 - File['/usr/hdf/current/nifi/conf/bootstrap.conf'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600} 2017-10-12 23:54:21,265 - File['/usr/hdf/current/nifi/conf/logback.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400} 2017-10-12 23:54:21,268 - File['/usr/hdf/current/nifi/conf/state-management.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400} 2017-10-12 23:54:21,273 - File['/usr/hdf/current/nifi/conf/authorizers.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400} 2017-10-12 23:54:21,277 - File['/usr/hdf/current/nifi/conf/login-identity-providers.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600} 2017-10-12 23:54:21,278 - File['/usr/hdf/current/nifi/bin/nifi-env.sh'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0755} 2017-10-12 23:54:21,280 - File['/usr/hdf/current/nifi/conf/bootstrap-notification-services.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0400} 2017-10-12 23:54:21,281 - Encrypting NiFi sensitive configuration properties 2017-10-12 23:54:21,281 - File['/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh'] {'mode': 0755} 2017-10-12 23:54:21,429 - Execute[(u'JAVA_HOME=/usr/jdk64/jdk1.8.0_60', '/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh', '-v', '-b', u'/usr/hdf/current/nifi/conf/bootstrap.conf', '-n', u'/usr/hdf/current/nifi/conf/nifi.properties', '-f', u'/data/nifi/conf/flow.xml.gz', '-s', [PROTECTED], '-l', u'/usr/hdf/current/nifi/conf/login-identity-providers.xml', '-p', [PROTECTED])] {'logoutput': False, 'user': 'nifi'} Command failed after 1 tries ... View more

Hi team, I have followed the below document to configure schema registry using Mysql https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1/bk_installing-hdf/content/database-config-mysql.html but after configured and started registry i am getting below error in registry UI. "An exception was thrown while processing request with message: [com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'registry.schema_metadata_info' doesn't exist]" And when i check the registry database no tables in that. do we need to create tables manually??? kindly help to close this thread. MariaDB [(none)]> use registry; Database changed MariaDB [registry]> show tables; Empty set (0.00 sec) MariaDB [registry]> ... View more