Member since
01-10-2020
52
Posts
1
Kudos Received
3
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2475 | 08-25-2020 09:30 AM | |
| 3531 | 08-21-2020 08:31 PM | |
| 2668 | 05-28-2020 03:53 PM |
08-25-2020
09:30 AM
@BGabor thanks for your response I'm trying to understand the difference between two blocks given in the document #BLOCK1 activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealm
activeDirectoryRealm.systemUsername = userNameA
activeDirectoryRealm.systemPassword = passwordA
activeDirectoryRealm.searchBase = CN=Users,DC=SOME_GROUP,DC=COMPANY,DC=COM
activeDirectoryRealm.url = ldap://ldap.test.com:389
activeDirectoryRealm.groupRolesMap = "CN=aGroupName,OU=groups,DC=SOME_GROUP,DC=COMPANY,DC=COM":"group1"
activeDirectoryRealm.authorizationCachingEnabled = false
activeDirectoryRealm.principalSuffix = @corp.company.net
##BLOCK2
ldapRealm = org.apache.zeppelin.realm.LdapGroupRealm
# search base for ldap groups (only relevant for LdapGroupRealm):
ldapRealm.contextFactory.environment[ldap.searchBase] = dc=COMPANY,dc=COM
ldapRealm.contextFactory.url = ldap://ldap.test.com:389
ldapRealm.userDnTemplate = uid={0},ou=Users,dc=COMPANY,dc=COM
ldapRealm.contextFactory.authenticationMechanism = simple also define roles/groups that you want to have in the system, like below; [roles]
admin = *
hr = *
finance = *
group1 = * . could you help me understand? i want to extract only one group and users from AD, which block should I use. thanks
... View more
08-24-2020
03:16 PM
please suggest if anyone has successfully integrated with AD. I'm trying to use below configs and it's not working ldapRealm= org.apache.zeppelin.realm.LdapRealm ldapRealm.contextFactory.url = ldap://ad.abc.com:389 ldapRealm.contextFactory.authenticationMechanism = simple ldapRealm.contextFactory.systemUsername = svc_abc ldapRealm.contextFactory.systemPassword = passwdddd #ldapRealm.searchBase = DC=abc,DC=abc,DC=COM ldapRealm.userSearchBase = OU=IDM,DC=abc,DC=abc,DC=com ldapRealm.userSearchScope = subtree ldapRealm.userSearchAttributeName = sAMAccountName ldapRealm.userSearchFilter = (&(objectclass=person)(sAMAccountName={0})) ldapRealm.authorizationEnabled = true ldapRealm.groupSearchBase = OU=abcGroups,DC=abc,DC=abc,DC=com ldapRealm.groupObjectClass = group ldapRealm.memberAttribute= member ldapRealm.groupSearchScope = subtree ldapRealm.groupSearchFilter = (&(objectclass=group)(member={0})) ldapRealm.memberAttributeValueTemplate= $CN=g_app_zep,OU=abcGroups,DC=abc,DC=abc,DC=com ldapRealm.groupSearchEnableMatchingRuleInChain = true ldapRealm.rolesByGroup = Zeppelin_Admin: admin_role ldapRealm.allowedRolesForAuthentication = admin_role,user_role ERROR: WARN [2020-08-24 16:31:14,497] ({main} WebAppContext.java[doStart]:554) - Failed startup of context o.e.j.w.WebAppContext@22635ba0{zeppelin-web,/,file:///opt/zepplin/zeppelin-0.8.2-bin-all/webapps/webapp/,UNAVAILABLE}{/opt/zepplin/zeppelin-0.8.2-bin-all/zeppelin-web-0.8.2.war} org.apache.shiro.config.UnresolveableReferenceException: The object with id [CN=g_app_zep,OU=abcGroups,DC=abc,DC=abc,DC=com] has not yet been defined and therefore cannot be referenced. Please ensure objects are defined in the order in which they should be created and made available for future reference. please share if there is any working configs
... View more
Labels:
- Labels:
-
Apache Hadoop
08-21-2020
08:31 PM
1 Kudo
figured out the issue, related to jira https://issues.apache.org/jira/browse/NIFI-6999 flow.xml.gz is 963 mb, where toolkit mem isn't enough to restart from ambari GUI. hence started through CLI restarted the nifi from CLI rather than AMBARI , came up flow.xml.gz file size is 963MB instance : 120GB memory NIFI allocated mem: 45GB toolkit 15GB mem error: from ambari agent logs ERROR:020/08/21 19:01:30 ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.lang.StringCoding.encode(StringCoding.java:350) at java.lang.String.getBytes(String.java:941) at org.apache.commons.io.IOUtils.write(IOUtils.java:2025) at org.apache.commons.io.IOUtils$write$0.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141) at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6$_closure30.doCall(ConfigEncryptionTool.groovy:870) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at groovy.lang.Closure.call(Closure.java:414) at groovy.lang.Closure.call(Closure.java:430) at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622) at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54) at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6.doCall(ConfigEncryptionTool.groovy:869) Requested array size exceeds VM limit
... View more
08-21-2020
06:02 PM
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in <module>
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 143, in start
self.configure(env, is_starting = True)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 111, in configure
self.write_configurations(params, is_starting)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 247, in write_configurations
support_encrypt_authorizers=params.stack_support_encrypt_authorizers
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util_common.py", line 574, in encrypt_sensitive_properties
Execute(encrypt_config_command, user=nifi_user, logoutput=False, environment=environment)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /grid/var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -a /usr/hdf/current/nifi/conf/authorizers.xml -m -e '[PROTECTED]' -p '[PROTECTED]'' returned 255. 2020/08/21 18:00:37 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2020/08/21 18:00:37 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten
2020/08/21 18:00:37 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of login-identity-providers.xml
2020/08/21 18:00:37 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source login-identity-providers.xml and destination login-identity-providers.xml are identical [/usr/hdf/current/nifi/conf/login-identity-providers.xml] so the original will be overwritten
2020/08/21 18:00:37 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml
2020/08/21 18:00:37 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten
2020/08/21 18:00:37 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz
2020/08/21 18:00:37 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/grid/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten
... View more
08-21-2020
03:26 PM
here is the error message resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /grid/var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -a /usr/hdf/current/nifi/conf/authorizers.xml -m -e '[PROTECTED]' -p '[PROTECTED]'' returned 255. 2020/08/21 09:07:10 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties 2020/08/21 09:07:10 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten 2020/08/21 09:07:10 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of login-identity-providers.xml 2020/08/21 09:07:10 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source login-identity-providers.xml and destination login-identity-providers.xml are identical [/usr/hdf/current/nifi/conf/login-identity-providers.xml] so the original will be overwritten 2020/08/21 09:07:10 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml 2020/08/21 09:07:10 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten 2020/08/21 09:07:10 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz 2020/08/21 09:07:10 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/grid/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten
... View more
08-21-2020
02:37 PM
NIFI flow.xml.gz is getting cleared out during NIFI restart HDF 3.4 NIFI 1.9 no configs have been changes but when restarted the existing flwo.xml.gz of size 70MB is getting overwritten. though restored from a backup and placed back in, again after restarting it's getting cleared. @MattWho please advice
... View more
Labels:
- Labels:
-
Apache NiFi
07-10-2020
12:49 AM
Thanks on hdf 3.4.1. cleared the users from the auth.xml and users.xml. back to normal operations. Appreciate your solution.
... View more
07-02-2020
12:54 PM
noticed below error from users.log but nothing changed from nifi configs but started noticing this error nifi-user.log:2020-07-02 12:16:13,857 INFO [NiFi Web Server-343673] o.a.n.w.a.c.ResourceNotFoundExceptionMapper org.apache.nifi.web.ResourceNotFoundException: Unable to find user with id '938eb61e-bbc4-383a-8475-aee80541b5a5'.. Returning Not Found response. @MattWho
... View more
Labels:
- Labels:
-
Apache NiFi
05-28-2020
03:53 PM
I am able to resolve the issue. Update nifi authorizers to use ranger ldap rather than file based.
... View more