About Hadoop16

Hadoop16 · ‎04-11-2025

Unable to start dfsrouter daemon. Cluster is kerberized and using hadoop 3.3.4. Trying to implement hdfs router based federation. Configured hdfs-rbf-site.xml with required properties. It is a working cluster. Zookeeper quorum is already configured in core-site.xml. Below is the error from log. "Zookeeper connection string cannot be null". ERROR router.FederationUtil - Could not instantiate: ZKDelegationTokenSecretManagerImpl java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hdfs.server.federation.router.FederationUtil.newInstance(FederationUtil.java:164) at org.apache.hadoop.hdfs.server.federation.router.FederationUtil.newSecretManager(FederationUtil.java:224) at org.apache.hadoop.hdfs.server.federation.router.security.RouterSecurityManager.<init>(RouterSecurityManager.java:60) at org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.<init>(RouterRpcServer.java:293) at org.apache.hadoop.hdfs.server.federation.router.Router.createRpcServer(Router.java:391) at org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:188) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69) Caused by: java.lang.NullPointerException: Zookeeper connection string cannot be null at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:899) at org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.<init>(ZKDelegationTokenSecretManager.java:168) at org.apache.hadoop.hdfs.server.federation.router.security.token.ZKDelegationTokenSecretManagerImpl.<init>(ZKDelegationTokenSecretManagerImpl.java:42) ... 12 more Any help is appreciated!

Hadoop16 · ‎03-31-2025

@upadhyayk04 Thank you! I tried with Kerberos enabled on Ranger and Kafka but still policies are downloading fine but not becoming active. I could see below error in Kafka log. DEBUG Failed to get groups for user ANONYMOUS (org.apache.hadoop.security.UserGroupInformation) java.io.IOException: No groups found for user ANONYMOUS at org.apache.hadoop.security.Groups.noGroupsForUser(Groups.java:200)

Hadoop16 · ‎03-30-2025

Kafka policies created in Ranger are getting downloaded but not becoming active. Using Apache Ranger 2.6 and Apache Kafka 3.6. Couldn't find any specific errors related to this issue. Ranger and Kafka are configured with LDAP and no kerberos. What could be the possible issue? Any help is appreciated! Ranger policies for HDFS and Hive works fine. Below are the ldap and ranger configs in Kafka authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer sasl.enabled.mechanisms=PLAIN listener.name.sasl_plaintext.sasl.enabled.mechanisms=PLAIN listener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required; listener.name.sasl_plaintext.plain.sasl.server.callback.handler.class=io.confluent.security.auth.provider.ldap.LdapAuthenticateCallbackHandler ldap.java.naming.provider.url=ldap://<ldap_host>:389 ldap.java.naming.security.authentication=simple ldap.java.naming.security.principal=CN=<bind_user>,OU=Service_Accounts,DC=hadoop,DC=hdp,DC=com ldap.java.naming.security.credentials= ldap.user.name.attribute=sAMAccountName ldap.user.object.class=user ldap.user.search.base=OU=User_Accounts,DC=hadoop,DC=hdp,DC=com;OU=Service_Accounts,DC=hadoop,DC=hdp,DC=com #server properties ldap.java.naming.provider.url=ldap://<ldap_host>:389 ldap.java.naming.security.authentication=simple ldap.java.naming.security.principal=CN=<bind_dn>,OU=Service_Accounts,DC=hadoop,DC=hdp,DC=com ldap.java.naming.security.credentials= ldap.search.mode=GROUPS ldap.user.search.base=OU=User_Accounts,DC=hadoop,DC=hdp,DC=com;OU=Service_Accounts,DC=hadoop,DC=hdp,DC=com ldap.user.object.class=user ldap.user.name.attribute=sAMAccountName ldap.group.search.base=OU=Groups,DC=hadoop,DC=hdp,DC=com ldap.group.object.class=group ldap.group.name.attribute=cn ldap.group.member.attribute=member

Hadoop16 · ‎07-16-2024

I have configure knox in host knox111.abc.hadoop.dc.corp.com and namenodes in nn01.abc.hadoop.dc.corp.com, nn02.abc.hadoop.dc.corp.com. The knoxsso redirection works fine when directly configuring NN with individual Knox host but when using VIP that is configured as knox-hadoop-def.corp.com the redirection to NN doesn't work. I have configured below whitelist setting but still redirection fails. VIP is configured with CA certs and individual knox hosts are configured with self-signed. gateway.dispatch.whitelist = .* knoxsso.redirect.whitelist.regex = .* knoxsso.redirect.whitelist.regex = corp.com Any help is appreciated!

Hadoop16 · ‎05-10-2024

Hello @Scharan From the debug log I think the issue is when Knoxsso is redirecting to NN UI, it is sending user as anonymous. AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(133)) - PrimaryPrincipal: anonymous Do you know what configs at hdfs or Knox could help here?

Hadoop16 · ‎05-10-2024

Hello @Scharan Below are the entries populated when reaching HDFSUI via knoxsso 2024-05-10 14:32:54,143 INFO knox.gateway (AclsAuthorizationFilter.java:init(72)) - Initializing AclsAuthz Provider for: knoxauth 2024-05-10 14:32:54,143 INFO knox.gateway (AclParser.java:parseAcls(50)) - ACLs found for: knoxauth 2024-05-10 14:33:04,139 INFO knox.gateway (KnoxLdapRealm.java:getUserDn(721)) - Computed userDn: CN=lastname\, firstname,OU=XXXX,OU=XXXXX,DC=XXX,DC=XXX,DC=com using ldapSearch for principal: userid 2024-05-10 14:33:04,790 INFO knox.gateway (AclsAuthorizationFilter.java:init(72)) - Initializing AclsAuthz Provider for: KNOXSSO 2024-05-10 14:33:04,790 INFO knox.gateway (AclParser.java:parseAcls(50)) - ACLs found for: KNOXSSO 2024-05-10 14:33:06,030 INFO knox.gateway (CookieUtils.java:getCookiesForName(46)) - Unable to find cookie with name: original-url 2024-05-10 14:33:06,095 INFO service.knoxsso (WebSSOResource.java:addJWTHadoopCookie(386)) - JWT cookie successfully added. 2024-05-10 14:33:06,095 INFO service.knoxsso (WebSSOResource.java:getAuthenticationToken(278)) - About to redirect to original URL: http://NN_host50070/index.html

Hadoop16 · ‎05-10-2024

I was able to resolve the "Invalid redirect" by adding knoxsso.redirect.whitelist.regex but when I enter AD credentials in the KnoxSSO page it keeps redirecting to the same login page. I could see below msgs in gateway.log 2024-05-10 09:04:16,722 DEBUG knox.gateway (AclsAuthorizationFilter.java:doFilter(105)) - Access Granted: true 2024-05-10 09:04:16,760 DEBUG knox.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /images/loading.gif 2024-05-10 09:04:16,761 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(133)) - PrimaryPrincipal: anonymous 2024-05-10 09:04:16,761 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(142)) - PrimaryPrincipal has access: true 2024-05-10 09:04:16,761 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(158)) - Remote IP Address: 2024-05-10 09:04:16,761 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(160)) - Remote IP Address has access: true 2024-05-10 09:04:16,762 DEBUG knox.gateway (AclsAuthorizationFilter.java:doFilter(105)) - Access Granted: true 2024-05-10 09:04:16,795 DEBUG knox.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /redirecting.jsp 2024-05-10 09:04:16,796 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(133)) - PrimaryPrincipal: anonymous 2024-05-10 09:04:16,796 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(142)) - PrimaryPrincipal has access: true 2024-05-10 09:04:16,797 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(158)) - Remote IP Address: 2024-05-10 09:04:16,797 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(160)) - Remote IP Address has access: true 2024-05-10 09:04:16,797 DEBUG knox.gateway (AclsAuthorizationFilter.java:doFilter(105)) - Access Granted: true 2024-05-10 09:04:20,773 DEBUG knox.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /images/loading.gif 2024-05-10 09:04:20,774 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(133)) - PrimaryPrincipal: anonymous 2024-05-10 09:04:20,775 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(142)) - PrimaryPrincipal has access: true 2024-05-10 09:04:20,775 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(158)) - Remote IP Address: 2024-05-10 09:04:20,775 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(160)) - Remote IP Address has access: true 2024-05-10 09:04:20,775 DEBUG knox.gateway (AclsAuthorizationFilter.java:doFilter(105)) - Access Granted: true 2024-05-10 09:04:20,916 DEBUG knox.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /api/v1/websso 2024-05-10 09:04:20,943 DEBUG knox.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /login.html 2024-05-10 09:04:20,944 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(133)) - PrimaryPrincipal: anonymous 2024-05-10 09:04:20,944 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(142)) - PrimaryPrincipal has access: true 2024-05-10 09:04:20,945 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(158)) - Remote IP Address: 2024-05-10 09:04:20,945 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(160)) - Remote IP Address has access: true 2024-05-10 09:04:20,945 DEBUG knox.gateway (AclsAuthorizationFilter.java:doFilter(105)) - Access Granted: true

Hadoop16 · ‎05-09-2024

Configured Knoxsso without Ambari. Knoxsso is configured with Shiro provider and updated the core-site.xml with below configs hadoop.http.authentication.type hadoop.http.authentication.authentication.provider.url hadoop.http.authentication.public.key.pem Followed: https://knox.apache.org/books/knox-1-6-0/user-guide.html#KnoxSSO+Setup+and+Configuration After restart, NN UI is redirecting to KnoxSSO and after entering the AD credentials it is throwing below error in the UI. The redirect to originalUrl looks valid from Knoxsso url. ERROR Invalid Redirect: Possible Phishing Attempt Any help is appreciated!

Hadoop16 · ‎05-08-2024

I'm able to list Knox topologies in Ranger and created Ranger policy for webhdfs. Ran the script ranger-knox-plugin.sh and created ranger-security, audits and policymgr-ssl xmls. Configured one topology for webhdfs in knox and added below <provider> <role>authorization</role> <name>XASecurePDPKnox</name> <enabled>true</enabled> </provider> When I run curl to webhdfs I'm getting 403 error and I could see in Ranger audits access is denied by ranger-acl. I could nail it down to policy not getting downloaded from Ranger but don't see any error in Knox gateway.log even after setting the logging to debug. Knox is enabled with self-signed and ranger with no ssl. I imported knox cert to Ranger cacert. Am I missing any steps in Knox configuration that could be preventing the policy download? Any help is appreciated!

Hadoop16 · ‎03-29-2024

@Scharan Thanks! I figured out the issue, following properties were not configured in hive-site hive.server2.authentication.spnego.keytab hive.server2.authentication.spnego.principal

Online	Offline
Last Visited	‎05-10-2025 07:59 PM

Member Since	‎03-15-2024 09:37 AM
Last Visited	‎05-10-2025 07:59 PM
Posts	14
Kudos received	8

Cloudera Community

Re: Knox SSO to HDFSUI failing

Re: Accessing Hive through Knox receiving HTTP 401...

HDFS router based federation daemon failing to sta...

Re: Kafka policies created in Ranger are not becom...

Kafka policies created in Ranger are not becoming ...

Configure knoxsso hdfs ui redirection to different...

Re: Knox SSO to HDFSUI failing

Re: Knox SSO to HDFSUI failing

Re: Knox SSO to HDFSUI failing

Knox SSO to HDFSUI failing

Knox not downloading Ranger polices

Re: Accessing Hive through Knox receiving HTTP 401...