Member since
06-24-2016
5
Posts
0
Kudos Received
0
Solutions
12-22-2024
05:33 AM
1 Kudo
@polingsky202 To configure HAProxy to connect three Kafka brokers with Kerberos authentication and resolve the Authentication failed due to invalid credentials with SASL mechanism GSSAPI error, follow these steps: Step 1: Review the Current Configuration The provided configuration shows: HAProxy is configured for load balancing using roundrobin. Kafka brokers are set up with: advertised.listeners and listeners for internal and LB connections. SASL GSSAPI configured with Kerberos. Issue Likely Causes: Kerberos principal or keytab file mismatch. Improper mapping of advertised listener names. Client-side misconfiguration for Kerberos authentication. Step 2: Correct and Optimize HAProxy Configuration Update the HAProxy configuration to correctly pass Kerberos authentication to Kafka brokers. Updated haproxy.cfg listen kafka bind *:6677 mode tcp balance roundrobin option tcp-check server kafka1 kafka-1.kafka.net:6668 check server kafka2 kafka-2.kafka.net:6669 check server kafka3 kafka-3.kafka.net:6666 check Key updates above in the haproxy config file: Mode TCP: Ensures TCP passthrough for Kerberos authentication. Option tcp-check: Validates backend server availability. Step 3: Verify Kafka Broker Configuration Ensure the Kerberos configuration for each broker is consistent and properly aligned. Key Points: advertised.listeners: Ensure the LB listener matches the address clients will connect to via HAProxy (e.g. gateway.kafka.net). Kerberos JAAS Configuration: Validate the listener.name.LB.gssapi.sasl.jaas.config entry for all brokers. Ensure the keyTab file exists and has correct permissions: ls -l /etc/security/keytabs/kafka.service.keytab Example Updated kafka1 Broker Configuration: advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6668 listeners=INTERNAL://:6667,LB://:6668 listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT inter.broker.listener.name=INTERNAL listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ doNotPrompt=true useKeyTab=true storeKey=true \ keyTab="/etc/security/keytabs/kafka.service.keytab" \ principal="kafka/gateway.kafka.net@KAFKA.NET"; Repeat similar updates for kafka2 and kafka3 with their respective listener ports. Step 4: Update Kerberos Configuration Ensure that Kerberos configuration is consistent across all systems. Validate Kerberos krb5.conf ensure the file includes the correct realm and KDC information: [libdefaults] default_realm = KAFKA.NET [realms] KAFKA.NET = { kdc = your-kdc-host admin_server = your-kdc-admin-host } 2. Test Kerberos Principal: Verify the principal works with the keytab: kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/gateway.kafka.net@KAFKA.NET Step 5: Verify Client Configuration The client is attempting to authenticate with Kerberos. Ensure the producer properties are configured correctly updated Producer Command: see below /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh \ --topic my-topic \ --broker-list gateway.kafka.net:6677 \ --producer-property security.protocol=SASL_PLAINTEXT \ --producer-property sasl.kerberos.service.name=kafka Key Properties: security.protocol=SASL_PLAINTEXT: Specifies Kerberos authentication. sasl.kerberos.service.name=kafka: Matches the Kerberos principal’s service name. Step 6: Test and Troubleshoot Enable Debug Logging: Add -Dsun.security.krb5.debug=true to the JVM options for the client to debug Kerberos issues export KAFKA_OPTS="-Dsun.security.krb5.debug=true" Check Logs: On the client side, check for detailed Kerberos errors in the output. On Kafka brokers, inspect logs for authentication errors: less /var/log/kafka/server.log 3. Verify Connectivity: Use telnet or nc to confirm connectivity to HAProxy and brokers telnet gateway.kafka.net 6677 telnet kafka-1.kafka.net 6668 Final Checklist Ensure all brokers have consistent Kerberos configurations. Verify the client-side security.protocol and sasl.kerberos.service.name settings. Ensure HAProxy uses TCP passthrough (mode tcp) for Kerberos. With these adjustments, the Kerberos authentication error should be resolved. Let me know if further clarification is needed! Happy hadooping
... View more
04-21-2024
11:33 PM
hi @polingsky202 could you please paste the runtime error stack
... View more
03-16-2022
05:50 PM
I try to change the composite-user-group-provider to file-user-group-provider, and it worked! authorizers.xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=nifi, OU=NIFI</property>
<property name="Initial User Identity 2">nifi</property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">nifi</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=nifi, OU=NIFI</property>
<property name="Node Identity 2">nifi</property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers> But I still have some questions, if ldap-user-group-provider or composite-user-group-provider can not used for secure cluster ?
... View more
08-07-2017
08:25 PM
Hi, Thanks for this guide i followed exacty how you explained but i still getting this error Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, URL: http://loadbalancer_FQDN:11000/oozie/versions?user.name=oozie, status: 403, message: Forbidden Any idea Thanks
... View more