HTTP Status 403 – Forbidden The server understood the request but refuses to authorize it (GSSException: No valid credentials provided)

[#1]

Hello,

I'm trying to add a new kms service, but the "test connection" is returning this error:

HTTP Status 403 – Forbidden The server understood the request but refuses to authorize it.

GSSException: No valid credentials provided

the users configured in the keytab file are : 

HTTP and ranger-admin for ranger admin server

HTTP and ranger-kms for ranger kms server

this is my configuration:

/opt/ranger-2.3.0-kms/ews/webapp/WEB-INF/classes/conf.dist/kms-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
    <!--
      Licensed under the Apache License, Version 2.0 (the "License");
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License.
    -->
    <configuration>

      <!-- KMS Backend KeyProvider -->

      <property>
        <name>hadoop.kms.key.provider.uri</name>
        <value>dbks://http@{{ template "ranger-kms.ranger-svc-0" . }}:9292/kms</value>
        <description>
          URI of the backing KeyProvider for the KMS.
        </description>
      </property>

      <property>
        <name>hadoop.security.keystore.JavaKeyStoreProvider.password</name>
        <value>none</value>
        <description>
          If using the JavaKeyStoreProvider, the password for the keystore file.
        </description>
      </property>

      <!-- KMS Cache -->

      <property>
        <name>hadoop.kms.cache.enable</name>
        <value>true</value>
        <description>
          Whether the KMS will act as a cache for the backing KeyProvider.
          When the cache is enabled, operations like getKeyVersion, getMetadata,
          and getCurrentKey will sometimes return cached data without consulting
          the backing KeyProvider. Cached values are flushed when keys are deleted
          or modified.
        </description>
      </property>

      <property>
        <name>hadoop.kms.cache.timeout.ms</name>
        <value>600000</value>
        <description>
          Expiry time for the KMS key version and key metadata cache, in
          milliseconds. This affects getKeyVersion and getMetadata.
        </description>
      </property>

      <property>
        <name>hadoop.kms.current.key.cache.timeout.ms</name>
        <value>30000</value>
        <description>
          Expiry time for the KMS current key cache, in milliseconds. This
          affects getCurrentKey operations.
        </description>
      </property>

      <!-- KMS Audit -->

      <property>
        <name>hadoop.kms.audit.aggregation.window.ms</name>
        <value>10000</value>
        <description>
          Duplicate audit log events within the aggregation window (specified in
          ms) are quashed to reduce log traffic. A single message for aggregated
          events is printed at the end of the window, along with a count of the
          number of aggregated events.
        </description>
      </property>

      <!-- KMS Security -->

      <property>
        <name>hadoop.kms.authentication.type</name>
        <value>kerberos</value>
        <description>
          Authentication type for the KMS. Can be either &quot;simple&quot;
          or &quot;kerberos&quot;.
        </description>
      </property>

      <property>
        <name>hadoop.kms.authentication.kerberos.keytab</name>
        <value>/etc/security/ranger-kms.keytab</value>
        <description>
          Path to the keytab with credentials for the configured Kerberos principal.
        </description>
      </property>

      <property>
        <name>hadoop.kms.authentication.kerberos.principal</name>
        <value>HTTP/_HOST@LABP7.CNAM</value>
        <description>
          The Kerberos principal to use for the HTTP endpoint.
          The principal must start with 'HTTP/' as per the Kerberos HTTP SPNEGO specification.
        </description>
      </property>

      <property>
        <name>hadoop.kms.authentication.kerberos.name.rules</name>
        <value>DEFAULT</value>
        <description>
          Rules used to resolve Kerberos principal names.
        </description>
      </property>

      <!-- Authentication cookie signature source -->

      <property>
        <name>hadoop.kms.authentication.signer.secret.provider</name>
        <value>random</value>
        <description>
          Indicates how the secret to sign the authentication cookies will be
          stored. Options are 'random' (default), 'string' and 'zookeeper'.
          If using a setup with multiple KMS instances, 'zookeeper' should be used.
        </description>
      </property>

      <property>
            <name>hadoop.kms.security.authorization.manager</name>
            <value>org.apache.ranger.authorization.kms.authorizer.RangerKmsAuthorizer</value>
      </property>

      <property>
            <name>hadoop.kms.proxyuser.ranger-kms.groups</name>
            <value>*</value>
      </property>
      <property>
            <name>hadoop.kms.proxyuser.ranger-kms.hosts</name>
            <value>*</value>
      </property>
      <property>
            <name>hadoop.kms.proxyuser.ranger-kms.users</name>
            <value>*</value>
      </property>
    </configuration>

install.properties:

# Licensed to the Apache Software Foundation (ASF) under one or more
    # contributor license agreements.  See the NOTICE file distributed with
    # this work for additional information regarding copyright ownership.
    # The ASF licenses this file to You under the Apache License, Version 2.0
    # (the "License"); you may not use this file except in compliance with
    # the License.  You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.

    #
    # This file provides a list of the deployment variables for the Ranger KMS Web Application
    #

    #------------------------- DB CONFIG - BEGIN ----------------------------------
    # Uncomment the below if the DBA steps need to be run separately
    #setup_mode=SeparateDBA

    PYTHON_COMMAND_INVOKER=python3

    #DB_FLAVOR=MYSQL|ORACLE|POSTGRES|MSSQL|SQLA
    DB_FLAVOR=POSTGRES

    #
    # Location of DB client library (please check the location of the jar file)
    #
    #SQL_CONNECTOR_JAR=/usr/share/java/ojdbc6.jar
    #SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar
    SQL_CONNECTOR_JAR=/usr/share/java/postgresql.jar
    #SQL_CONNECTOR_JAR=/usr/share/java/sqljdbc4.jar
    #SQL_CONNECTOR_JAR=/opt/sqlanywhere17/java/sajdbc4.jar
    #SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar


    #
    # DB password for the DB admin user-id
    # **************************************************************************
    # ** If the password is left empty or not-defined here,
    # ** it will be prompted to enter the password during installation process
    # **************************************************************************
    #
    #db_root_user=root|SYS|postgres|sa|dba
    #db_host=host:port              # for DB_FLAVOR=MYSQL|POSTGRES|SQLA|MSSQL       #for example: db_host=localhost:3306
    #db_host=host:port:SID          # for DB_FLAVOR=ORACLE                          #for SID example: db_host=localhost:1521:ORCL
    #db_host=host:port/ServiceName  # for DB_FLAVOR=ORACLE                          #for Service example: db_host=localhost:1521/XE
    #db_host=host:port:GL           # for DB_FLAVOR=ORACLE                          #for TNSNAME example: db_host=localhost:1521:GL
    db_root_user=postgres
    db_root_password=$POSTGRES_PASSWORD
    db_host={{ include "postgresql.primary.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local:{{ include "postgresql.service.port" . }}
    #SSL config
    db_ssl_enabled=false
    db_ssl_required=false
    db_ssl_verifyServerCertificate=false
    #db_ssl_auth_type=1-way|2-way, where 1-way represents standard one way ssl authentication and 2-way represents mutual ssl authentication
    db_ssl_auth_type=2-way
    javax_net_ssl_keyStore=
    javax_net_ssl_keyStorePassword=
    javax_net_ssl_trustStore=
    javax_net_ssl_trustStorePassword=
    javax_net_ssl_trustStore_type=jks
    javax_net_ssl_keyStore_type=jks

    # For postgresql db
    db_ssl_certificate_file=

    #
    # DB UserId used for the Ranger KMS schema
    #
    db_name=ranger_kms
    db_user=postgres
    db_password=$POSTGRES_PASSWORD

    XAAUDIT.DB.IS_ENABLED=true
    XAAUDIT.DB.FLAVOUR=POSTGRES
    XAAUDIT.DB.HOSTNAME={{ include "postgresql.primary.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local:{{ include "postgresql.service.port" . }}
    XAAUDIT.DB.DATABASE_NAME=ranger_kms_audit
    XAAUDIT.DB.USER_NAME=postgres
    XAAUDIT.DB.PASSWORD=$POSTGRES_PASSWORD

    #For over-riding the jdbc url.
    is_override_db_connection_string=false
    db_override_connection_string=


    #------------------------- DB CONFIG - END ----------------------------------
    #KMS Server config
    ranger_kms_http_enabled=true
    ranger_kms_https_keystore_file=
    ranger_kms_https_keystore_keyalias=rangerkms
    ranger_kms_https_keystore_password=

    #------------------------- RANGER KMS Install Dir ------------------
    COMPONENT_INSTALL_DIR_NAME=/etc/ranger/install

    #------------------------- RANGER KMS Master Key Crypt Key ------------------
    KMS_MASTER_KEY_PASSWD=labp7kmsadmin

    #------------------------- Ranger KMS Kerberos Configuration ---------------------------
    kms_principal=ranger-kms/{{ template "ranger-kms.ranger-svc-0" . }}@LABP7.CNAM
    kms_keytab=/etc/security/ranger-kms.keytab
    hadoop_conf=/etc/hadoop/conf/

    #------------------------- Ranger KMS HSM CONFIG ------------------------------
    HSM_TYPE=LunaProvider
    HSM_ENABLED=false
    HSM_PARTITION_NAME=par19
    HSM_PARTITION_PASSWORD=S@fenet123

    #------------------------- Ranger SAFENET KEYSECURE CONFIG ------------------------------
    KEYSECURE_ENABLED=false
    KEYSECURE_USER_PASSWORD_AUTHENTICATION=true
    KEYSECURE_MASTERKEY_NAME=safenetkeysecure
    KEYSECURE_USERNAME=user1
    KEYSECURE_PASSWORD=t1e2s3t4
    KEYSECURE_HOSTNAME=SunPKCS11-keysecurehn
    KEYSECURE_MASTER_KEY_SIZE=256
    KEYSECURE_LIB_CONFIG_PATH=/opt/safenetConf/64/8.3.1/sunpkcs11.cfg

    #------------------------- Ranger Azure Key Vault ------------------------------
    AZURE_KEYVAULT_ENABLED=false
    AZURE_KEYVAULT_SSL_ENABLED=false
    AZURE_CLIENT_ID=50fd7ca6-fd4f-4785-a13f-1a6cc4e95e42
    AZURE_CLIENT_SECRET=<AzureKeyVaultPassword>
    AZURE_AUTH_KEYVAULT_CERTIFICATE_PATH=/home/machine/Desktop/azureAuthCertificate/keyvault-MyCert.pfx
    # Initialize below prop if your certificate file has any password
    #AZURE_AUTH_KEYVAULT_CERTIFICATE_PASSWORD=certPass
    AZURE_MASTERKEY_NAME=RangerMasterKey
    # E.G. RSA, RSA_HSM, EC, EC_HSM, OCT
    AZURE_MASTER_KEY_TYPE=RSA
    # E.G. RSA_OAEP, RSA_OAEP_256, RSA1_5, RSA_OAEP
    ZONE_KEY_ENCRYPTION_ALGO=RSA_OAEP
    AZURE_KEYVAULT_URL=https://shahkeyvault.vault.azure.net/

    #------------------------- Ranger Google Cloud HSM ------------------------------
    IS_GCP_ENABLED=false
    GCP_KEYRING_ID=
    GCP_CRED_JSON_FILE=/full/path/to/credfile.json
    GCP_PROJECT_ID=
    GCP_LOCATION_ID=
    GCP_MASTER_KEY_NAME=MyMasterKeyNameChangeIt

    #------------------------- Ranger Tencent KMS ------------------------------
    TENCENT_KMS_ENABLED=false
    TENCENT_MASTERKEY_ID=b756b016-6e11-11ec-a735-525400fe0300
    TENCENT_CLIENT_ID=AKIDrXx6ybx2qNdiaBWaNs76pGQJvFJ6crpW
    TENCENT_CLIENT_SECRET=<TencentSecretKey>
    TENCENT_CLIENT_REGION=ap-beijing

    # ------- UNIX User CONFIG ----------------
    #
    unix_user=root
    unix_user_pwd=
    unix_group=root
    #
    # ------- UNIX User CONFIG  - END ----------------
    #

    #
    # -------
    #

    # Location of Policy Manager URL
    #
    # Example:
    # POLICY_MGR_URL=http://policymanager.xasecure.net:6080
    #
    POLICY_MGR_URL=http://{{ template "ranger-admin.ranger-svc-0" . }}:6080

    #
    # This is the repository name created within policy manager
    #
    # Example:
    # REPOSITORY_NAME=kmsdev
    #
    REPOSITORY_NAME=kmsdev

    # AUDIT configuration with V3 properties

    #Should audit be summarized at source
    XAAUDIT.SUMMARY.ENABLE=false

    # Enable audit logs to Solr
    #Example
    #XAAUDIT.SOLR.ENABLE=true
    #XAAUDIT.SOLR.URL=http://localhost:6083/solr/ranger_audits
    #XAAUDIT.SOLR.ZOOKEEPER=
    #XAAUDIT.SOLR.FILE_SPOOL_DIR=/var/log/ranger/kms/audit/solr/spool

    XAAUDIT.SOLR.ENABLE=false
    XAAUDIT.SOLR.URL=NONE
    XAAUDIT.SOLR.USER=NONE
    XAAUDIT.SOLR.PASSWORD=NONE
    XAAUDIT.SOLR.ZOOKEEPER=NONE
    XAAUDIT.SOLR.FILE_SPOOL_DIR=/var/log/ranger/kms/audit/solr/spool

    # Enable audit logs to ElasticSearch
    #Example
    #XAAUDIT.ELASTICSEARCH.ENABLE=true
    #XAAUDIT.ELASTICSEARCH.URL=localhost
    #XAAUDIT.ELASTICSEARCH.INDEX=audit

    XAAUDIT.ELASTICSEARCH.ENABLE=false
    XAAUDIT.ELASTICSEARCH.URL=NONE
    XAAUDIT.ELASTICSEARCH.USER=NONE
    XAAUDIT.ELASTICSEARCH.PASSWORD=NONE
    XAAUDIT.ELASTICSEARCH.INDEX=NONE
    XAAUDIT.ELASTICSEARCH.PORT=NONE
    XAAUDIT.ELASTICSEARCH.PROTOCOL=NONE

    # Enable audit logs to HDFS
    #Example
    #XAAUDIT.HDFS.ENABLE=true
    #XAAUDIT.HDFS.HDFS_DIR=hdfs://node-1.example.com:8020/ranger/audit
    #  If using Azure Blob Storage
    #XAAUDIT.HDFS.HDFS_DIR=wasb[s]://<containername>@<accountname>.blob.core.windows.net/<path>
    #XAAUDIT.HDFS.HDFS_DIR=wasb://ranger_audit_container@my-azure-account.blob.core.windows.net/ranger/audit
    #XAAUDIT.HDFS.FILE_SPOOL_DIR=/var/log/ranger/kms/audit/hdfs/spool

    XAAUDIT.HDFS.ENABLE=false
    XAAUDIT.HDFS.HDFS_DIR=hdfs://__REPLACE__NAME_NODE_HOST:8020/ranger/audit
    XAAUDIT.HDFS.FILE_SPOOL_DIR=/var/log/ranger/kms/audit/hdfs/spool

    # Following additional propertis are needed When auditing to Azure Blob Storage via HDFS
    # Get these values from your /etc/hadoop/conf/core-site.xml
    #XAAUDIT.HDFS.HDFS_DIR=wasb[s]://<containername>@<accountname>.blob.core.windows.net/<path>
    XAAUDIT.HDFS.AZURE_ACCOUNTNAME=__REPLACE_AZURE_ACCOUNT_NAME
    XAAUDIT.HDFS.AZURE_ACCOUNTKEY=__REPLACE_AZURE_ACCOUNT_KEY
    XAAUDIT.HDFS.AZURE_SHELL_KEY_PROVIDER=__REPLACE_AZURE_SHELL_KEY_PROVIDER
    XAAUDIT.HDFS.AZURE_ACCOUNTKEY_PROVIDER=__REPLACE_AZURE_ACCOUNT_KEY_PROVIDER

    #Log4j Audit Provider
    XAAUDIT.LOG4J.ENABLE=false
    XAAUDIT.LOG4J.IS_ASYNC=false
    XAAUDIT.LOG4J.ASYNC.MAX.QUEUE.SIZE=10240
    XAAUDIT.LOG4J.ASYNC.MAX.FLUSH.INTERVAL.MS=30000
    XAAUDIT.LOG4J.DESTINATION.LOG4J=true
    XAAUDIT.LOG4J.DESTINATION.LOG4J.LOGGER=xaaudit

    # Enable audit logs to Amazon CloudWatch Logs
    #Example
    #XAAUDIT.AMAZON_CLOUDWATCH.ENABLE=true
    #XAAUDIT.AMAZON_CLOUDWATCH.LOG_GROUP=ranger_audits
    #XAAUDIT.AMAZON_CLOUDWATCH.LOG_STREAM={instance_id}
    #XAAUDIT.AMAZON_CLOUDWATCH.FILE_SPOOL_DIR=/var/log/hive/audit/amazon_cloudwatch/spool

    XAAUDIT.AMAZON_CLOUDWATCH.ENABLE=false
    XAAUDIT.AMAZON_CLOUDWATCH.LOG_GROUP=NONE
    XAAUDIT.AMAZON_CLOUDWATCH.LOG_STREAM_PREFIX=NONE
    XAAUDIT.AMAZON_CLOUDWATCH.FILE_SPOOL_DIR=NONE
    XAAUDIT.AMAZON_CLOUDWATCH.REGION=NONE

    # End of V3 properties


    #
    #  Audit to HDFS Configuration
    #
    # If XAAUDIT.HDFS.IS_ENABLED is set to true, please replace tokens
    # that start with __REPLACE__ with appropriate values
    #  XAAUDIT.HDFS.IS_ENABLED=true
    #  XAAUDIT.HDFS.DESTINATION_DIRECTORY=hdfs://__REPLACE__NAME_NODE_HOST:8020/ranger/audit/%app-type%/%time:yyyyMMdd%
    #  XAAUDIT.HDFS.LOCAL_BUFFER_DIRECTORY=__REPLACE__LOG_DIR/kms/audit
    #  XAAUDIT.HDFS.LOCAL_ARCHIVE_DIRECTORY=__REPLACE__LOG_DIR/kms/audit/archive
    #
    #
    # Example:
    #  XAAUDIT.HDFS.IS_ENABLED=true
    #  XAAUDIT.HDFS.DESTINATION_DIRECTORY=hdfs://namenode.example.com:8020/ranger/audit/%app-type%/%time:yyyyMMdd%
    #  XAAUDIT.HDFS.LOCAL_BUFFER_DIRECTORY=/var/log/kms/audit
    #  XAAUDIT.HDFS.LOCAL_ARCHIVE_DIRECTORY=/var/log/kms/audit/archive
    #
    XAAUDIT.HDFS.IS_ENABLED=false
    XAAUDIT.HDFS.DESTINATION_DIRECTORY=hdfs://__REPLACE__NAME_NODE_HOST:8020/ranger/audit/%app-type%/%time:yyyyMMdd%
    XAAUDIT.HDFS.LOCAL_BUFFER_DIRECTORY=__REPLACE__LOG_DIR/kms/audit
    XAAUDIT.HDFS.LOCAL_ARCHIVE_DIRECTORY=__REPLACE__LOG_DIR/kms/audit/archive

    XAAUDIT.HDFS.DESTINTATION_FILE=%hostname%-audit.log
    XAAUDIT.HDFS.DESTINTATION_FLUSH_INTERVAL_SECONDS=900
    XAAUDIT.HDFS.DESTINTATION_ROLLOVER_INTERVAL_SECONDS=86400
    XAAUDIT.HDFS.DESTINTATION_OPEN_RETRY_INTERVAL_SECONDS=60
    XAAUDIT.HDFS.LOCAL_BUFFER_FILE=%time:yyyyMMdd-HHmm.ss%.log
    XAAUDIT.HDFS.LOCAL_BUFFER_FLUSH_INTERVAL_SECONDS=60
    XAAUDIT.HDFS.LOCAL_BUFFER_ROLLOVER_INTERVAL_SECONDS=600
    XAAUDIT.HDFS.LOCAL_ARCHIVE_MAX_FILE_COUNT=10

    #Solr Audit Provider
    XAAUDIT.SOLR.IS_ENABLED=false
    XAAUDIT.SOLR.MAX_QUEUE_SIZE=1
    XAAUDIT.SOLR.MAX_FLUSH_INTERVAL_MS=1000
    XAAUDIT.SOLR.SOLR_URL=http://localhost:6083/solr/ranger_audits

    # End of V2 properties

    #
    # SSL Client Certificate Information
    #
    # Example:
    # SSL_KEYSTORE_FILE_PATH=/etc/ranger/kms/conf/ranger-plugin-keystore.jks
    # SSL_KEYSTORE_PASSWORD=none
    # SSL_TRUSTSTORE_FILE_PATH=/etc/ranger/kms/conf/ranger-plugin-truststore.jks
    # SSL_TRUSTSTORE_PASSWORD=none
    #
    # You do not need use SSL between agent and security admin tool, please leave these sample value as it is.
    #
    SSL_KEYSTORE_FILE_PATH=/etc/ranger/kms/conf/ranger-plugin-keystore.jks
    SSL_KEYSTORE_PASSWORD=myKeyFilePassword
    SSL_TRUSTSTORE_FILE_PATH=/etc/ranger/kms/conf/ranger-plugin-truststore.jks
    SSL_TRUSTSTORE_PASSWORD=changeit

    # Custom log directory path
    RANGER_KMS_LOG_DIR=$PWD

    #PID file path
    RANGER_KMS_PID_DIR_PATH=/var/run/ranger_kms
    # #################  DO NOT MODIFY ANY VARIABLES BELOW #########################
    #
    # --- These deployment variables are not to be modified unless you understand the full impact of the changes
    #
    ################################################################################
    KMS_DIR=$PWD
    app_home=$PWD/ews/webapp
    TMPFILE=$PWD/.fi_tmp
    LOGFILE=$PWD/logfile

    JAVA_BIN='java'
    JAVA_VERSION_REQUIRED='1.8'
    JAVA_ORACLE='Java(TM) SE Runtime Environment'

    mysql_core_file=db/mysql/kms_core_db.sql

    oracle_core_file=db/oracle/kms_core_db_oracle.sql

    postgres_core_file=db/postgres/kms_core_db_postgres.sql

    sqlserver_core_file=db/sqlserver/kms_core_db_sqlserver.sql

    sqlanywhere_core_file=db/sqlanywhere/kms_core_db_sqlanywhere.sql
    cred_keystore_filename=$app_home/WEB-INF/classes/conf/.jceks/rangerkms.jceks

    KMS_BLACKLIST_DECRYPT_EEK=hdfs

ps: all the environment is kerberized 

Thank you for your help 😊

[#2]

@Adel-Mejri, Welcome to our community! To help you get the best possible answer, I have tagged in our Ranger experts @Scharan @Kartik_Agarwal @vamsi_redd @xsmehul  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

Hello, any help will be greatly appreciated @Scharan @Kartik_Agarwal @vamsi_redd @xsmehul 

Thank you