Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (2)

1. HA provider for webhdfs is needed in your topology.

<provider>
   <role>ha</role>
   <name>HaProvider</name>
   <enabled>true</enabled>
   <param>
      <name>WEBHDFS</name>
      <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value>
   </param>
</provider>

2. The namenode service url value should contain your name service ID. (This can be found in your hdfs-default.xml under parameter dfs.internal.nameservices)

<service>
   <role>NAMENODE</role>
   <url>hdfs://chupa</url>
</service>

3. Make sure webhdfs url for each namenode is added in your WEBHDFS service area.

<service>
    <role>WEBHDFS</role>
    <url>http://chupa1.openstacklocal:50070/webhdfs</url>
    <url>http://chupa2.openstacklocal:50070/webhdfs</url>
</service>

4. Here is a working topology using the knox default demo LDAP.

<topology>
    <gateway>
        <provider>
            <role>authentication</role>
            <name>ShiroProvider</name>
            <enabled>true</enabled>
            <param>
                <name>sessionTimeout</name>
                <value>30</value>
            </param>
            <param>
                <name>main.ldapRealm</name>
                <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
            </param>
            <param>
                <name>main.ldapRealm.userDnTemplate</name>
                <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory.url</name>
                <value>ldap://chupa1.openstacklocal:33389</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
                <value>simple</value>
            </param>
            <param>
                <name>urls./**</name>
                <value>authcBasic</value>
            </param>
        </provider>
        <provider>
            <role>identity-assertion</role>
            <name>Default</name>
            <enabled>true</enabled>
        </provider>
        <provider>
            <role>authorization</role>
            <name>XASecurePDPKnox</name>
            <enabled>true</enabled>
        </provider>
        <provider>
            <role>ha</role>
            <name>HaProvider</name>
            <enabled>true</enabled>
            <param>
                <name>WEBHDFS</name>
                <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value>
            </param>
        </provider>
    </gateway>
    <service>
        <role>NAMENODE</role>
        <url>hdfs://chupa</url>
    </service>
    <service>
        <role>JOBTRACKER</role>
        <url>rpc://chupa3.openstacklocal:8050</url>
    </service>
    <service>
        <role>WEBHDFS</role>
        <url>http://chupa1.openstacklocal:50070/webhdfs</url>
        <url>http://chupa2.openstacklocal:50070/webhdfs</url>
    </service>
    <service>
        <role>WEBHCAT</role>
        <url>http://chupa2.openstacklocal:50111/templeton</url>
    </service>
    <service>
        <role>OOZIE</role>
        <url>http://chupa2.openstacklocal:11000/oozie</url>
    </service>
    <service>
        <role>WEBHBASE</role>
        <url>http://chupa1.openstacklocal:8080</url>
    </service>
    <service>
        <role>HIVE</role>
        <url>http://chupa2.openstacklocal:10001/cliservice</url>
    </service>
    <service>
        <role>RESOURCEMANAGER</role>
        <url>http://chupa3.openstacklocal:8088/ws</url>
    </service>
    <service>
        <role>RANGERUI</role>
        <url>http://chupa3.openstacklocal:6080</url>
    </service>
</topology>

5. If you would like to test that it is working you can issue the following command to manually failover the cluster and test.

hdfs haadmin -failover nn1 nn2

6. Test with Knox connection string to webhdfs.

curl -vik -u admin:admin-password 'https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS'
1,795 Views
Comments
New Contributor

Good info -- thanks David

Hi @dvillarreal

I'm just wondering if I need to use a namenode service ID for NAMENODE role to use webHDFS?

New Contributor

Hi, i have problem with knox when i call webhdfs he returned a crypted result :

{"sub":null,"aud":null,"code":"eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJwMDgwMzM0IiwiaXNzIjoiS05PWFNTTyJ9.X8HojHZ_wdQ8h_osOw0p_qRaWKmVLSJKwdhKwdjjOGQwB5DJy5D5JB-49gEvfDWcPNFnKgqsdUrzFcVYGforRxRuVR8b91yL4T_EPwDeN4vlPr5HKgfvPeL2zudR0l7x82G8m5yx09veuwGkDAs6y0GJfY4JTmQgmIS-wRwqlUxjxK7GT6Ktvft7ciwrQny00qSwrrO-RunBbBugPDFvGjqgiufyMpLAqTG58iS5rcKghYS_mHKWIdcvGdNCzCFURvDKr8gqZeN9hj6QqLnjHsP0gmUJ5YzvoJtEVMxoxMy8w7f9KSo7BwPkHjknpa7yFEltXDUvWgDpjdFcn_TPfw","iss":"KNOXSSO","exp":null}

i think the ssl cert is not valid but i can't fix it ?

@Hajime It is not mandatory for WEBHDFS to work. However, It is good practice to make this change in NN HA env. as other services like oozie use this for doing rewrites.

@badr bakkou This would probably be best answered if you submitted as a new question. Provide the gateway.log & gateway-audit.log outputs, topology, and lastly the configuration string you are using with its associated output. Best regards, David

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎03-01-2017 12:40 AM
Updated by:
 
Contributors
Top Kudoed Authors