- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
Created on 03-01-2017 12:40 AM
1. HA provider for webhdfs is needed in your topology.
<provider> <role>ha</role> <name>HaProvider</name> <enabled>true</enabled> <param> <name>WEBHDFS</name> <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value> </param> </provider>
2. The namenode service url value should contain your name service ID. (This can be found in your hdfs-default.xml under parameter dfs.internal.nameservices)
<service> <role>NAMENODE</role> <url>hdfs://chupa</url> </service>
3. Make sure webhdfs url for each namenode is added in your WEBHDFS service area.
<service> <role>WEBHDFS</role> <url>http://chupa1.openstacklocal:50070/webhdfs</url> <url>http://chupa2.openstacklocal:50070/webhdfs</url> </service>
4. Here is a working topology using the knox default demo LDAP.
<topology> <gateway> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>sessionTimeout</name> <value>30</value> </param> <param> <name>main.ldapRealm</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> </param> <param> <name>main.ldapRealm.userDnTemplate</name> <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> </param> <param> <name>main.ldapRealm.contextFactory.url</name> <value>ldap://chupa1.openstacklocal:33389</value> </param> <param> <name>main.ldapRealm.contextFactory.authenticationMechanism</name> <value>simple</value> </param> <param> <name>urls./**</name> <value>authcBasic</value> </param> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> <provider> <role>authorization</role> <name>XASecurePDPKnox</name> <enabled>true</enabled> </provider> <provider> <role>ha</role> <name>HaProvider</name> <enabled>true</enabled> <param> <name>WEBHDFS</name> <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value> </param> </provider> </gateway> <service> <role>NAMENODE</role> <url>hdfs://chupa</url> </service> <service> <role>JOBTRACKER</role> <url>rpc://chupa3.openstacklocal:8050</url> </service> <service> <role>WEBHDFS</role> <url>http://chupa1.openstacklocal:50070/webhdfs</url> <url>http://chupa2.openstacklocal:50070/webhdfs</url> </service> <service> <role>WEBHCAT</role> <url>http://chupa2.openstacklocal:50111/templeton</url> </service> <service> <role>OOZIE</role> <url>http://chupa2.openstacklocal:11000/oozie</url> </service> <service> <role>WEBHBASE</role> <url>http://chupa1.openstacklocal:8080</url> </service> <service> <role>HIVE</role> <url>http://chupa2.openstacklocal:10001/cliservice</url> </service> <service> <role>RESOURCEMANAGER</role> <url>http://chupa3.openstacklocal:8088/ws</url> </service> <service> <role>RANGERUI</role> <url>http://chupa3.openstacklocal:6080</url> </service> </topology>
5. If you would like to test that it is working you can issue the following command to manually failover the cluster and test.
hdfs haadmin -failover nn1 nn2
6. Test with Knox connection string to webhdfs.
curl -vik -u admin:admin-password 'https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS'
Created on 03-03-2017 10:11 AM
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
Good info -- thanks David
Created on 03-06-2017 12:30 AM
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
Hi @dvillarreal
I'm just wondering if I need to use a namenode service ID for NAMENODE role to use webHDFS?
Created on 03-06-2017 03:50 PM
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
Hi, i have problem with knox when i call webhdfs he returned a crypted result :
{"sub":null,"aud":null,"code":"eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJwMDgwMzM0IiwiaXNzIjoiS05PWFNTTyJ9.X8HojHZ_wdQ8h_osOw0p_qRaWKmVLSJKwdhKwdjjOGQwB5DJy5D5JB-49gEvfDWcPNFnKgqsdUrzFcVYGforRxRuVR8b91yL4T_EPwDeN4vlPr5HKgfvPeL2zudR0l7x82G8m5yx09veuwGkDAs6y0GJfY4JTmQgmIS-wRwqlUxjxK7GT6Ktvft7ciwrQny00qSwrrO-RunBbBugPDFvGjqgiufyMpLAqTG58iS5rcKghYS_mHKWIdcvGdNCzCFURvDKr8gqZeN9hj6QqLnjHsP0gmUJ5YzvoJtEVMxoxMy8w7f9KSo7BwPkHjknpa7yFEltXDUvWgDpjdFcn_TPfw","iss":"KNOXSSO","exp":null}
i think the ssl cert is not valid but i can't fix it ?
Created on 03-06-2017 08:56 PM
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
@Hajime It is not mandatory for WEBHDFS to work. However, It is good practice to make this change in NN HA env. as other services like oozie use this for doing rewrites.
Created on 03-06-2017 09:11 PM
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
@badr bakkou This would probably be best answered if you submitted as a new question. Provide the gateway.log & gateway-audit.log outputs, topology, and lastly the configuration string you are using with its associated output. Best regards, David