Configuring HTTPFS To configure HTTPFS you can refer below post, it clearly explains the steps. We can install httpfs in any host where Hadoop clients are installed and we call it <HTTPFS_HOST>. HTTPFS - Configure and Run with HDP Create topology file In Knox Gateway host under /etc/knox/conf/topologies directory create a new topology file (copy default.xml) say example.xml After copying it looks like below [root@test-3 topologies]# ls -l /etc/knox/conf/topologies
total 20
-rw-r--r--. 1 knox knox 89 Oct 4 11:25 README
-rw-r--r--. 1 knox knox 4422 Oct 4 11:25 admin.xml
-rw-r--r--. 1 knox knox 3026 Oct 12 10:22 default.xml
-rw-r--r--. 1 knox knox 3026 Oct 12 10:35 example.xml
In example.xml topology file for WEBHDFS role change host to <HTTPFS_HOST> and port to 14000 . By default HTTPFS_HTTP_PORT is 14000 <service>
<role>WEBHDFS</role>
<url>http://<HTTPFS_HOST>:14000/webhdfs</url>
</service>
Try it out!! HTTPFS -> FileSystem curl -i -X GET 'http://<HTTPFS_HOST>:14000/webhdfs/v1?user.name=hdfs&op=GETHOMEDIRECTORY' Knox -> HTTPFS -> FileSystem curl -u guest:guest-password -ik -X GET 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1?op=GETHOMEDIRECTORY' Here example is topology name If above 2 commands gives response OK then your setup complete. CURL command for knox to access HDFS via HttpFS Below are curl commands for few HDFS operations. 1. Make Directory curl -u guest:guest-password -ik -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase?op=MKDIRS' 2. Copy file from local filesystem into HDFS curl -u guest:guest-password -ik -T input.txt -L -H "Content-Type: application/octet-stream" -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op=CREATE&user.name=guest' 3. Print the contents of file curl -u guest:guest-password -ik 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op=OPEN' 4. Lists the directory Statuse's curl -u guest:guest-password -ik 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase?op=LISTSTATUS' 5. Display's the content summary curl -u guest:guest-password -ik 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op==GETCONTENTSUMMARY' 6. Display's the file checksum curl -u guest:guest-password -ik 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op==GETFILECHECKSUM' 7. Display's the file status curl -u guest:guest-password -ik 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op=GETFILESTATUS' 8. Rename's the file in HDFS curl -u guest:guest-password -ik -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op=RENAME&destination=/tmp/TestKnoxHDFS/testFullUseCase/data_file_new' 9. Change the replication factor of a file curl -u guest:guest-password -ik -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file_new?SETREPLICATION&replication=1' 10. Change the permission of a file curl -u guest:guest-password -ik -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file_new?op=SETPERMISSION&permission=777' 11. Change the modification time of a file curl -u guest:guest-password -ik -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file_new?op=SETTIMES&modificationtime=1475865881000' 12. Delete a directory curl -u guest:guest-password -ik -X DELETE 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase?op=DELETE&recursive=true'
... View more
