Knox with HttpFS

[#1]

Configuring HTTPFS

To configure HTTPFS you can refer below post, it clearly explains the steps. We can install httpfs in any host where Hadoop clients are installed and we call it <HTTPFS_HOST>.

HTTPFS - Configure and Run with HDP

Create topology file

In Knox Gateway host under /etc/knox/conf/topologies directory create a new topology file (copy default.xml) say example.xml

After copying it looks like below

[root@test-3 topologies]# ls -l /etc/knox/conf/topologies
total 20
-rw-r--r--. 1 knox knox   89 Oct  4 11:25 README
-rw-r--r--. 1 knox knox 4422 Oct  4 11:25 admin.xml
-rw-r--r--. 1 knox knox 3026 Oct 12 10:22 default.xml
-rw-r--r--. 1 knox knox 3026 Oct 12 10:35 example.xml

In example.xml topology file for WEBHDFS role change host to <HTTPFS_HOST> and port to 14000 . By default HTTPFS_HTTP_PORT is 14000

            <service>
                <role>WEBHDFS</role>
                <url>http://<HTTPFS_HOST>:14000/webhdfs</url>
            </service>

Try it out!!

HTTPFS -> FileSystem

curl -i -X GET 'http://<HTTPFS_HOST>:14000/webhdfs/v1?user.name=hdfs&op=GETHOMEDIRECTORY'

Knox -> HTTPFS -> FileSystem

curl -u guest:guest-password -ik -X GET 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1?op=GETHOMEDIRECTORY'                                                                                 Here example is topology name 

If above 2 commands gives response OK then your setup complete.

CURL command for knox to access HDFS via HttpFS

Below are curl commands for few HDFS operations.

1. Make Directory

curl -u guest:guest-password -ik -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase?op=MKDIRS'

2. Copy file from local filesystem into HDFS

curl -u guest:guest-password -ik -T input.txt -L -H "Content-Type: application/octet-stream" -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op=CREATE&user.name=guest'

3. Print the contents of file

curl -u guest:guest-password -ik 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op=OPEN'

4. Lists the directory Statuse's

curl -u guest:guest-password -ik 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase?op=LISTSTATUS'

5. Display's the content summary

curl -u guest:guest-password -ik 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op==GETCONTENTSUMMARY'

6. Display's the file checksum

curl -u guest:guest-password -ik 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op==GETFILECHECKSUM'

7. Display's the file status

curl -u guest:guest-password -ik 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op=GETFILESTATUS'

8. Rename's the file in HDFS

curl -u guest:guest-password -ik  -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file?op=RENAME&destination=/tmp/TestKnoxHDFS/testFullUseCase/data_file_new'

9. Change the replication factor of a file

curl -u guest:guest-password -ik  -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file_new?SETREPLICATION&replication=1'

10. Change the permission of a file

curl -u guest:guest-password -ik  -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file_new?op=SETPERMISSION&permission=777'

11. Change the modification time of a file

curl -u guest:guest-password -ik  -X PUT 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase/data_file_new?op=SETTIMES&modificationtime=1475865881000'

12. Delete a directory

curl -u guest:guest-password -ik  -X DELETE 'https://<KNOX_GATEWAY_HOST>:8443/gateway/example/webhdfs/v1/tmp/TestKnoxHDFS/testFullUseCase?op=DELETE&recursive=true'