Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Is Webhdfs supported in HDCloud Protected Gateway?

avatar
Explorer

Looking at the cluster console I see for network config:

PROTECTED GATEWAY ACCESSAmbari and Zeppelin Web UIs Hive JDBC Cluster Components (NameNode, ResourceManager, Spark History Server, Job History Server)
However, for https://hortonworks.github.io/hdp-aws/security-network/index.html it says that hdfs is supported as a gateway service.
When I try to browse hdfs via https://ec2-54-87-212-190.compute-1.amazonaws.com/cluster1/services/hdfs/explorer.html#/ I just get an error.

1 ACCEPTED SOLUTION

avatar
@Tim Shephard

You were right about WebHDFS not exposed, but HDFS UI is exposed via the NameNode service so it should be accessible if you enable "Protected Gateway Access to Cluster Components".

If you would like to enable WebHDFS, there is a workaround:

  • SSH to master node of the cluster
  • Edit /srv/pillar/gateway/init.sls
  • Add "WEBHDFS" to gateway:exposed like below:
gateway:
  address: 172.21.250.198
  exposed: [WEBHDFS]
  location:
....
  • After save, run salt '*' state.highstate (this will regenerate Knox topology)

Hope this helps!

View solution in original post

8 REPLIES 8

avatar
Rising Star

Did you try port 443, per the page you referenced?

avatar
Explorer

HTTPS defaults to port 443

avatar

@Tim Shephard

Are you using your custom VPC and subnet? If so, have you checked this checklist:

https://hortonworks.github.io/hdp-aws/security-vpc/index.html#configuring-your-own-vpc

Hope this helps!

avatar
Explorer

So are you saying that the knox gateway supports webhdfs? It's not listed on the UI in the cloud cluster console:

PROTECTED GATEWAY ACCESSAmbari and Zeppelin Web UIs Hive JDBC Cluster Components (NameNode, ResourceManager, Spark History Server, Job History Server)

avatar

@Tim Shephard

No, I have not written anything related the webhdfs in my answer, but actually both hdfs ui and webhdfs is supported and should work if checked in UI.

Hope this helps!

avatar
Explorer

@pdarvasi that's a very cool reference. Great to see the source code. Unfortunately, I didn't see any option in the UI for

HDFSUI or for WEBHDFS, however I did see options for Ambari, Zeppelin, Hive, JDBC, NameNode, ResourceManager, Spark History Service, and Job History server. That's why I'm concluding that HDFS/WEBHDFS is not supported

avatar
@Tim Shephard

You were right about WebHDFS not exposed, but HDFS UI is exposed via the NameNode service so it should be accessible if you enable "Protected Gateway Access to Cluster Components".

If you would like to enable WebHDFS, there is a workaround:

  • SSH to master node of the cluster
  • Edit /srv/pillar/gateway/init.sls
  • Add "WEBHDFS" to gateway:exposed like below:
gateway:
  address: 172.21.250.198
  exposed: [WEBHDFS]
  location:
....
  • After save, run salt '*' state.highstate (this will regenerate Knox topology)

Hope this helps!

avatar
Explorer

Wow, awesome. Thx!