Support Questions
Find answers, ask questions, and share your expertise

Is Webhdfs supported in HDCloud Protected Gateway?

Explorer

Looking at the cluster console I see for network config:

PROTECTED GATEWAY ACCESSAmbari and Zeppelin Web UIs Hive JDBC Cluster Components (NameNode, ResourceManager, Spark History Server, Job History Server)
However, for https://hortonworks.github.io/hdp-aws/security-network/index.html it says that hdfs is supported as a gateway service.
When I try to browse hdfs via https://ec2-54-87-212-190.compute-1.amazonaws.com/cluster1/services/hdfs/explorer.html#/ I just get an error.

1 ACCEPTED SOLUTION

@Tim Shephard

You were right about WebHDFS not exposed, but HDFS UI is exposed via the NameNode service so it should be accessible if you enable "Protected Gateway Access to Cluster Components".

If you would like to enable WebHDFS, there is a workaround:

  • SSH to master node of the cluster
  • Edit /srv/pillar/gateway/init.sls
  • Add "WEBHDFS" to gateway:exposed like below:
gateway:
  address: 172.21.250.198
  exposed: [WEBHDFS]
  location:
....
  • After save, run salt '*' state.highstate (this will regenerate Knox topology)

Hope this helps!

View solution in original post

8 REPLIES 8

Cloudera Employee

Did you try port 443, per the page you referenced?

Explorer

HTTPS defaults to port 443

@Tim Shephard

Are you using your custom VPC and subnet? If so, have you checked this checklist:

https://hortonworks.github.io/hdp-aws/security-vpc/index.html#configuring-your-own-vpc

Hope this helps!

Explorer

So are you saying that the knox gateway supports webhdfs? It's not listed on the UI in the cloud cluster console:

PROTECTED GATEWAY ACCESSAmbari and Zeppelin Web UIs Hive JDBC Cluster Components (NameNode, ResourceManager, Spark History Server, Job History Server)

@Tim Shephard

No, I have not written anything related the webhdfs in my answer, but actually both hdfs ui and webhdfs is supported and should work if checked in UI.

Hope this helps!

Explorer

@pdarvasi that's a very cool reference. Great to see the source code. Unfortunately, I didn't see any option in the UI for

HDFSUI or for WEBHDFS, however I did see options for Ambari, Zeppelin, Hive, JDBC, NameNode, ResourceManager, Spark History Service, and Job History server. That's why I'm concluding that HDFS/WEBHDFS is not supported

@Tim Shephard

You were right about WebHDFS not exposed, but HDFS UI is exposed via the NameNode service so it should be accessible if you enable "Protected Gateway Access to Cluster Components".

If you would like to enable WebHDFS, there is a workaround:

  • SSH to master node of the cluster
  • Edit /srv/pillar/gateway/init.sls
  • Add "WEBHDFS" to gateway:exposed like below:
gateway:
  address: 172.21.250.198
  exposed: [WEBHDFS]
  location:
....
  • After save, run salt '*' state.highstate (this will regenerate Knox topology)

Hope this helps!

Explorer

Wow, awesome. Thx!

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.