question Re: Is Webhdfs supported in HDCloud Protected Gateway? in Archives of Support Questions (Read Only)

Is Webhdfs supported in HDCloud Protected Gateway?

tshephard — Fri, 06 Oct 2017 02:17:38 GMT

Looking at the cluster console I see for network config:

PROTECTED GATEWAY ACCESSAmbari and Zeppelin Web UIs Hive JDBC Cluster Components (NameNode, ResourceManager, Spark History Server, Job History Server)
However, for https://hortonworks.github.io/hdp-aws/security-network/index.html it says that hdfs is supported as a gateway service.
When I try to browse hdfs via https://ec2-54-87-212-190.compute-1.amazonaws.com/cluster1/services/hdfs/explorer.html#/ I just get an error.

Re: Is Webhdfs supported in HDCloud Protected Gateway?

pzampino — Fri, 06 Oct 2017 03:35:35 GMT

Did you try port 443, per the page you referenced?

Re: Is Webhdfs supported in HDCloud Protected Gateway?

tshephard — Fri, 06 Oct 2017 04:59:10 GMT

HTTPS defaults to port 443

Re: Is Webhdfs supported in HDCloud Protected Gateway?

darvasip — Fri, 06 Oct 2017 16:26:19 GMT

@Tim Shephard

Are you using your custom VPC and subnet? If so, have you checked this checklist:

https://hortonworks.github.io/hdp-aws/security-vpc/index.html#configuring-your-own-vpc

Hope this helps!

Re: Is Webhdfs supported in HDCloud Protected Gateway?

tshephard — Tue, 10 Oct 2017 04:00:52 GMT

So are you saying that the knox gateway supports webhdfs? It's not listed on the UI in the cloud cluster console:

PROTECTED GATEWAY ACCESSAmbari and Zeppelin Web UIs Hive JDBC Cluster Components (NameNode, ResourceManager, Spark History Server, Job History Server)

Re: Is Webhdfs supported in HDCloud Protected Gateway?

darvasip — Wed, 11 Oct 2017 00:06:26 GMT

@Tim Shephard

No, I have not written anything related the webhdfs in my answer, but actually both hdfs ui and webhdfs is supported and should work if checked in UI.

Hope this helps!

Re: Is Webhdfs supported in HDCloud Protected Gateway?

tshephard — Wed, 11 Oct 2017 03:15:05 GMT

@pdarvasi that's a very cool reference. Great to see the source code. Unfortunately, I didn't see any option in the UI for

HDFSUI or for WEBHDFS, however I did see options for Ambari, Zeppelin, Hive, JDBC, NameNode, ResourceManager, Spark History Service, and Job History server. That's why I'm concluding that HDFS/WEBHDFS is not supported

Re: Is Webhdfs supported in HDCloud Protected Gateway?

darvasip — Wed, 11 Oct 2017 16:12:25 GMT

@Tim Shephard

You were right about WebHDFS not exposed, but HDFS UI is exposed via the NameNode service so it should be accessible if you enable "Protected Gateway Access to Cluster Components".

If you would like to enable WebHDFS, there is a workaround:

SSH to master node of the cluster
Edit /srv/pillar/gateway/init.sls
Add "WEBHDFS" to gateway:exposed like below:

gateway:
  address: 172.21.250.198
  exposed: [WEBHDFS]
  location:
....

After save, run salt '*' state.highstate (this will regenerate Knox topology)

Hope this helps!

Re: Is Webhdfs supported in HDCloud Protected Gateway?

tshephard — Thu, 12 Oct 2017 03:45:08 GMT

Wow, awesome. Thx!