Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to configure HDFS to use Proxy Server configuration

Highlighted

How to configure HDFS to use Proxy Server configuration

Contributor

Dear Experts,

Is it possible to configure for hdfs to use proxy server configurations? We have a proxy server and all outgoing traffic needs to go to that proxy server, we want to access azure blob storage with hdfs commands, but it does not seem to work at the moment because hdfs does not use proxy server, but tries to make direct http / https call to the azure blob storage which is not allowed.

Does anybody have idea that which hadoop properties we can configure to make it happen?

Thanks in advance.

Regards,

Hammad

2 REPLIES 2
Highlighted

Re: How to configure HDFS to use Proxy Server configuration

Rising Star

Have you seen the HDP Reference Guide at http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_HDP_Reference_Guide/bk_HDP_Reference_Guid...? There are instructions for setting up a trusted proxy server. Maybe that will help.

Re: How to configure HDFS to use Proxy Server configuration

Contributor

@Beverley Andalora Thanks for sending me this, as I said in my question, we already have a proxy server setup and it works fine locally on linux, so for example we are able to donwload all hortonworks packages. The problem is while using hdfs commands, for example if we have a storage account on azure blob (its a http connection) and we want to access it via hdfs dfs -ls wasb://<storage account>/ , but hdfs tries to go to the URL directly rather than going to the proxy server first and hence no connection. So can we setup something inside hadoop to make it understand to use proxy server. Thanks for your help.

Don't have an account?
Coming from Hortonworks? Activate your account here