question How to connect to a remote hdfs (for example hortonworks VM) from a local machine in python in Support Questions

How to connect to a remote hdfs (for example hortonworks VM) from a local machine in python

ali_mohammadi — Tue, 04 Apr 2017 18:25:03 GMT

I want to read and write files to and from a remote HDFS. I program by Pycharm in local machine and I want to connect to a remote hdfs (HDP 2.5). Is there any solution?

How can I configure hdfs and how can I refer to a file in hdfs?

Thanks a million,

Shanghoosh

Re: How to connect to a remote hdfs (for example hortonworks VM) from a local machine in python

jsensharma — Tue, 04 Apr 2017 18:33:17 GMT

@Ali Mohammadi Shanghoshabad

Please check this article : https://community.hortonworks.com/articles/26416/how-to-install-snakebite-in-hdp.html You only need to pass the hdfs config files of your remote cluster to the python client.

For example in the python code :you can tell where is your core-site.xml /hdfs-site.xml present. It can be any path where these files are present.

https://github.com/spotify/snakebite/blob/6a456e6100b0c1be66cc1f7f9d7f50494f369da3/snakebite/client.py#L1553

Re: How to connect to a remote hdfs (for example hortonworks VM) from a local machine in python

ali_mohammadi — Wed, 05 Apr 2017 12:26:29 GMT

Hi @Jay SenSharma

Thank you for excellent notes.

I solved the problem before reading your notes as follows:

from hdfs import InsecureClient
client = InsecureClient('http://localhost:50070')
# for reading  a file
with client.read('/tmp/tweets_staging/tweets-082940117.json') as reader:
  features = reader.read()
# for writing a file
with client.write('/tmp/tweets_staging/1.json', overwrite=True) as writer:
   writer.write(features)

but I am going to try your solution too.

Regards,

Shanghoosh