How to connect to a remote hdfs (for example hortonworks VM) from a local machine in python

[#1]

I want to read and write files to and from a remote HDFS. I program by Pycharm in local machine and I want to connect to a remote hdfs (HDP 2.5). Is there any solution?

How can I configure hdfs and how can I refer to a file in hdfs?

Thanks a million,

Shanghoosh

[#2 · Accepted solution]

@Ali Mohammadi Shanghoshabad

Please check this article : https://community.hortonworks.com/articles/26416/how-to-install-snakebite-in-hdp.html You only need to pass the hdfs config files of your remote cluster to the python client.

.

For example in the python code :you can tell where is your core-site.xml /hdfs-site.xml present. It can be any path where these files are present.

https://github.com/spotify/snakebite/blob/6a456e6100b0c1be66cc1f7f9d7f50494f369da3/snakebite/client....

[#3]

Hi @Jay SenSharma

Thank you for excellent notes.

I solved the problem before reading your notes as follows:

from hdfs import InsecureClient
client = InsecureClient('http://localhost:50070')
# for reading  a file
with client.read('/tmp/tweets_staging/tweets-082940117.json') as reader:
  features = reader.read()
# for writing a file
with client.write('/tmp/tweets_staging/1.json', overwrite=True) as writer:
   writer.write(features)

but I am going to try your solution too.

Regards,

Shanghoosh