Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to connect to a remote hdfs (for example hortonworks VM) from a local machine in python

avatar
Contributor

I want to read and write files to and from a remote HDFS. I program by Pycharm in local machine and I want to connect to a remote hdfs (HDP 2.5). Is there any solution?

How can I configure hdfs and how can I refer to a file in hdfs?

Thanks a million,

Shanghoosh

1 ACCEPTED SOLUTION

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
2 REPLIES 2

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Contributor

Hi @Jay SenSharma

Thank you for excellent notes.

I solved the problem before reading your notes as follows:

from hdfs import InsecureClient
client = InsecureClient('http://localhost:50070')
# for reading  a file
with client.read('/tmp/tweets_staging/tweets-082940117.json') as reader:
  features = reader.read()
# for writing a file
with client.write('/tmp/tweets_staging/1.json', overwrite=True) as writer:
   writer.write(features)

but I am going to try your solution too.

Regards,

Shanghoosh