Skip to main content
Ben1996
Ben1996
Silver ⭐️⭐️
August 16, 2023
Solved

How to write to HDFS remotely using pandas

Suppose I have below code:

```python
import pandas as pd

df = pd.read_csv(SOME_PATH)

df.to_csv(OUTPATH,index=False)
```

 

How do I configure OUTPATH to be HDFS?

 

rki_
rki_Answer
Gold ⭐️
August 17, 2023

Hi @Ben1996, As this is very specific use case, I don't see any Cloudera doc which could cover this. The basic requirements would be to import the hdfs module and provide the HDFS configs.

 

For ex, https://tahiriamine9.medium.com/python-hdfs-cd822199799e#:~:text=Let's%20have%20an%20example%20of,into%20HDFS%20with%20CSV%20format.

View in thread

3 replies

VidyaSargur
Community Manager
VidyaSargur
Community Manager
August 17, 2023

@Ben1996, Welcome to our community! To help you get the best possible answer, I have tagged in our HDFS experts @willx @ChethanYM @Chella  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Translate
Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
rki_
Gold ⭐️
rki_Answer
Gold ⭐️
August 17, 2023

Hi @Ben1996, As this is very specific use case, I don't see any Cloudera doc which could cover this. The basic requirements would be to import the hdfs module and provide the HDFS configs.

 

For ex, https://tahiriamine9.medium.com/python-hdfs-cd822199799e#:~:text=Let's%20have%20an%20example%20of,into%20HDFS%20with%20CSV%20format.

Translate
VidyaSargur
Community Manager
VidyaSargur
Community Manager
August 21, 2023

@Ben1996, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Translate
Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
Terms & ConditionsAccessibility statement