question Re: How to write to HDFS remotely using pandas in Support Questions

How to write to HDFS remotely using pandas

Ben1996 — Wed, 16 Aug 2023 07:13:39 GMT

Suppose I have below code:

```python
import pandas as pd

df = pd.read_csv(SOME_PATH)

df.to_csv(OUTPATH,index=False)
```

How do I configure OUTPATH to be HDFS?

Re: How to write to HDFS remotely using pandas

VidyaSargur — Thu, 17 Aug 2023 05:21:40 GMT

@Ben1996, Welcome to our community! To help you get the best possible answer, I have tagged in our HDFS experts @willx @ChethanYM @Chella who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: How to write to HDFS remotely using pandas

rki_ — Thu, 17 Aug 2023 07:51:04 GMT

Hi @Ben1996, As this is very specific use case, I don't see any Cloudera doc which could cover this. The basic requirements would be to import the hdfs module and provide the HDFS configs.

For ex, https://tahiriamine9.medium.com/python-hdfs-cd822199799e#:~:text=Let's%20have%20an%20example%20of,into%20HDFS%20with%20CSV%20format.

Re: How to write to HDFS remotely using pandas

VidyaSargur — Mon, 21 Aug 2023 09:52:54 GMT

@Ben1996, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.