Support Questions
Find answers, ask questions, and share your expertise

Pyspark logs to different file

Pyspark logs to different file

Explorer

Hi All,

I want logger=INFO to be stored in a different file.

I am running pyspark script using JupyterHub so I want to know where all i need to change the configurations to achieve the activity of getting logs into a different file path of my wish. I mean do i need to change the jupyterhub pyspark config files.

I found this link :https://medium.com/@shantanualshi/logging-in-pyspark-36b0bd4dec55

But it is for spark i need the same for pyspark. If i update spark does it work for pySpark as well?

Please help in this.

Regards,

Manjunath P N

Don't have an account?