Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Pyspark logs to different file

Highlighted

Pyspark logs to different file

Explorer

Hi All,

I want logger=INFO to be stored in a different file.

I am running pyspark script using JupyterHub so I want to know where all i need to change the configurations to achieve the activity of getting logs into a different file path of my wish. I mean do i need to change the jupyterhub pyspark config files.

I found this link :https://medium.com/@shantanualshi/logging-in-pyspark-36b0bd4dec55

But it is for spark i need the same for pyspark. If i update spark does it work for pySpark as well?

Please help in this.

Regards,

Manjunath P N