I want logger=INFO to be stored in a different file.
I am running pyspark script using JupyterHub so I want to know where all i need to change the configurations to achieve the activity of getting logs into a different file path of my wish. I mean do i need to change the jupyterhub pyspark config files.
I found this link :https://medium.com/@shantanualshi/logging-in-pyspark-36b0bd4dec55
But it is for spark i need the same for pyspark. If i update spark does it work for pySpark as well?