Support Questions
Find answers, ask questions, and share your expertise

Logging from Pyspark executor

Logging from Pyspark executor

New Contributor

Want to use logging either to a different file or using Spark logging from executor.

I am calling a function in foreachPartition, want to log some information from that function. With regular logging it is not working. Tried with below mentioned code which I got from internet, code is getting executed but unable to find the log.

This code will placed in in hdfs and importing using sc.addPyfile

import os
import logging
import sys

class YarnLogger:
    def setup_logger():
        file = 'pyspark.log'
        logging.basicConfig(filename=file, level=logging.INFO, 
                format='%(asctime)s.%(msecs)03d %(levelname)s %(module)s - %(funcName)s: %(message)s')

    def __getattr__(self, key):
        return getattr(logging, key)