Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Logging from Pyspark executor

Highlighted

Logging from Pyspark executor

New Contributor

Want to use logging either to a different file or using Spark logging from executor.

I am calling a function in foreachPartition, want to log some information from that function. With regular logging it is not working. Tried with below mentioned code which I got from internet, code is getting executed but unable to find the log.

This code will placed in logger.py in hdfs and importing using sc.addPyfile

import os
import logging
import sys

class YarnLogger:
    @staticmethod
    def setup_logger():
        file = 'pyspark.log'
        logging.basicConfig(filename=file, level=logging.INFO, 
                format='%(asctime)s.%(msecs)03d %(levelname)s %(module)s - %(funcName)s: %(message)s')

    def __getattr__(self, key):
        return getattr(logging, key)

YarnLogger.setup_logger()