Want to use logging either to a different file or using Spark logging from executor.
I am calling a function in foreachPartition, want to log some information from that function. With regular logging it is not working. Tried with below mentioned code which I got from internet, code is getting executed but unable to find the log.
This code will placed in logger.py in hdfs and importing using sc.addPyfile
import os
import logging
import sys
class YarnLogger:
@staticmethod
def setup_logger():
file = 'pyspark.log'
logging.basicConfig(filename=file, level=logging.INFO,
format='%(asctime)s.%(msecs)03d %(levelname)s %(module)s - %(funcName)s: %(message)s')
def __getattr__(self, key):
return getattr(logging, key)
YarnLogger.setup_logger()
