Want to use logging either to a different file or using Spark logging from executor.
I am calling a function in foreachPartition, want to log some information from that function. With regular logging it is not working. Tried with below mentioned code which I got from internet, code is getting executed but unable to find the log.
This code will placed in logger.py in hdfs and importing using sc.addPyfile