Member since
10-04-2017
5
Posts
1
Kudos Received
0
Solutions
12-01-2017
05:25 AM
Want to use logging either to a different file or using Spark logging from executor. I am calling a function in foreachPartition, want to log some information from that function. With regular logging it is not working. Tried with below mentioned code which I got from internet, code is getting executed but unable to find the log. This code will placed in logger.py in hdfs and importing using sc.addPyfile import os
import logging
import sys
class YarnLogger:
@staticmethod
def setup_logger():
file = 'pyspark.log'
logging.basicConfig(filename=file, level=logging.INFO,
format='%(asctime)s.%(msecs)03d %(levelname)s %(module)s - %(funcName)s: %(message)s')
def __getattr__(self, key):
return getattr(logging, key)
YarnLogger.setup_logger()
... View more
10-05-2017
05:32 AM
Thanks a lot @Saumil Mayani. Process was running for parser.py. After killing those processes, it got stopped. Thanks again
... View more
10-04-2017
06:31 PM
I am not sure which process is running but my files(which is part of the job process) are getting processed after killing job
... View more
10-04-2017
01:41 PM
Hi @Saumil Mayani, thanks for the reply. Tried that way and killed the process with name as org.apache.spark.deploy.SparkSubmit and yarn application as well. Still process is going on
... View more
10-04-2017
11:34 AM
1 Kudo
Submitting a spark job by using shell script as follows: nohup spark-submit --master yarn-client --py-files libs.zip parser.py --jobname dicomparser >> out.log & Job started to perform required processing, in between killed using yarn application -kill <application_id> and job disappeared from pending list. But could see the process still going on What am I doing wrong?
... View more
Labels: