03-27-2017 08:23 AM
I would like to use Pythons Logging library, but want the output of the logs to land in HDFS instead of the local file system for the worker node. Is there a way to do that?
My code for setting up logging is below:
03-27-2017 12:30 PM
You can achive this by giving fully qualified path.
## To use HDFS path
## To use Local path
Some additional Notes: It is not recommended to have logs in HDFS for two reasons
1. HDFS maintains 3 replication factors by default.
2. If HDFS goes down, you cannot check the logs