I would like to use Pythons Logging library, but want the output of the logs to land in HDFS instead of the local file system for the worker node.  Is there a way to do that?  My code for setting up logging is below:   import logging logging.basicConfig(filename='/var/log/DataFramedriversRddConvert.log',level=logging.DEBUG) logging.basicConfig(format='%(asctime)s %(message)s') logging.info('++++Started DataFramedriversRddConvert++++') ... View more

I'm trying to install Jupyter using these instructions http://www.cloudera.com/documentation/enterprise/5-6-x/topics/spark_ipython.html.   I've have installed under Anaconda and when I try to run pyspark ( from section  Starting a Notebook with PySpark) I receive the following error:     $ pyspark Traceback (most recent call last):   File "/opt/cloudera/parcels/Anaconda/bin/jupyter", line 4, in <module>     from jupyter_core.command import main ImportError: No module named jupyter_core.command   Any help is appreciated. ... View more