Spark2 2.3.0 on cdh.5.13 deployed with CSD and parcels.   by default it uses /etc/spark2/conf, but it seems it should use /etc/spark2/conf.cloudera.spark2_on_yarn/   is there a way to set it properly? i could set it in spark2-env.sh but it seems not the appropriate place. ... View more

so i have a flume agent  with a spooldir channel that got too full (too many files error).   the fix was to move the files out of the spooldir, move the checkpoint, and restart the agent, resulting in data flowing again.   but now i have some flume-formatted data that i would like to ingest. can i move the files into the spooldir 10% at a time fx? would i have to read the files and add them to the flume source?     additional notes: the reason for the spool dir was a missing sink (another flume agent- turned off) ... View more

i have a logging setup where i have a fluent/webhdfs host, for writing logs  from fluent to hdfs.   a problem is i can only write to the ACTIVE namenode, this sets some requirements to the logic in the fluent/webhdfs host  that i would like to avoid.   is there a way to make sure youre always posting to the ACTIVE namenode?   ... View more