@Oliver Meyn This is the correct jira https://issues.apache.org/jira/browse/SPARK-12177 and yes, SASL_SSL is only available from Spark 2.0 and not in HDP2.4.2 which has Spark 1.6.1. ... View more

@Fernando Lopez Bello You need to have a hivecontext to access hive tables. from pyspark.sql import HiveContext sqlCtx = HiveContext(sc1) ... View more

@Jayanna TM You can use ignorePattern property to ignore .tmp files from a spool directory (https://flume.apache.org/FlumeUserGuide.html) ignorePattern=\.*tmp$ ... View more

@bigdata.neophyte You would need to use this API to fetch the job status.(https://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/mapreduce/JobStatus.html) If you want a simple solution you could try something like: 1) Set unique job name (eg:date or time) using -Dmapred.job.name=testdist01 2) Get the app status using : yarn application -list -appStates ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUNNING,FINISHED,FAILED,KILLED | grep -i "distcp: testdist01" | awk '{print $7,$8}' FINISHED SUCCEEDED ... View more