Not sure of your exact question, but typically it is a good idea to compress the output of your map step in map-reduce jobs. This is because this data is written to disk and then sent within your cluster to the reducer (shuffle) and the overhead of compressing/decompressing is almost always minimal compared to the large gains from sending over the wire significantly lower data volumes from compressed data. To set this for all of your jobs, use these configs in mapred-site.xml" <property> <name>mapred.compress.map.output</name> <value>true</value> </property> <property> <name>mapred.map.output.compression.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> You can of course set the first value to false in mapred-site.xml and override it by setting it for each job (e.g. as a parameter in the command line or set at the top of a pig script). See this link for details: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_hdfs_admin_tools/content/ch04.html ... View more

# Check Commands # -------------- # Spark Scala # ----------- # Optionally export Spark Home export SPARK_HOME=/usr/hdp/current/spark-client # Spark submit example in local mode spark-submit --class org.apache.spark.examples.SparkPi --driver-memory 512m --executor-memory 512m --executor-cores 1 $SPARK_HOME/lib/spark-examples*.jar 10 # Spark submit example in client mode spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 $SPARK_HOME/lib/spark-examples*.jar 10 # Spark submit example in cluster mode spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 $SPARK_HOME/lib/spark-examples*.jar 10 # Spark shell with yarn client spark-shell --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 # Pyspark # ------- # Optionally export Hadoop COnf and PySpark Python export HADOOP_CONF_DIR=/etc/hadoop/conf export PYSPARK_PYTHON=/opath/to/bin/python # PySpark submit example in local mode spark-submit --verbose /usr/hdp/2.3.0.0-2557/spark/examples/src/main/python/pi.py 100 # PySpark submit example in client mode spark-submit --verbose --master yarn-client /usr/hdp/2.3.0.0-2557/spark/examples/src/main/python/pi.py 100 # PySpark submit example in cluster mode spark-submit --verbose --master yarn-cluster /usr/hdp/2.3.0.0-2557/spark/examples/src/main/python/pi.py 100 # PySpark shell with yarn client pyspark --master yarn-client @jigar.patel ... View more