Hi @sbpothineni Why not using CombineFileInputFormat? ... View more

@Yuexin Zhang Thanks for your response.   since i'm accessing it from the Spark History UI, i'm not sure if the UI is running with cloudera-scm user.   few things i'm trying to figure out which may hekp me to find a solution for this issue.   1- How it working on spark 1.6 differenty, and not in Spark 2, in Spark 1.6 the jobs under hdfs://name-node/user/spark/applicationhistory is written with the user cloudera-scm and group spark with permissions 770.   2- How i can know with which user the UI is pulling the data?   3- can i change the permission of the files under the hdfs spark history dir by adding specific config?   for example : something like spark.eventLog.permissions=755 ... View more

Hi @Yuexin Zhang   Thanks for your response.   The /var/run/process and the ps -ef show that the user and the group is cloudera-scm   /var/run/cloudera-scm-agent/process [root@serever process]# ll | grep Spark [root@server process]# ll | grep SPARK drwxr-x--x 7 cloudera-scm cloudera-scm 280 May 27 03:05 19175-spark_on_yarn-SPARK_YARN_HISTORY_SERVER drwxr-x--x 8 cloudera-scm cloudera-scm 300 May 27 03:17 19240-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER   1829 cloudera  20   0 6682m 451m  33m S  0.3  0.4 379:36.38 /var/jdk8/bin/java -cp /var/run/cloudera-scm-agent/process/19240-spark2_on_yar   Also it's intersting for me why it's working for Spark 1.6 and not for Spark2.   It may also worth mentioning that my cluster is running with single user "cloudera-scm" as i'm using the cloudera manager in express version       ... View more

Hi @AAS Caan you please share the placement Rules?   Seems that the rule of Use the pool root.[username], only if the pool exists has higher priority ... View more

Hi @Tim Armstrong Hope you are doing well,   It will be nice if we have a metirc for the memory part of the daemons_memory_limit used by impala daemon in a given time.   So when i get a query failing on memory, i can investigate the memory usage thatwill help me to understand when to increase the limit, secondly, i can learn trend and usage over time and i can plan my increase. Currently i see only the resident memory per node but this memory isn't used by the queries, so it's a diffcult task for me to investigate the impala behaviour once a query failed on memory.   Yes i have a metric of the total memory used by node, but i have different roles in the node, so it hard to track this issue.  ... View more

@hendry Hi Hendry,   122 GB in the big Data is not considered as too much, and it's depend on the logic you are doing in the map, normally the logic in the reducer which you should cosider increasing its memory.   To learn the mapreduce memory usage, i would recommend you to use one of the tool that can help you identifying where you are loosing ,memory.   In my side i'm usind Dr. elephant which help me to understand what is the best configuration to fully utilize my resources.   For more details:   https://github.com/linkedin/dr-elephant ... View more

Thanks all and specially @GeKas < just to update that i was able to solve the issue, it was some of the lefover of enabling keberos on the cluster, i was install the oracle JDK which installed java1.7_cloudera, once i removed this package from the node, the LZO error gone. ... View more

@GeKas Can you have a quick look here and help please. ... View more

Hi Guys,   I'm running into issue where my spark jobs are failing on the below error, I'm using Spark 1.6.0 with CDH 5.13.0. I tried to figure it out with no success. Will appreciate any help or a direction how to attack this issue. User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 3, xxxxxx, executor 1): java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getDecompressorType(LzoCodec.java:193) at org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:181) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1995) at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1881) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1830) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1844) at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54) at com.liveperson.dallas.lp.utils.incremental.DallasGenericTextFileRecordReader.initialize(DallasGenericTextFileRecordReader.java:64) at com.liveperson.hadoop.fs.inputs.LPCombineFileRecordReaderWrapper.initialize(LPCombineFileRecordReaderWrapper.java:38) at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initialize(CombineFileRecordReader.java:63) at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:168) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:133) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:65) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Driver stacktrace:   I see the LZO at GPextras: ll total 104 -rw-r--r-- 1 cloudera-scm cloudera-scm 35308 Oct 4 2017 COPYING.hadoop-lzo -rw-r--r-- 1 cloudera-scm cloudera-scm 62268 Oct 4 2017 hadoop-lzo-0.4.15-cdh5.13.0.jar lrwxrwxrwx 1 cloudera-scm cloudera-scm 31 May 3 07:23 hadoop-lzo.jar -> hadoop-lzo-0.4.15-cdh5.13.0.jar drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Oct 4 2017 native   i see only lzo only for impala [root@xxxxxxx ~]# locate *lzo*.so* /opt/cloudera/parcels/GPLEXTRAS-5.13.0-1.cdh5.13.0.p0.29/lib/impala/lib/libimpalalzo.so /usr/lib64/liblzo2.so.2 /usr/lib64/liblzo2.so.2.0.0 the /opt/cloudera/parcels/GPLEXTRAS-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop/lib/native has : -rwxr-xr-x 1 cloudera-scm cloudera-scm 22918 Oct 4 2017 libgplcompression.a -rwxr-xr-x 1 cloudera-scm cloudera-scm 1204 Oct 4 2017 libgplcompression.la -rwxr-xr-x 1 cloudera-scm cloudera-scm 1205 Oct 4 2017 libgplcompression.lai -rwxr-xr-x 1 cloudera-scm cloudera-scm 15760 Oct 4 2017 libgplcompression.so -rwxr-xr-x 1 cloudera-scm cloudera-scm 15768 Oct 4 2017 libgplcompression.so.0 -rwxr-xr-x 1 cloudera-scm cloudera-scm 15768 Oct 4 2017 libgplcompression.so.0.0.0 and /opt/cloudera/parcels/GPLEXTRAS-5.13.0-1.cdh5.13.0.p0.29/lib/spark-netlib/lib has: -rw-r--r-- 1 cloudera-scm cloudera-scm 8673 Oct 4 2017 jniloader-1.1.jar -rw-r--r-- 1 cloudera-scm cloudera-scm 53249 Oct 4 2017 native_ref-java-1.1.jar -rw-r--r-- 1 cloudera-scm cloudera-scm 53295 Oct 4 2017 native_system-java-1.1.jar -rw-r--r-- 1 cloudera-scm cloudera-scm 1732268 Oct 4 2017 netlib-native_ref-linux-x86_64-1.1-natives.jar -rw-r--r-- 1 cloudera-scm cloudera-scm 446694 Oct 4 2017 netlib-native_system-linux-x86_64-1.1-natives.jar Note: The issue occuring only with the spark job, mapreduce job working fine. ... View more