Hi everyone, I have recently installed CM 6.0.1 with a cluster of two nodes. When I type spark-shell, in order to open spark it gets stuck at this first message "Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel)." When I run spark-shell --master local , it works well, so I suppose it is  yarn configuration problem. By the way, according to CM, all the roles in Spark and Yarn are working fine. I suppose it is a problem of configuration about memory in the different roles of Yarn, but I do not know ho to manage. If it useful, the node which has CM installed has 15 GB of RAM an the other one 10. Can anyone help? How should I customize yarn.app.mapreduce.am.resource.mb, Tamaño de montón máximo Java de ApplicationMaster, yarn.nodemanager.resource.memory-mb etc etc.??   Thanks a lot   Miki ... View more

Hi everyone, Inside Hue, whe I try to execute a query as select it works well. But when I try to execute a simple query of "insert overwrite" or count (*), that it implies map and reduce, it lasts a lot of time and finally it appears error code 2.   I read the logs but I do not understand them very well. I attached  part of it, I am not able the full file,if someone that understands could help me please.   By the way, when I execute these queris in the quickstart cloudera virtual box, it works perfectly. Help me please 😉   The information of logs are:   1st log  Log Type: syslog Log Upload Time: mié abr 04 22:50:02 +0200 2018 Log Length: 86872 0- exited : java.lang.RuntimeException: Error in configuring object     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)     at java.security.AccessController.doPrivileged(Native Method)     at javax.security.auth.Subject.doAs(Subject.java:415)     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)     ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)     ... 14 more Caused by: java.lang.reflect.InvocationTargetException     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)     ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed     at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:147)     ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found     at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:323)     at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:333)     at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116)     ... 22 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2185)     at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:137)     at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:297)     ... 24 more 2018-04-04 22:49:19,064 INFO [IPC Server handler 28 on 41836] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1522871964335_0004_m_000000_0: Error: java.lang.RuntimeException: Error in configuring object     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)     at java.security.AccessController.doPrivileged(Native Method)     at javax.security.auth.Subject.doAs(Subject.java:415)     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)     ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)     ... 14 more Caused by: java.lang.reflect.InvocationTargetException     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)     ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed     at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:147)     ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found     at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:323)     at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:333)     at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116)     ... 22 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2185)     at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:137)     at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:297)     ... 24 more       ... View more