I have recently created two node hadoop cluster with CDH5.11.0 with Cloudera manager. It installed spark 1.16 and I am able to launch spark-shell successfully. Later, I installed spark2 from CSD and getting below error when I try to start the service.     + exec /data/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/75-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER/spark2-conf/spark-history-server.conf Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:278) at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala) Caused by: java.io.FileNotFoundException: Log directory specified does not exist: file:/user/spark/spark2ApplicationHistory at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:207) at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:153) at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:149) at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:77) ... 6 more Caused by: java.io.FileNotFoundException: File file:/user/spark/spark2ApplicationHistory does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:598) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:811) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:588) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425) at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:197) ... 9 more   File file:/user/spark/spark2ApplicationHistory does not exist I have verified this file exists in HDFS and has full access to any user. Please help me if there are any solution to come out of this problem. ... View more

Hi,   I have setuped cloudera 5.3 in 3 node cluster (1 Namenode, 2 Datanodes). All are running with same OS, JAVA and other softwares with same configuration (4GB RAM). Loaded some data files, everything is replicated across two datanodes (Replication factor 3) but still have some unreplicated blocks (thats not a concern). When I issue query through HIVE, the job creates around 37mappers to achieve the results, but all are getting executed in single datanode instead two datanodes. I dont know what is happening, but below are the details I captured related to job info.   Application OverviewUser:Name:Application Type:Application Tags:State:FinalStatus:Started:Elapsed:Tracking URL:Diagnostics: hdfs select sair_str_num,s...ir_upc_num,salesdate(Stage-1) MAPREDUCE   FINISHED SUCCEEDED Thu Mar 05 17:15:51 -0500 2015 8mins, 50sec History     Application MetricsTotal Resource Preempted:Total Number of Non-AM Containers Preempted:Total Number of AM Containers Preempted:Resource Preempted from Current Attempt:Number of Non-AM Containers Preempted from Current Attempt:Aggregate Resource Allocation: <memory:0, vCores:0> 0 0 <memory:0, vCores:0> 0 1070822 MB-seconds, 1028 vcore-seconds ApplicationMasterAttempt Number Start Time Node Logs 1 Thu Mar 05 17:15:51 -0500 2015 server003.enterprisenet.org:8042 logs      Logs:   Log Type: stderr Log Length: 2257 Mar 05, 2015 5:16:08 PM com.google.inject.servlet.InternalServletModule$BackwardsCompatibleServletContextProvider get WARNING: You are attempting to use a deprecated API (specifically, attempting to @Inject ServletContext inside an eagerly created singleton. While we allow this for backwards compatibility, be warned that this MAY have unexpected behavior if you have more than one injector (with ServletModule) running in the same JVM. Please consult the Guice documentation at http://code.google.com/p/google-guice/wiki/Servlets for more information. Mar 05, 2015 5:16:09 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class Mar 05, 2015 5:16:09 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Mar 05, 2015 5:16:09 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class Mar 05, 2015 5:16:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Mar 05, 2015 5:16:09 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton" Mar 05, 2015 5:16:09 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Mar 05, 2015 5:16:10 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest" log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.   Log Type: stdout Log Length: 0   Log Type: syslog Log Length: 528383 Showing 4096 bytes of 528383 total. Click here for the full log. 1425593751461-hdfs-select+sair_str_num%2Cs...ir_upc_num%2Csalesdate%28Stage-1425594282045-37-10-SUCCEEDED-root.hdfs-1425593764268.jhist_tmp 2015-03-05 17:24:42,157 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002-1425593751461-hdfs-select+sair_str_num%2Cs...ir_upc_num%2Csalesdate%28Stage-1425594282045-37-10-SUCCEEDED-root.hdfs-1425593764268.jhist_tmp 2015-03-05 17:24:42,162 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://server001.enterprisenet.org:8020/user/hdfs/.staging/job_1425593189300_0002/job_1425593189300_0002_1_conf.xml to hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002_conf.xml_tmp 2015-03-05 17:24:42,197 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002_conf.xml_tmp 2015-03-05 17:24:42,210 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002.summary_tmp to hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002.summary 2015-03-05 17:24:42,214 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002_conf.xml_tmp to hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002_conf.xml 2015-03-05 17:24:42,218 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002-1425593751461-hdfs-select+sair_str_num%2Cs...ir_upc_num%2Csalesdate%28Stage-1425594282045-37-10-SUCCEEDED-root.hdfs-1425593764268.jhist_tmp to hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002-1425593751461-hdfs-select+sair_str_num%2Cs...ir_upc_num%2Csalesdate%28Stage-1425594282045-37-10-SUCCEEDED-root.hdfs-1425593764268.jhist 2015-03-05 17:24:42,219 INFO [Thread-184] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 2015-03-05 17:24:42,224 INFO [Thread-184] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to 2015-03-05 17:24:42,225 INFO [Thread-184] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://server001.enterprisenet.org:19888/jobhistory/job/job_1425593189300_0002 2015-03-05 17:24:42,234 INFO [Thread-184] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered. 2015-03-05 17:24:43,236 INFO [Thread-184] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:1 CompletedMaps:37 CompletedReds:9 ContAlloc:47 ContRel:0 HostLocal:19 RackLocal:18 2015-03-05 17:24:43,239 INFO [Thread-184] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://server001.enterprisenet.org:8020 /user/hdfs/.staging/job_1425593189300_0002 2015-03-05 17:24:43,252 INFO [Thread-184] org.apache.hadoop.ipc.Server: Stopping server on 48930 2015-03-05 17:24:43,262 INFO [IPC Server listener on 48930] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 48930 2015-03-05 17:24:43,262 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2015-03-05 17:24:43,262 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted   Please help me to run mapreduce in both datanodes. ... View more

Getting below error when trying to start HDFS service.   Service did not start successfully; not all of the required roles started: Service has only 0 DataNode roles running instead of minimum required 1. 11:35:09.022 AM FATAL org.apache.hadoop.hdfs.server.datanode.DataNode Exception in secureMain java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/rms/dfs/dn/" "/rms/dfs/dn2/" at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2319) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2292) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2184) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2231) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2407) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2431)   Both directories exists in both datanodes (2 Datanode cluster). Below are the directories and their permission.   drwxrwxrwx 3 cloudera-scm cloudera-scm 4096 Mar 1 12:31 dn2 drwxrwxrwx 3 cloudera-scm cloudera-scm 4096 Mar 1 12:31 dn   Please provide your solution to resolve this. ... View more

I choose Path C tarball install for cloudera manager installation. Followed all steps described in the installation document. I have sucessfully start scm server, but getting below error when I try to start agent service.    sudo /usr/local/cloudera-manager/cm-5.3.1/etc/init.d/cloudera-scm-agent start Starting cloudera-scm-agent: [FAILED]   There are no log files under /usr/local/cloudera-manager/cm-5.3.1/log/cloudera-scm-agent, only .out file is available.   Traceback (most recent call last): File "/usr/local/cloudera-manager/cm-5.3.1/lib64/cmf/agent/src/cmf/agent.py", line 19, in <module> import psutil File "/usr/local/cloudera-manager/cm-5.3.1/lib64/cmf/agent/build/env/lib/python2.6/site-packages/psutil-2.1.3-py2.6-linux-x86_64.egg/psutil/__init__.py", line 89, in <module> import psutil._pslinux as _psplatform File "/usr/local/cloudera-manager/cm-5.3.1/lib64/cmf/agent/build/env/lib/python2.6/site-packages/psutil-2.1.3-py2.6-linux-x86_64.egg/psutil/_pslinux.py", line 24, in <module> import _psutil_linux as cext ImportError: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by /usr/local/cloudera-manager/cm-5.3.1/lib64/cmf/agent/build/env/lib/python2.6/site-packages/psutil-2.1.3-py2.6-linux-x86_64.egg/_psutil_linux.so)   Please help me to come out of this issue. ... View more