Member since
11-11-2014
21
Posts
3
Kudos Received
0
Solutions
02-02-2017
07:57 PM
And guessing from what is happening here I thought I should try starting up the thrift server in http mode on an http port. but http port for thrift server is not defined anywhere by Ambari, so created an custom propery for thrift server: 'hive.server2.thrift.http.port: 10013' and defined 'hive.server2.transport.mode: http' And it started thrift server on a different port (10013) in http mode! Regards Rakesh
... View more
02-02-2017
07:35 PM
Hi @Smart Solutions I am lil late to the party, but I was able to run both HS2 and STS on the same machine on a kerberized cluster using: HS2 => hive.server2.thrift.http.port: 10001, transportmode: http, STS => hive.server2.thrift.port:10015, hive.server2.transport.mode: binary STS does not start and throws the bind exception when I use: STS => hive.server2.thrift.port:10015, hive.server2.transport.mode: http So changing transport mode for STS to binary works for me. Tested on HDP 2.4.2.29-4 Regards Rakesh
... View more
08-14-2016
09:51 PM
Thanks Artem, you are correct, but due to some constraints we can not wait until upgrade. I am unable to find a fix for this.
... View more
08-11-2016
03:45 PM
We are using spark 1.3 on hdp2.2.4 and I found there is a bug in the spark-assembly-1.3.1.2.3.0.0-2557-hadoop2.7.1.2.3.0.0-2557.jar that ships with spark. the Mllib check for the version of numpy is incorrect and MLlib throws an exception. I know the fix, I have to change the below file in the jar: mllib/__init__.py" below is the current code in the above mention python file: import numpy
if numpy.version.version < '1.4':
raise Exception("MLlib requires NumPy 1.4+") It can be fixed by changing to: import numpy
ver = [int(x) for x in numpy.version.version.split('.')[:2]]
if ver < [1, 4]:
raise Exception("MLlib requires NumPy 1.4+") I have tried editing the 'spark-assembly-1.3.1.2.3.0.0-2557-hadoop2.7.1.2.3.0.0-2557.jar', to correct the code.
I un-zipped the jar file, fixed the code, re packed it using zip. But after placing the fix, it gives EOF error: Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, xxxxxx.xxxx.uk.hxxx): org.apache.spark.SparkException:
Error from python worker:
/opt/anaconda/envs/sparkAnaconda/bin/python: No module named pyspark
PYTHONPATH was:
/data/4/hadoop/yarn/local/usercache/xxxxxxxx/filecache/33/spark-assembly-1.3.1.2.3.0.0-2557-hadoop2.7.1.2.3.0.0-2557.jar
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:163)
at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:86)
at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:105)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
... View more
Labels:
07-28-2016
08:13 PM
I have disabled the audit logging to a database by setting: XAAUDIT.DB.IS_ENABLED=false But it still asks for a username.
... View more
07-28-2016
06:43 PM
at com.xasecure.utils.install.XmlConfigChanger.run(XmlConfigChanger.java:208)
at com.xasecure.utils.install.XmlConfigChanger.main(XmlConfigChanger.java:77)
Caused by: com.xasecure.utils.install.XmlConfigChanger$ValidationException: ERROR: configuration token [XAAUDIT.DB.USER_NAME] is not defined in the file: [/usr/hdp/2.2.4.10-3/ranger-hdfs-plugin/install.properties]
at com.xasecure.utils.install.XmlConfigChanger.replaceProp(XmlConfigChanger.java:447)
at com.xasecure.utils.install.XmlConfigChanger.run(XmlConfigChanger.java:205)
... 1 more
*************************************************************************
ERROR: Unable to make changes to config. file: /usr/hdp/2.2.4.10-3/hadoop/conf/xasecure-audit.xml
... View more
07-20-2016
05:09 PM
I am facing a simillar issue, i am kind of new to the kms. it would really help if you can elaborate on the steps.
... View more
06-17-2016
11:23 AM
Many Thanks for sharing this! it worked for me as well but I am not sure if this is the correct way of fixing it, or is it a only work around only? I need to put a fix in the production env for the same.
... View more
12-18-2015
05:30 PM
1 Kudo
Thanks Billie for your response! I was able run solr on yarn, the mistake was "site.global.app_root" did not have the correct name of my solr version which was solr-5.3.1 However when I stop the solr application via slider (slider stop solr-yarn8) and restart it, 1) the cores I created disappear and, which is bad. 2) new instances start on new ports, can I fix the ports? 3) also I am only able to connect to only one of the solr instances (solr UI). 4) Is it yet possible to deploy solr cloud on yarn using multiple instances of solr? Regards, Rakesh
... View more
12-11-2015
11:19 AM
Thanks for the response, but the slider application failed to start again. When I look at the HDFS path: [solr@sandbox solr-slider]$ hadoop fs -cat /user/solr/.slider/cluster/solr-yarn4/app_config.json
{
"schema" : "http://example.org/specification/v2.0.0",
"metadata" : { },
"global" : {
"site.global.gc_tune" : "-XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime",
"site.fs.default.name" : "hdfs://sandbox.hortonworks.com:8020",
"site.global.solr_host" : "${SOLR_HOST}",
"site.global.solr_opts" : "",
"zookeeper.hosts" : "sandbox.hortonworks.com",
"site.global.server_module" : "--module=http",
"site.global.stop_key" : "solrrocks",
"java_home" : "/usr/lib/jvm/java-1.7.0-openjdk.x86_64/",
"site.fs.defaultFS" : "hdfs://sandbox.hortonworks.com:8020",
"site.global.zk_timeout" : "15000",
"env.MALLOC_ARENA_MAX" : "4",
"zookeeper.path" : "/services/slider/users/solr/solr-yarn4",
"site.global.listen_port" : "8983",
"zookeeper.quorum" : "sandbox.hortonworks.com:2181",
"site.global.xmx_val" : "1g",
"site.global.zk_host" : "${ZK_HOST}",
"site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/solr-5.3.1-SNAPSHOT",
"application.def" : "/user/solr/.slider/package/solr-yarn/solr-on-yarn.zip",
"site.global.xms_val" : "1g"
},
"credentials" : { },
"components" : {
"slider-appmaster" : {
"jvm.heapsize" : "512M"
},
"SOLR" : { }
}
- The variable names "${ZK_HOST}" shoul nt they be replaced with actual values? - Where should I look for the Solr specific logs as I am not able to find anything in the container logs. - What is the value of ${AGENT_WORK_ROOT}? what is the absolute path? - Is there any detailed documentation on how to deploy Solr application on yarn via Slider.
Regards,
... View more
12-10-2015
05:32 PM
2 Kudos
Hi, I am trying to run solr on yarn using the link lucidworksSolrSlider, apart from taking help from slider.incubator.apache.org/docs/getting_started.html Here is my folder structure: [solrs@ip-10-0-0-217 solr-slider]$ ls -lrt
total 131744
-rw-rw-r--. 1 solrs solrs 3182 Dec 10 01:17 README.md
drwxrwxr-x. 4 solrs solrs 32 Dec 10 01:17 package
-rw-rw-r--. 1 solrs solrs 2089 Dec 10 01:17 metainfo.xml
-rw-rw-r--. 1 solrs solrs 11358 Dec 10 01:17 LICENSE
-rw-rw-r--. 1 solrs solrs 134874517 Dec 10 01:37 solr-on-yarn.zip
-rw-rw-r--. 1 solrs solrs 277 Dec 10 01:49 resources-default.json
-rw-rw-r--. 1 solrs solrs 1355 Dec 10 15:33 appConfig-default.json
appConfig-default.json: {
"schema": "http://example.org/specification/v2.0.0",
"metadata": {
},
"global": {
"application.def": "/user/solrs/.slider/package/solryarn/solr-on-yarn.zip",
"java_home": "/usr/jdk64/jdk1.8.0_40",
"site.global.app_root": "${AGENT_WORK_ROOT}/app/install/solr-5.2.0-SNAPSHOT",
"site.global.zk_host": "localhost:2181",
"site.global.solr_host": "${SOLR_HOST}",
"site.global.listen_port": "${SOLR.ALLOCATED_PORT}",
"site.global.xmx_val": "1g",
"site.global.xms_val": "1g",
"site.global.gc_tune": "-XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewG$
"site.global.zk_timeout": "15000",
"site.global.server_module": "--module=http",
"site.global.stop_key": "solrrocks",
"site.global.solr_opts": ""
},
"components": {
"slider-appmaster": {
"jvm.heapsize": "512M"
},
"SOLR": {
}
}
}
resources-default.json: {
"schema" : "http://example.org/specification/v2.0.0",
"metadata" : {
},
"global" : {
},
"components": {
"slider-appmaster": {
},
"SOLR": {
"yarn.role.priority": "1",
"yarn.component.instances": "3",
"yarn.memory": "1024"
}
}
}
Could you please suggest me what will be the value of below parameters in appConfig-default.json file: "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/solr-5.2.0-SNAPSHOT",
"site.global.solr_host": "${SOLR_HOST}",
"site.global.listen_port": "${SOLR.ALLOCATED_PORT}", Basically where should I find "/app/install/solr-5.2.0-SNAPSHOT"?? My Environment: HDP 2.3, Slider Core-0.80.0.2.3.2.0-2950 Thanks, hoping a quick reply.
... View more
Labels:
05-01-2015
01:58 AM
Hi, What happens if the exam is aborted because of faulty internet connection or some other technical bug? Or due to anything which is not under my control. Regards, Rakesh
... View more
01-15-2015
05:47 AM
some log from cloudera-scm-server: 2015-01-15 11:00:48,683 INFO Metric-schema-update:com.cloudera.cmon.components.MetricSchemaManager: Updating schema work aggregates
2015-01-15 11:00:50,314 INFO Metric-schema-update:com.cloudera.cmon.components.MetricSchemaManager: Registering work aggregates
2015-01-15 11:00:50,656 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy100.writeMetrics(Unknown Source)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:287)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:274)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:129)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
... 11 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
... 11 more
... View more
01-15-2015
05:39 AM
The cluster is pretty small, just 3 nodes with 1 gb ram each. ubuntu 14.04 Trusty 64 bit. It fails at the connect tests at the embedded database test page before installing the components. I tried different combination of roles so that only one node is not pressed for resources than others. That did work, but now the process hangs during installing various services on the nodes. The cloudera-scm-server has to be restarted manually, to proceed again. Thanks!
... View more
01-15-2015
03:12 AM
I tried to install cdh5.3 using cloudera manager, but the process hangs in between. Might be because of extra services provided by the cloudera manager enterprise edition are not supported on my hardware. I want to go back to cloudera manager express.
... View more
12-11-2014
06:57 AM
So concretely: 1) I have to remove cdh components like hadoop-namenode, etc.. (which I installed using repository, then apt-get install) 2) Remove all the directories I created for different hadoop componenets like /data/1/dfs/nn etc.. basically doing ' hadoop fs -rmr / ' and also removing the local directories. 3) Start the CM admin console and run the CM wizard to install cdh components using Parcels. (which will download everything again and install) Cant we make use of the existing installation of cdh components? May be the CM wizard finds that oh... the components are alraedy installed so it moves to the next step of defining roles and services? Do we really need to remove them in step 1? Thanks for the support!
... View more
12-11-2014
05:26 AM
Thanks for responding so quickly. I do not have significant data on my cluster, so I can go with this approach. I had installed the cdh components by adding the CDH 5 repository and then using apt-get install.. I followed most of the default configuration. As I understand the process: 1) Back up my configuration files, or any other non default configurations. 2) Install the components again via CM5. 3) Define services/roles. 4) reapply the configurations from step 1. Two doubts: Do I need to un install the earlier cdh packages before doing the above steps? And in step 2, will CM install the existing components again? Thanks for the support!
... View more
12-11-2014
04:43 AM
I have a CDH5.2 installation with yarn, hdfs, spark 1.1.0. I was able to successfully install CM5 using the manual path B: (http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v5-0-0/Cloudera-Manager-Installation-Guide/cm5ig_install_path_B.html#cmig_topic_6_6_10_unique_1). Now, I am able to see the cloudera manager admin screen, and also able to see my nodes under "Hosts" tab. when you go "Hosts>> Components" I am able to see my installed components. But not able to see anything under the "Cluster" tab. How do I go about configuring the cluster and the various services. Please let me know if I am on the right path. how do i move forward now? Regards, Rakesh
... View more
11-14-2014
03:43 AM
Looks like I have to try upgrading cdh to 5.2 and use SPARK that comes with it, but does support all modes of spark. i.e 'yarn-cluster', 'yarn-client' etc...
... View more
11-11-2014
06:09 AM
More Logs: Application application_1415193640322_0016 failed 2 times due to Error launching appattempt_1415193640322_0016_000002. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: java.io.EOFException at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:710) at org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readUTF(DataInputStream.java:609) at java.io.DataInputStream.readUTF(DataInputStream.java:564) at org.apache.hadoop.yarn.security.ContainerTokenIdentifier.readFields(ContainerTokenIdentifier.java:151) at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142) at org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerTokenIdentifier(BuilderUtils.java:262) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:696) ... 10 more at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:99) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnException): java.io.EOFException at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:710) at org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readUTF(DataInputStream.java:609) at java.io.DataInputStream.readUTF(DataInputStream.java:564) at org.apache.hadoop.yarn.security.ContainerTokenIdentifier.readFields(ContainerTokenIdentifier.java:151) at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142) at org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerTokenIdentifier(BuilderUtils.java:262) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:696) ... 10 more at org.apache.hadoop.ipc.Client.call(Client.java:1409) at org.apache.hadoop.ipc.Client.call(Client.java:1362) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy69.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) ... 5 more . Failing the application. When I go to node Manager logs: Log Type: stderr Log Length: 87 Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher
... View more
11-11-2014
05:04 AM
Hello!! I have a similar issue, I am having CDH 5 installed on my cluster (version Hadoop 2.3.0-cdh5.1.3) I have installed and configured a prebuilt version of Spark 1.1.0 (Apache Version), built for hadoop 2.3 on my cluster. when I run the Pi example in the ‘client mode’, it runs succesfully, but it fails in the ‘yarn-cluster’ mode. The spark job is successfully submitted, but fails after sometime saying: *********************************** $ ./bin/spark-submit –class org.apache.spark.examples.SparkPi –master yarn-cluster –num-executors 2 –driver-memory 500m –executor-cores 2 lib/spark-examples*.jar 3 Logs: 14/11/05 20:47:47 INFO yarn.Client: Application report from ResourceManager: application identifier: application_1415193640322_0013 appId: 13 clientToAMToken: null appDiagnostics: Application application_1415193640322_0013 failed 2 times due to AM Container for appattempt_1415193640322_0013_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: org.apache.hadoop.util.Shell$ExitCodeException: *********************************** Can you please suggest any solution. Do you think I should compile the spark code on my cluster. Or should I use Spark provided with CDH5.1 Any help will be appreciated!
... View more