We are using spark 1.3 on hdp2.2.4 and I found there is a bug in the spark-assembly-1.3.1.2.3.0.0-2557-hadoop2.7.1.2.3.0.0-2557.jar that ships with spark. the Mllib check for the version of numpy is incorrect and MLlib throws an exception. I know the fix, I have to change the below file in the jar: mllib/__init__.py" below is the current code in the above mention python file: import numpy if numpy.version.version < '1.4': raise Exception("MLlib requires NumPy 1.4+") It can be fixed by changing to: import numpy ver = [int(x) for x in numpy.version.version.split('.')[:2]] if ver < [1, 4]: raise Exception("MLlib requires NumPy 1.4+") I have tried editing the 'spark-assembly-1.3.1.2.3.0.0-2557-hadoop2.7.1.2.3.0.0-2557.jar', to correct the code. I un-zipped the jar file, fixed the code, re packed it using zip. But after placing the fix, it gives EOF error: Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, xxxxxx.xxxx.uk.hxxx): org.apache.spark.SparkException: Error from python worker:   /opt/anaconda/envs/sparkAnaconda/bin/python: No module named pyspark PYTHONPATH was:   /data/4/hadoop/yarn/local/usercache/xxxxxxxx/filecache/33/spark-assembly-1.3.1.2.3.0.0-2557-hadoop2.7.1.2.3.0.0-2557.jar java.io.EOFException   at java.io.DataInputStream.readInt(DataInputStream.java:392)   at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:163)   at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:86)   at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)   at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:105)   at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) ... View more

Hi, I am trying to run solr on yarn using the link lucidworksSolrSlider, apart from taking help from slider.incubator.apache.org/docs/getting_started.html Here is my folder structure: [solrs@ip-10-0-0-217 solr-slider]$ ls -lrt total 131744 -rw-rw-r--. 1 solrs solrs 3182 Dec 10 01:17 README.md drwxrwxr-x. 4 solrs solrs 32 Dec 10 01:17 package -rw-rw-r--. 1 solrs solrs 2089 Dec 10 01:17 metainfo.xml -rw-rw-r--. 1 solrs solrs 11358 Dec 10 01:17 LICENSE -rw-rw-r--. 1 solrs solrs 134874517 Dec 10 01:37 solr-on-yarn.zip -rw-rw-r--. 1 solrs solrs 277 Dec 10 01:49 resources-default.json -rw-rw-r--. 1 solrs solrs 1355 Dec 10 15:33 appConfig-default.json appConfig-default.json: { "schema": "http://example.org/specification/v2.0.0", "metadata": { }, "global": { "application.def": "/user/solrs/.slider/package/solryarn/solr-on-yarn.zip", "java_home": "/usr/jdk64/jdk1.8.0_40", "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/solr-5.2.0-SNAPSHOT", "site.global.zk_host": "localhost:2181", "site.global.solr_host": "${SOLR_HOST}", "site.global.listen_port": "${SOLR.ALLOCATED_PORT}", "site.global.xmx_val": "1g", "site.global.xms_val": "1g", "site.global.gc_tune": "-XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewG$ "site.global.zk_timeout": "15000", "site.global.server_module": "--module=http", "site.global.stop_key": "solrrocks", "site.global.solr_opts": "" }, "components": { "slider-appmaster": { "jvm.heapsize": "512M" }, "SOLR": { } } } resources-default.json: { "schema" : "http://example.org/specification/v2.0.0", "metadata" : { }, "global" : { }, "components": { "slider-appmaster": { }, "SOLR": { "yarn.role.priority": "1", "yarn.component.instances": "3", "yarn.memory": "1024" } } } Could you please suggest me what will be the value of below parameters in appConfig-default.json file: "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/solr-5.2.0-SNAPSHOT", "site.global.solr_host": "${SOLR_HOST}", "site.global.listen_port": "${SOLR.ALLOCATED_PORT}", Basically where should I find "/app/install/solr-5.2.0-SNAPSHOT"?? My Environment: HDP 2.3, Slider Core-0.80.0.2.3.2.0-2950 Thanks, hoping a quick reply. ... View more