As far as I know, up to Knox-0.6 (currently used in HDP) there can be only one Auth. provider per topology. Last time when I faced such a requirement I created two topologies, one for each Auth. provider. Edit: KNOX-236 is going to provide such a feature but "in the future". ... View more

Hi @Rajendra Vechalapu, you can omit setting master in your source, see this example: val conf = new SparkConf().setAppName("Spark Pi") val spark = new SparkContext(conf) You can then launch your application using spark-submit and provide the master there there using "--master" and "--deploy-mode" options. Refer to Spark programming guide for this and other useful hints. Edit: When you run spark-submit on Sandbox, be sure to supply additional arguments for master, num-executors, driver-memory, executor-memory, and executor-cores as given below. Note that larger values for last 4 arguments will not work on the Sandbox! Follow (and you can also try) this example computing Pi in Python (as any user who has access to HDFS/Yarn): cd /usr/hdp/current/spark-client/ spark-submit --master yarnclient --num-executors 1 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/src/main/python/pi.py 10 "--master yarncluster" works too. You can also set these 4 in spark-env in Ambari. They are already there but commented out, and not all with values like here. See also Spark guide on HDP. ... View more

I saw this, and I think it's fixed in latest Ambari versions. Can you try: cd /usr/lib/python2.6/site-packages/resource_monitoring/ python psutil/build.py and restart metrics monitors. ... View more

Use fsck, it's a tool of choice to manage HDFS. "Orphans" are corrupted files (with missing blocks) in HDFS lingo. You can use "-move" or "-delete" options to move corrupted files to /lost+found or to delete them. fsck will also tell you about under-replicated blocks (having at least 1 replica but less than configured replication factor) but HDFS will repair them little by little by creating missing replicas. ... View more