Hi, I automate cluster creation using hdc command. When cluster is up and running I can pull details of the cluster using hdc describe-cluster --cluster-name my_cluster but it returns the same json I used to set this cluster up, with no details about hostnames/ips of each worker/master/compute server. In order to schedule some jobs I need to know how to connect to edge server (here master). I have a workaround that uses your proxy curl https://my-hortonworks-hdcloud/my_cluster/services/hdfs/ --insecure -v --user 'admin:password' and I extract ip-123-456-789-012.example.com domain and based on that I can get private ip of one of the master servers that runs namenode. But I guess there should be more elegant way. Hints? ... View more

Hi, I've set up a fresh cluster using HDC console. When following instructions: > export SPARK_MAJOR_VERSION=2 > spark-shell —master yarn [...] AccessControlException: Permission denied: user=cloudbreak, access=WRITE, inode=“/user/cloudbreak/.sparkStaging/application_1510057948417_0004":hdfs:hdfs:drwxr-xr-x Same happens with pyspark. It looks like home directory is missing but I'm unable to create one (lack of access to hdfs account). Is something missing in template or steps I follow. I can try to do workarounds like pyspark --master yarn --conf spark.yarn.stagingDir=/tmp/ but still I end up with: 17/11/07 13:30:59 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. Running example: spark-submit --conf spark.yarn.stagingDir=/tmp/ --class org.apache.spark.examples.SparkPi --master yarn --executor-memory 2G --num-executors 5 /usr/hdp/current/spark2-client/examples/jars/spark-examples_2.11-2.1.1.2.6.1.4-2.jar 100 failed with same issue, on RM site I can find: Application application_1510057948417_0022 failed 2 times due to AM Container for appattempt_1510057948417_0022_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://ip-172-30-12-239.example.com:8088/cluster/app/application_1510057948417_0022 Then click on links to logs of each attempt. Diagnostics: Failing this attempt. Failing the application. But there are no logs available for attempt and yarn cmd doesn't provide logs as well: "Can not find the logs for the application: application_1510057948417_0022 with the appOwner: cloudbreak" ... View more