Member since
07-04-2019
5
Posts
1
Kudos Received
0
Solutions
07-27-2020
12:19 AM
Does this work for a Spark 3.0?
... View more
11-18-2019
03:54 AM
I'm using HDP-3.1 (spark2.3, python 3.6, Centos7)
I was able to configure it to work with python2.7, but with 3.6 I got the following error when running a notebook:
Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
I have python 3.6 installed on all of the workers and environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set (see my configuration bellow):
Am I missing something?
... View more
Labels:
10-28-2019
07:56 AM
There is a really good article in IBM documentation about managing queue access with ACLs
(https://developer.ibm.com/hadoop/2017/06/30/managing-queue-access-acls/)
However when I change the ACL config and add specific group:
yarn.scheduler.capacity.root.default.acl_submit_applications=yarn,ambari-qa,clusterusers
And after I refresh queues with:
$mapred queue -showacls
I get the following output.
Queue acls for user : myuser
Queue Operations ===================== root default
My temporal solution was to set:
yarn.scheduler.capacity.root.default.acl_submit_applications=*
Which works (see the output):
Queue acls for user : myuser
Queue Operations ===================== root default SUBMIT_APPLICATIONS
Note: myuser is a part of a group clusterusers
I would like to know why this config does not work if I set some groups on my own. Any ideas?
... View more
Labels:
10-22-2019
06:31 AM
I can't find where could I disable authentication for a Web UI.
I'm using HDP 3.1.
Any help appreciated!
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)