Hi community, looking at security, I am in process of disabling any interfaces without proper authentication / authorization (or even encryption). I came across the debug web UIs of Cloudera Management services.   According to https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_ports_cm.html, the debug WebUIs can be disabled by setting the port property to -1. This works for Reports Manager (8083), Event Server (8084), Navigator Audit Server (8089), Telemetry Publisher (10111).   This does not work, however, for  Service Monitor (8086 / 9086 TLS), Activity Monitor (8087 / 9087 TLS), Host Monitor (8091 / 9091 TLS). Setting port to -1 leads to non-starting services without a proper ERROR in the log file.   Cloudera Manager agent even tries to check, if the server successfully bound to port -1 and runs into errors: [15/Aug/2019 12:06:03 +0000] 65646 Thread-14 process ERROR [918-cloudera-mgmt-HOSTMONITOR] Failed port check: Command '['ss', '-np', 'state', 'listening', '(', 'sport', '=', '-1', 'or', 'sport', '=', '9995', 'or', 'sport', '=', '9994', ')']' returned non-zero exit status 255     How do you disable the debug web UIs for those management services. Or is there a way to properly secure them by authentication and authorization?   Thanks and best regards Benjamin ... View more

Hi community,   I got CDH 5.16 and a cluster with Impala / Hive / Sentry (database-backed) set up.   In Impala, I got different databases and want to define policies so that a group / role can access all databases read-only.   I tried this Grant but it does not work:   "GRANT SELECT ON DATABASE ALL TO ROLE my_role;"     How can I define SELECT-permissions for all databases in Sentry?   Thanks! Benjamin ... View more

Hi community, I've got a Spark 2.3 application in which I need to broadcast rather large (1-3 GB) objects. To do so, I am collecting the DataSets and broadcasting them. Performance measurements show, that the driver spends a long time, serializing the objects, which are fairly complex indeed. During broadcasting / serialisation, the driver is only busy doing this task. I am wondering, how to reduce this waiting-time. Is there a way to parallelize tasks such as broadcasting / serialization on the driver? It would be for instance be helpful to erform multiple broadcasts in parallel, continue with other driver code during broadcasting or having a way to parallelize an individual broadcasting. Best, Benjamin ... View more

Hi community, does anybody have experience with the following scenario in Kerberos setup: My scenario: Run every Hadoop Service as the same Linux user, let’s call him myhadoopuser. Create the following principals for the cluster: Service Principals for all nodes: myhadoopuser/_HOST HTTP/_HOST Shared user principal for HDFS, ambari smoketest, spark, ... All of the above principals are mapped to myhadoopuser with auth_to_local My question is: Does it work technically and are there strong reasons not do it? Potential issues I see: missing isolation among Hadoop services (anybody got an example what would be possible and why this is bad?) inability to set different proxyuser privileges for different services Thank you! Best, Roland ... View more

Hi guys, I have problems upgrading from HDP 2.5.0 to 2.6.0 on a RHEL 6.7 system. The Ambari Upgrade to 2.5.0 is finished. Installing packages for HDP 2.6, Ambari fails with this error: Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_6_0_3_8-client' returned 1. Error: Package: hadoop_2_6_0_3_8-hdfs-2.7.3.2.6.0.3-8.x86_64 (HDP-2.6.0.3) Requires: libtirpc-devel You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Traceback (most recent call last): And indeed, the package is not available. # yum install libtirpc-devel Loaded plugins: product-id, security, subscription-manager Setting up Install Process No package libtirpc-devel available. Error: Nothing to do Is there guidance, how to work around this problem? RHEL 6.7 is explicitly supported by Ambari 2.5 and HDP 2.6. Installing unofficial packages is not allowed in our environment. Thanks for your help PS: I saw this community questions which, however, is about RHEL 7.x: https://community.hortonworks.com/questions/96763/hdp-26-ambari-install-fails-on-rhel-7-on-libtirpc.html ... View more

Hi, I am running a HDP 2.5 cluster, which is secured by Kerberos and Ranger. On a folder, I have set permissive HDFS POSIX rights (600) on files and manage all access rights via Ranger. The folder is exported via NFS gateway and mounted on some client (options rw,sync,vers=3,proto=tcp,nolock). The UIDs are all synchronized over the hosts. In this folder, a user, which has Read, Write, Execute via Ranger but is neither owner nor in the group of the file, IS able to read and delete files. However, he IS NOT allowed to "touch" files or change the owner. The owner of the file, however, is able to run the touch command on these files. [ someuser@host ]$ ll total 2 drwxr-xr-x 6 hdfs hdfs 192 Jan 26 15:01 . drwxr-xr-x 4 hdfs hdfs 128 Sep 9 15:27 .. -rw-r--r-- 1 otheruser hdfs 0 Sep 9 11:11 test1.txt -rw-rw-r-- 1 someuser hdfs 0 Jan 26 15:03 test2.txt [ someuser@host ]$ touch test2.txt [ someuser@host ]$ touch test1.txt touch: cannot touch `test1.txt': Permission denied Are the Ranger policies not applied on metadata changes on files, such as changing the access time or the owner? Is this a bug or by design? The "Permission Denied" is not even audited in the Ranger Audit Log. The allowed touch-command is however audited. Thanks for your help ... View more