Context As of CDP 7.1.2, Sentry is deprecated with Kudu, and Ranger becomes the solution for Fine-Grained Authorization in Kudu. The question this article is meant to address is what does it mean for the Impala/Kudu stack to enable Ranger for Kudu, especially if Impala is already using Ranger? This is not obvious up front because the information is spread over three different sets of documents.   Fundamentally the answer is nothing. The reason for this is straightforward: enabling the Kudu module in Ranger should automatically configure the --trusted_user_acl flag in Kudu to include Impala (if installed) and Hive (if installed). What this flag does is exempt the listed users from being checked against Kudu's authorization model, which in CDP installations mean Ranger policies. So, the correct way to enforce access to Impala and Hive tables, which are stored as Kudu, is the Hadoop-SQL policies set for Impala and Hive.   By default, enabling Ranger for Kudu should have no impact on your Impala or Hive operations; any further changes you want to make to authorization should be done in the Hadoop-SQL policies.   What is the motivation for using Ranger with Kudu, if that is the case? These tables are all still accessible at the Kudu level by users, and changes made from the Kudu level can cause inconsistencies or data loss. Enabling Ranger at the Kudu level prevents this.   There are two other use-cases where policies are set at the Kudu level in cm_kudu: these are Spark and NiFi. These services are on a by-user basis instead of having a service user communicate with Kudu; so the completely normal logic of writing Ranger policies applies. Please see the documentation of the respective service for more information. ... View more

The Qualys tool reports vulnerabilities in ZooKeeper, even when the ZooKeeper security configuration is applied (HDP doc, CDP doc). There are two kinds of reports Qualys makes that are not addressable by Cloudera: The security guidance keeps several znodes in the affected services as world-readable, like so: /zookeeper/quota sasl:zookeeper:cdrwa,world:anyone:r Any world-readable znode will appear on the scanner, and will require an exception filed for it. This is the position of Qualys, as reported by our customers who use Qualys. The security guidance does not cover several services. The following components require no action according to our documentation: Calcite Knox MapReduce Spark Tez Zeppelin The Qualys tool will still report znodes owned by these services. Note: it is possible to harden the ACLs beyond the Best Practices recommendation in the documentation, and to harden the ACLs of services not covered in the Best Practices. However, Cloudera cannot provide what the correct ACLs are in that case. Testing on the customer's side is required. It is very easy to set the ACLs such that services that need access to the znode will not have it, and this needs to be handled on a znode by znode basis. If an attempt at hardening the ACLs is going to be made, these suggestions may help: Try implementing SASL (this is the same method used in most of the Best Practices recommendations) Try restricting privileges to the service user If something breaks, try to identify the user that performed the failed action, and add the necessary privileges only for that user ... View more

Note: Cloudera does not support antivirus software of any kind. This article contains recommendations for excluding HDP components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude which are unique to them. These details will be addressed in individual articles dedicated to the service in question. The three primary locations you will want to exclude from Antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes. In general, consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /hadoop /usr/hdp /etc/hadoop /etc/<component> /var/lib/<component> Runtime and Logging /var/run/<component> /var/log/<component> Scratch and Temp /var/tmp/<component> /tmp/<component> Note: The <component> does not only refer to the service name, as a given service may have multiple daemons with their own directories. Example: ambari-agent and ambari-server. Across HDP services there are also many user-configurable locations. Most of these can be found in Ambari properties with names like 'service.scratch.dir' and 'service.data.dir'; go to Ambari > Service > Configs > Advanced and search for any property containing "dir", all of which may be considered for exclusion. Instructions for specific services follow:   Ambari: Note: Ambari has a special requirement in the form of a user-configurable database. I recommend you exclude this database. However, the details of this database are set on installation; the database may be colocated with ambari-server, or on a remote host. Consult with your database administrators for details on the path where the database information is stored; Ambari does not keep this information anywhere in its configuration. If you need details about which database Ambari is using, search for JDBC in the amber.properties file. # grep 'jdbc' /etc/ambari-server/conf/ambari.properties Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /usr/hdp /usr/lib/ambari-agent /usr/lib/ambari-server /etc/hadoop /etc/ambari-agent /etc/ambari-server /var/lib/ambari-agent /var/lib/ambari-server Runtime and Logging /var/run/ambari-agent /var/run/ambari-server /var/log/ambari-agent /var/log/ambari-server HDFS: Note: The directories in HDFS are user-configurable. I recommend you exclude these, especially the data directory for the DataNode and the meta directories for the NameNode and JournalNode. These details can be found in the 'hdfs-site.xml' file: # grep -A1 "dir" /etc/hadoop/conf/hdfs-site.xml Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /usr/hdp /etc/hadoop /var/lib/hadoop-hdfs Runtime and Logging /var/run/hadoop /var/log/hadoop Scratch and Temp /tmp/hadoop-hdfs Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to exclude the other components. YARN: Note: The directories YARN uses are user-configurable. I recommend you exclude them. These properties can be found in Ambari > YARN > Configs > Advanced: yarn.nodemanager.local-dirs yarn.nodemanager.log-dirs yarn.nodemanager.recovery.dir yarn.timeline-service.leveldb-state-store.path yarn.timeline-service.leveldb-timeline-store.path Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /usr/hdp /etc/hadoop /var/lib/hadoop-yarn Runtime and Logging /var/run/hadoop-yarn /var/log/hadoop-yarn Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. MapReduce: Note: Some directories in MapReduce are user-configurable. I recommend you exclude them. These properties can be found in Ambari > YARN > Configs > Advanced and this one, in particular, should be excluded: mapreduce.jobhistory.recovery.store.leveldb.path Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /usr/hdp /etc/hadoop /var/lib/hadoop-mapreduce Runtime and Logging /var/run/hadoop-mapreduce /var/log/hadoop-mapreduce Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. ZooKeeper: Note: ZooKeeper has a user-configurable data directory. I recommend you exclude it. This directory can be found by running the following command: # grep dataDir /etc/zookeeper/conf/zoo.cfg Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /usr/hdp /etc/hadoop Runtime and Logging /var/run/zookeeper /var/log/zookeeper Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. ... View more

Note: Cloudera does not support Antivirus software of any kind. This article contains recommendations for excluding CDH components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude which are unique to them. These details will be addressed in individual articles dedicated to the service in question. The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore, take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes. In general, consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera /etc/<component> /var/lib/<component> Runtime and Logging /var/run/<component> /var/log/<component> Scratch and Temp /var/tmp/<component> /tmp/<component> Note: The <component> does not only refer to the service name, as a given service may have multiple daemons with their own directories. Example: cloudera-scm-server and cloudera-scm-agent. Across CDH services, there are also many user-configurable locations. Most of these can be found in Cloudera Manager properties with names like service.scratch.dir and service.data.dir; go to CM > Service > Configurations and search for any property containing "dir", all of which may be considered for exclusion. Instructions for specific services follow: Cloudera Manager: Note: Cloudera Manager has a special requirement in the form of a user-configurable database. I recommend you exclude this database. However, the details of this database are set on installation; the database may be colocated with cloudera-scm-server, or on a remote host. Consult with your database administrators for details on the path where the database information is stored. Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera/cm /opt/cloudera/cm-agent /etc/cloudera-scm-agent /etc/cloudera-scm-server /var/lib/cloudera-host-monitor /var/lib/cloudera-scm-agent /var/lib/cloudera-scm-eventserver /var/lib/cloudera-scm-server /var/lib/cloudera-scm-server-db /var/lib/cloudera-service-monitor Runtime and Logging /var/run/cloudera-scm-agent /var/run/cloudera-scm-server /var/log/cloudera-scm-agent /var/log/cloudera-scm-alertpublisher /var/log/cloudera-scm-eventserver /var/log/cloudera-scm-firehose /var/log/cloudera-scm-server HDFS: Note: The directories in HDFS are user-configurable. I recommend you exclude these, especially the data directory for the DataNode and the meta directories for the NameNode and JournalNode. These details can be found in the hdfs-site.xml file: # grep -A1 "dir" /etc/hadoop/conf/hdfs-site.xml Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera /var/lib/hadoop-hdfs Runtime and Logging /var/log/hadoop-hdfs Scratch and Temp /tmp/hadoop-hdfs Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to exclude the other components. YARN: Note: The directories YARN uses are user-configurable. I recommend you exclude them. These properties can be found in Cloudera Manager > YARN > Configuration: yarn.nodemanager.local-dirs yarn.nodemanager.log-dirs yarn.nodemanager.recovery.dir yarn.timeline-service.leveldb-state-store.path yarn.timeline-service.leveldb-timeline-store.path Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera /var/lib/hadoop-yarn Runtime and Logging   /var/log/hadoop-yarn   Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. MapReduce: Note: Some directories in MapReduce are user-configurable. I recommend you exclude them. These properties can be found in Cloudera Manager > MapReduce > Configs, and this one, in particular, should be excluded: mapreduce.jobhistory.recovery.store.leveldb.path Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera /var/lib/hadoop-mapreduce Runtime and Logging   /var/log/hadoop-mapreduce   Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. ZooKeeper: Note: ZooKeeper has a user-configurable data directory. I recommend you exclude it. This directory can be found by running the following command: # grep dataDir /etc/zookeeper/conf/zoo.cfg Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera Runtime and Logging /var/log/zookeeper Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. ... View more

Note: Cloudera does not support Antivirus software of any kind. This article contains recommendations for excluding CDP components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude that are unique to them. These details will be addressed in individual articles dedicated to the service in question. The three primary locations you will want to exclude from Antivirus are: Data directories: These can be very large, and therefore, take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes. In general, consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera /etc/<component> /var/lib/<component> Runtime and Logging /var/run/<component> /var/log/<component> Scratch and Temp /var/tmp/<component> /tmp/<component> Note: The <component> does not only refer to the service name, as a given service may have multiple daemons with their own directories. Example: cloudera-scm-agent and cloudera-scm-server Across CDP services, there are also many user-configurable locations. Most of these can be found in Cloudera Manager properties with names like "service.scratch.dir" and "service.data.dir"; go to Cloudera Manager > Service > Configuration and search for any property containing "dir", all of which may be considered for exclusion. Instructions for specific services follow: Cloudera Manager: Note: Cloudera Manager has a special requirement in the form of a user-configurable database. I recommend you to exclude this database. However, the details of this database are set on installation; the database may be co-located with cloudera-scm-server, or on a remote host. Consult with your database administrators for details on the path where the database information is stored. Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera/cm /opt/cloudera/cm-agent /etc/cloudera-scm-agent /etc/cloudera-scm-server /var/lib/cloudera-host-monitor /var/lib/cloudera-scm-agent /var/lib/cloudera-scm-eventserver /var/lib/cloudera-scm-server /var/lib/cloudera-scm-server-db /var/lib/cloudera-service-monitor Runtime and Logging /var/run/cloudera-scm-agent /var/run/cloudera-scm-server /var/run/cloudera-scm-server-db /var/log/cloudera-scm-agent /var/log/cloudera-scm-alertpublisher /var/log/cloudera-scm-eventserver /var/log/cloudera-scm-firehose /var/log/cloudera-scm-server HDFS: Note: The directories in HDFS are user-configurable. I recommend you exclude these, especially the data directory for the DataNode and the meta directories for the NameNode and JournalNode. These details can be found in the hdfs-site.xml file: # grep -A1 "dir" /etc/hadoop/conf/hdfs-site.xml Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera /var/lib/hadoop-hdfs Runtime and Logging /var/log/hadoop-hdfs Scratch and Temp /tmp/hadoop-hdfs Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to exclude the other components. YARN: Note: The directories YARN uses are user-configurable. I recommend you exclude them. These properties can be found in Cloudera Manager > YARN > Configuration: yarn.nodemanager.local-dirs yarn.nodemanager.log-dirs yarn.nodemanager.recovery.dir yarn.timeline-service.leveldb-state-store.path yarn.timeline-service.leveldb-timeline-store.path Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera /var/lib/hadoop-yarn Runtime and Logging /var/log/hadoop-yarn Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. MapReduce: Note: Some directories in MapReduce are user-configurable. I recommend you exclude them. These properties can be found in Cloudera Manager > MapReduce > Configs, and this one, in particular, should be excluded: mapreduce.jobhistory.recovery.store.leveldb.path Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera /var/lib/hadoop-mapreduce Runtime and Logging /var/log/hadoop-mapreduce Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. ZooKeeper: Note: ZooKeeper has a user-configurable data directory. I recommend you exclude it. This directory can be found by running the following command: # grep dataDir /etc/zookeeper/conf/zoo.cfg Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera Runtime and Logging /var/log/zookeeper Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. ... View more