@mike_bronson7    Docker containers provide an ideal foundation for running Kafka-as-a-Service on-premises or in the public cloud. However, using Docker containers in production environments poses some challenges including container management, scheduling, network configuration and security, and performance Containerized applications have no resource constraints and can use as much of a given resource as the host’s kernel scheduler allows and also each container’s access to the host machine’s CPU cycles is unlimited. It is important not to allow a running container to consume too much of the host machine’s memory As you are aware kafka will need Zookeepers so you got to architect well your Kafka deployment but once you master then its a piece of cake and it brings a lot of advantages like upgrades, scaling out, etc   As I reiterated that a good move get you hands dirty 🙂 ... View more

@moubaba  I know its a dompting task but unfortunately, I could create a link by an article for Jordan Moore let me try to elaborate.    Simple advice always run a crontab job to dump all your databases or at leat the ambari DB  nightly !   Firstly Ambari doesn't control any data residing on the data nodes, so you should be safe there. Stop all ambari-agents in the cluster, maybe even uninstalling it so as to let all the Hadoop components remain running "in the dark". Install and set up a new Ambari server, add a cluster but do not register any  hosts [very important]. If you uninstalled the ambari -agents on the nodes please do re-install ensure the ambari-server and the agents are the same versions. Do the below if you removed Ambari and the agents # yum repolist On the ambari-server # yum install ambari-server # yum install ambari-agent On the other hosts # yum install ambari-agent Reconfigure manually the ambari agents [ambari-agent.ini] to point at the new Ambari server address, and start them.   Add the hosts in the Ambari server UI, selecting "manual registration" option the hosts register successfully since the agents are running a point to note install and configure an ambari-agent on the Ambari server to point to itself !!!! After this, you get the option of installing clients and servers. Now, you could try to "reinstall" what is already there, but you might want to deselect all the servers on the data node column. In theory, it will try to perform OS package installation, and say that the service already exists, and doesn't error out. If it does error, then restart the install process, but deselect everything. At which point, it should continue and now you have Ambari back up and running with all the hosts monitored, just with no processes to configure. To add the services back, you would need to use the Ambari REST API to add back in the respective Services, Components, and Host Components that you have running on the cluster. If you can't remember all the Services, Components, and Host Components then go to each host and do a process check to see what's running.   Variables export AMBARI_USER=admin export AMBARI_PASSWD=admin export CLUSTER_NAME=<New_Cluster_name> export AMBARI_SERVER=<Your_new_ambari_server_FQDN> export AMBARI_PORT=8080 List all services related to hive In the below example I am listing all hive components  curl -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X GET "http://$AMBARI_SERVER:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/services/HIVE" Adding back pig curl -k -u $AMBARI_USER:$AMBARI_PASSWD -H "X-Requested-By:ambari" -i -X POST -d '{"RequestInfo":{"context":"Install PIG"}, "Body":{"HostRoles":{"state":"INSTALLED"}}}' 'http://'$AMBARI_SERVER':$AMBARI_PORT/api/v1/clusters/'$CLUSTER_NAME'/services/PIG/components/PIG' This is the time to get your hands dirty 🙂 This isn't production so it should train you for a real situation Happy hadooping !! ... View more

@MIkeL  The best technical reference before you embark on deploying your cluster is to check the compatibility of the different moving parts of HDP/Cloudera binaries against an operating system of your choice, the first source of truth is please filter all the possible valid options using   supportmatrix cloudera/hortonworks tool Hortonworks and Cloudera do run exhaustive tests on a particular Operating system before certifying it as production-ready and from the about  RHEL/Centos 7.7 are not yet certified so I highly doubt whether  RHEL/Centos 8  is certified that explains the Python errors you are encountering. HTH ... View more

@Mnju  Ranger and Sentry don't offer data quality but a centralized security framework to manage fine-grained access control and policies across the cluster. Security administrators use it to easily manage policies for access to files, folders, databases, tables, or columns. These policies can be set for individual users or groups and then enforced consistently across the Cluster.   The latest version of Ranger that ships with CDP now available for AWS and later this year for Azure manage access and authorization to the below resources using ranger plugins. HDFS Hive Ozone Atlas Nifi-Registry Storm HBase Knox Kafka YARN NiFi Solr   Sentry is a granular, role-based authorization module for Hadoop and provides the ability to control and enforce precise levels of privileges on data for authenticated users and applications on a Hadoop cluster. It works out of the box with Apache Hive, Hive Metastore/HCatalog, Apache Solr, Impala, and HDFS (limited to Hive table data) and allows you to define authorization rules to validate a user or application’s access requests for Hadoop resources Both are security tools built for Hadoop that are usually combined with Kerberos, KMS, and TLS to provide a robust security framework. Data quality is a broad subject of discussion but to my knowledge, no tool manages data quality unless you are talking of catalog tools like Alation, waterline. Atlas is a tool that provides metadata management, data lineage and governance capabilities to build a catalog of data assets, classification, and governance across these assets. HTH   ... View more