@HoldYourBreath  To add to @GangWar  answer  Azure is your best bet as you want to install Oracle VirtualBox and import your  Cloudera Quickstart VM image. Don't forget to set up a Windows 10 with at most 16GB  with enough CPU's and remember to set up auto-shutdown to avoid extra costs when your VM  isn't running Create-windows-virtual-machine-in-azure   How to install windows 10 in Azure Hope this information is useful Happy hadooping ... View more

Hi  @Shelton    Steps followed when integrating NiFi with LDAP:-   Step 1:- Generated SSL certificates using tls toolkit   $ ./bin/tls-toolkit.sh standalone -n "ambari.server" -C 'CN=admin,OU=NIFI' -o admin   Generated files using above command:   [root@ambari nifi-toolkit]# tree admin/ admin/ ├── ambari.server │   ├── keystore.jks │   ├── nifi.properties │   └── truststore.jks ├── CN=admin_OU=NIFI.p12 ├── CN=admin_OU=NIFI.password ├── nifi-cert.pem └── nifi-key.key   Step 2:- Did NiFi SSL setup using Apache Ambari UI   2.1) Copied keystore.jks, nifi.properties & truststore.jks in NiFi conf directory   2.2) Configured Apache NiFi SSL setup https://drive.google.com/file/d/1DJUZE7cWseQkWl_En0_Rh_VSI39y3FiY/view?usp=sharing   2.3) Imported CN=admin_OU=NIFI.p12 certificate in browser for Initial Admin Login   2.4) Logged in with Initial Admin Identity via certificates. Also created users and given view & modify privileges in NiFi as mentioned in https://pierrevillard.com/2017/01/24/integration-of-nifi-with-ldap/comment-page-1/    Step 3:- Did NiFi LDAP configuration   LDAP configuration: <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=admin,dc=example,dc=org</property> <property name="Manager Password">admin</property> <property name="Referral Strategy">IGNORE</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://ldap_ip_address:389</property> <property name="User Search Base">dc=example,dc=org</property> <property name="User Search Filter">cn={0}</property> <property name="Authentication Expiration">12 hours</property> </provider>   Just wondering is this host existing ?? >> Yes, LDAP and the associated LDIF host exist and tested with ldapsearch   [root@ambari nifi-toolkit]# ldapsearch -x -b "dc=example,dc=org" -h ldap_ip_address -p 389 -D "cn=admin,dc=example,dc=org" -W Enter LDAP Password: # extended LDIF # # LDAPv3 # base <dc=example,dc=org> with scope subtree # filter: (objectclass=*) # requesting: ALL # # example.org dn: dc=example,dc=org objectClass: top objectClass: dcObject objectClass: organization o: Example Inc. dc: example # admin, example.org dn: cn=admin,dc=example,dc=org objectClass: simpleSecurityObject objectClass: organizationalRole cn: admin description: LDAP administrator userPassword:: e1NTSEF9TWpBN21xQ3VVdURYQ1k1cVpRd0x0NjE0NE9HcnVEQm0= # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2   After following above steps when I try to login with LDAP users, I am getting unable to login in NiFi UI   Can you suggest where things are going wrong? ... View more

Thanks @Shelton for your detailed answer! ... View more

@PauloNeves  Yes, the command show databases will list all databases in a Hive instance whether you are authorized to access it or not. I am sure this is cluster devoid of Ranger or Sentry which are the 2 authorization tools in Cloudera!!!   Once the ranger plugin is enabled then authorization is delegated to Ranger to provide fine-grained data access control in Hive, including row-level filtering and column-level masking. This is the recommended setting to make your database administration easier as it provides a centralized security administration, access control, and detailed auditing for user access within the Hadoop, Hive, HBase, and other components in the ecosystem.   Unfortunately, I had already enabled the Ranger plugin for hive on my cluster but all the same, it confirms what I wrote above. Once the ranger plugin is enabled for a component ie. hive,HBase or Kafka then the authorization is managed exclusively through Ranger Database listing before Ranger Below is what happens if my user sheltong has not explicitly been given authorization through Ranger, see [screenshots] I see no database though I have over 8 databases  See the output of the hive user who has explicit access to all the tables due to the default policy he could see the databases. Database listing after Ranger After creating a policy explicitly giving the user sheltong access to the 3 databases  Policy granting explicit access to 3 databases Now when I re-run the show databases bingo! Back to your question show tables from forbidden_db, it returns an empty list, this can be true especially if the database is empty! has not table like the screenshot below though I have access to the database it's empty Now I create a table and re-run the select now I am able to see the table I hope this demonstrates the power of Ranger and explains maybe what you are encountering, I am also thinking if your cluster has Ranger hive plugin enabled you could have select on the databases but you will need explicit minimum select or the following permission on the underlying database  tables to be able to see them. Happy Hadooping   ... View more

@Chahat_0  Hadoop is designed to ensure that compute (Node Managers) runs as close to data (Data Nodes) as possible. Usually containers for jobs are allocated on the same nodes where the data is present. Hence in a typical Hadoop cluster, both Data Nodes and Node Manager run on the same machine.   Node Manager is the RM slave process while the Data Nodes is the Namenode slave process which  responsible for coordinating HDFS functions   Resource Manager: Runs on a master daemon and manages the resource allocation in the cluster. Node Manager: They run on the slave daemons and are responsible for the execution of a task on every single Data Node Node Managers manage the containers requested by jobs Data Nodes manage the data The NodeManager (NM) is YARN’s per-node agent and takes care of the individual compute nodes in a Hadoop cluster. This includes keeping up-to-date with the ResourceManager (RM), overseeing containers’ life-cycle management; monitoring resource usage (memory, CPU) of individual containers, tracking node-health, log’s management, and auxiliary services that may be exploited by different YARN applications. NodeManager communicates directly with the ResourceManager. Resource manager and Namenode both as master components [processes] that can run in single or HA setup should run on separate identical usually high spec servers [nodes] as compared to the data nodes. Zookeeper is another important component   ResourceManager and NodeManager combine together to form a data-computation framework. ResourceManager acts as the scheduler and allocates resources amongst all the applications in the system. NodeManager takes navigation from the ResourceManager and it runs on each node in the cluster. Resources available on a single node is managed by NodeManager. ApplicationMaster, a framework-specific library is responsible for running specific YARN job and for negotiating resources from the ResourceManager, and working with NodeManager to execute and monitor containers. Hope that helps  ... View more

Hi@Shelton    Thank you for the response. I got it.   It helped.   Best regards, ... View more

@prasanna06    Your problem resembles this one  check your cluster UI to ensure that workers are registered and have sufficient resources    Happy hadooping     ... View more

@bvishal  I provided an answer to such a situation Ambari MySQL database lost please have a look at it and see if that resolves you problem it did for someone in a similar situation. Happy Hadooping ... View more

@Sud  Your question isn't detailed. What sort of access are you thinking of to restrict as read-only data or UI? For Ambari you have the Cluster User role which is a read-only for its services, including configurations, service status, and health alerts. Then the other is about reading data in HDFS where you can use HDFS ACL's which is POSIX compliant like rwx but that won't work for Hive tables. You should know that Ranger controls authorization for the following HDFS,Hive,HBase,Kafka,Knox,YARN,Storm,Atlas and other components depending oon the software HDP,CDH or CDP. Happy hadooping ... View more