Cloudera Community
Shelton
user rank
Master Mentor

Re: hadoop + ambari cluster change configuration

by Master Mentor in Support Questions
‎08-09-2017 08:33 PM
‎08-09-2017 08:33 PM
@uri ben-ari The curl command you have run is a GET not POST. To register a blueprint with Ambari you should have done this while in the directory where the blueprint is located. Below is an example curl -H "X-Requested-By: ambari"-X POST -u admin:admin http://<ambari-server>:8080/api/v1/blueprints/multinode-hdp -d @1-HDP6_blueprint.json Hope that helps ... View more

Re: Kerberos Setup on HDP 2.6

by Master Mentor in Support Questions
‎08-09-2017 02:47 PM
‎08-09-2017 02:47 PM
Assumption REALM=GOLD.COM REALM Password =welcome1 KDC_HOST =KDC_SERVER_FQDN Install kerberos package yum install -y krb5-server krb5-libs krb5-workstation Edit Server Configuration Files The configuration files for the server are located in the directory /var/kerberos/krb5kdc edit the 2 files kdc.conf kadm5.acl The conf file is the server configuration and the acl file, well the ACL. The ACL grants all privileges to anyone one with the admin role. Change the Kerberos GOLD.COM realm cd /var/kerberos/krb5kdc/ vi kadm5.acl replace the REALM after the @ with the below */admin@GOLD.COM * Edit the kdc.conf [kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88 [realms] GOLD.COM = { #master_key_type = aes256-cts acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/share/dict/words admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal } Edit the Client Configuration Update the krb.conf on the KDC server copy this file to all the host in the cluster in the below directory /etc/krb5.conf vi /etc/krb5.conf # Configuration snippets may be placed in this directory as well includedir /etc/krb5.conf.d/ [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] dns_lookup_realm = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true rdns = false default_realm = GOLD.COM default_ccache_name = KEYRING:persistent:%{uid} [realms] GOLD.COM = { kdc = KDC_SERVER_FQDN admin_server = KDC_SERVER_FQDN } [domain_realm] .gold.com = GOLD.COM gold.com = GOLD.COM Create the DC database When creating the KDC database we will need an entropy pool of random data sudo kdb5_util create -s -r GOLD.COM Loading random data Initializing database '/var/kerberos/krb5kdc/principal' for realm 'GOLD.COM', master key name 'K/M@GOLD.COM' You will be prompted for the database Master Password. It is important that you NOT FORGET this password.Enter KDC database master key: welcome1 Re-enter KDC database master key to verify: welcome1 Start and Enable Kerberos Start and enable the two services ensure you run status command to validate that kadmin is running # sudo systemctl start krb5kdc kadmin # sudo systemctl enable krb5kdc kadmin Created symlink from /etc/systemd/system/multi-user.target.wants/krb5kdc.service to /usr/lib/systemd/system/krb5kdc.service. Created symlink from /etc/systemd/system/multi-user.target.wants/kadmin.service to /usr/lib/systemd/system/kadmin.service. Check the Principals # sudo kadmin.local Authenticating as principal root/admin@GOLD.COM with password. kadmin.local: listprincs K/M@GOLD.COM kadmin/admin@GOLD.COM kadmin/changepw@GOLD.COM kadmin/KDC_SERVER_FQDN@GOLD.COM kiprop/KDC_SERVER_FQDN@GOLD.COM krbtgt/UPUTEST.CH@UPUTEST.CH kadmin.local: Must create a root principal for kerberization kadmin.local: addprinc root/admin WARNING: no policy specified for root/admin@GOLD.COM; defaulting to no policy Enter password for principal "root/admin@GOLD.COM": Re-enter password for principal "root/admin@GOLD.COM": Principal "root/admin@GOLD.COM" created. Install kerberos Client package on tall the hosts in the cluster install the client yum install -y krb5-libs krb5-workstation This will deliver the necessary files copy the krb5.conf from the KDC server to all the client hosts Invoke kerberos tool in Ambari When running the kerberization tool in Ambari use root/admin@GOLD.COM password welcome1 Follow the instructions take care that you installed JCE on the KDC server ... View more

Re: Beeline connection throws error

by Master Mentor in Support Questions
‎08-09-2017 06:03 AM
‎08-09-2017 06:03 AM
@pv poreddy Here are the different methods to connect. Assumptions: HS2 Host(s): test01.com and test02.com HS2 Binary Port: 10010 HS2 HTTP Port: 10011 ZooKeeper Quorom: test01.com:2181,test02.com:2181:test03.com:2181 HttpPath: cliservice HS2 ZooKeeper Namespace: hiveserver2 User: hiveuser Password: hivepswd NOTE: <db> is the database in the examples below and is optional. The leading slash '/' is required. WARNING: When using 'beeline' and specifying the connection url (-u) at the command line, be sure to quote the url. Non-Secure Environments Direct - Binary Transport Mode beeline -n hiveuser -p hivepswd -u "jdbc:hive2://test01.com:10010/<db>" Direct - HTTP Transport Mode beeline -n hiveuser -p hivepswd -u "jdbc:hive2://test01.com:10011/<db>;transportMode=http;httpPath=cliservice" ZooKeeper - Binary Transport Mode beeline -n hiveuser -p hivepswd -u "jdbc:hive2://test01.com:2181,test02.com:2181,test03.com:2181/<db>" ZooKeeper - Http Transport Mode beeline -n hiveuser -p hivepswd -u "jdbc:hive2://test01.com:2181,test02.com:2181,test03.com:2181/<db>;transportMode=http;httpPath=cliservice" Alternate Connectivity Thru Knox jdbc:hive2://<knox_host>:8443/;ssl=true;sslTrustStore=/var/lib/knox/data/security/keystores/gateway.jks;trustStorePassword=<password>?hive.server2.transport.mode=http;hive.server2.thrift.http.path=gateway/<CLUSTER>/hive Hope that helps ... View more

Re: Beeline connection throws error

by Master Mentor in Support Questions
‎08-08-2017 10:31 PM
‎08-08-2017 10:31 PM
@pv poreddy see my response ! ... View more

Re: Beeline connection throws error

by Master Mentor in Support Questions
‎08-08-2017 10:04 PM
‎08-08-2017 10:04 PM
@pv poreddy Here is a coonect string I just created in a non kerberized cluster. Assumptions hive host=bombay.new.com Default port = 10000 Hive database= hive Hive user =hive Hive password =hive Here you go [hive@bombo ~]$ beeline WARNING: Use "yarn jar" to launch YARN applications. Beeline version 1.2.1.2.3.6.0-3796 by Apache Hive beeline> !connect jdbc:hive2://bombay.new.com:10000/hive Connecting to jdbc:hive2://bombay.new.com:10000/hive Enter username for jdbc:hive2://bombo.new.com:10000/hive: hive Enter password for jdbc:hive2://bombo.new.com:10000/hive: **** Connected to: Apache Hive (version 1.2.1.2.3.6.0-3796) Driver: Hive JDBC (version 1.2.1.2.3.6.0-3796) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://bombo.uganda.com:10000/hive> show databases; +----------------+--+ | database_name | +----------------+--+ | default | +----------------+--+ 1 row selected (5.577 seconds) 0: jdbc:hive2://bombo.uganda.com:10000/hive> So you should have a successful login ... View more

Re: Beeline connection throws error

by Master Mentor in Support Questions
‎08-08-2017 08:41 PM
‎08-08-2017 08:41 PM
@pv poreddy The below output is for a keberized cluster you should replace the principal and REALM with the correct values derived from $ klist -ket /etc/security/keytabs/hive.keytab Keytab name: FILE:/etc/security/keytabs/hive.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 12/28/16 22:25:31 hive-{hostname}@REALM.COM (des3-cbc-sha1) 1 12/28/16 22:25:31 hive-{hostname}@REALM.COM (aes128-cts-hmac-sha1-96) 1 12/28/16 22:25:31 hive-{hostname}@REALM.COM (arcfour-hmac) 1 12/28/16 22:25:31 hive-{hostname}@REALM.COM (des-cbc-md5) 1 12/28/16 22:25:31 hive-{hostname}@REALM.COM (aes256-cts-hmac-sha1-96) To see if you grabbed a validate kerberos ticket [hive@{hostname} ~]$ klist Ticket cache: FILE:/tmp/krb5cc_507 Default principal: hive-{hostname}@REALM.COM Valid starting Expires Service principal 12/28/16 22:57:11 12/29/16 22:57:11 krbtgt/REALM.COM@REALM.COM renew until 12/28/16 22:57:11 12/28/16 22:57:11 12/29/16 22:57:11 HTTP/{hostname}.REALM.COM@renew until 12/28/16 22:57:11 12/28/16 22:57:11 12/29/16 22:57:11 HTTP/{hostname}.REALM.COM@REALM.COM renew until 12/28/16 22:57:11 Then use the above to connect and just PRESS enter at the username and password prompt Beeline version 1.2.1000.2.5.3.0-37 by Apache Hive beeline> ! connect jdbc:hive2://test.com:10000/;principal=hive/test.com@REALM.COM Connecting to jdbc:hive2://test.com:10000/;principal=hive/test.com@REALM.COM Enter username for jdbc:hive2://test.com:10000/;principal=hive/test.com@REALM.COM: Enter password for jdbc:hive2://test.com:10000/;principal=hive/test.com@REALM.COM: Connected to: Apache Hive (version 1.2.1000.2.5.3.0-37) Driver: Hive JDBC (version 1.2.1000.2.5.3.0-37) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://test.com:10000/> show databases; +----------------+--+ | database_name | +----------------+--+ | default | | moroto | | olum | +----------------+--+ 3 rows selected (2.863 seconds) You should now connect successfully otherwise in un kerberized environment [hive@host~]$ beeline .... ..... !connect jdbc:hive2://test.com:10000/<db> Cheers ... View more

Re: how to update ambari cluster with blueprint.js...

by Master Mentor in Support Questions
‎08-08-2017 04:01 PM
‎08-08-2017 04:01 PM
@uri ben-ari Setup YARN HA for example ... View more

Re: how to update ambari cluster with blueprint.js...

by Master Mentor in Support Questions
‎08-08-2017 03:30 PM
‎08-08-2017 03:30 PM
@uri ben-ari If I understand you well,you want to make a change a cluster and replicate that to all the other lab clusters ? Right I would think that is doable if you can script you curl do loop through the clusters you will have to deal with inter cluster authentication and authorization. In a secure setup kerberized cluster that can be handled through KNOX. Personally I have not done that, having said that blue prints resolve the tedious administration of initial cluster setup. Below is a link that could inspire you. Dynamically Add Hosts to a Cluster with Blueprints You had earlier asked an initial question which I think I replied to could you reward the best answer for your previous question as this is a new question. This ensure people can take time to respond and feel gratified if their answer help a member. ... View more

Re: ambari + Update a blueprint

by Master Mentor in Support Questions
‎08-08-2017 12:54 PM
1 Kudo
‎08-08-2017 12:54 PM
1 Kudo
@uri ben-ari In the below example I am removing and re-installing RANGER Stop RANGER component curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop All Components"},"Body":{"ServiceComponentInfo":{"state":"INSTALLED"}}}' http://{ambari_host}:8080/api/v1/clusters/{cluster_name}/services/OOZIE/components/RANGER Delete RANGER component curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://{ambari_host}:8080/api/v1/clusters/{cluster_name}/services/RANGER Install RANGER component curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop Component"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://{ambari_host}:8080/api/v1/clusters/{cluster_name}/hosts/HOSTNAME/host_components/RANGER Hope that helps ... View more

Re: ambari + Update a blueprint

by Master Mentor in Support Questions
‎08-08-2017 12:10 PM
‎08-08-2017 12:10 PM
@uri ben-ari Usually when you add or delete components in a cluster its advisable to download the new blueprint, to enable you reproduce the cluster like from UAT/DEV to PROD. The other was round I have never tried though you can add components to an existing cluster using a blueprint. see Look at this ... View more
Contact Me
Online
Offline
Last Visited
‎06-05-2025 02:03 PM
Community Statistics
Member Since ‎01-19-2017 04:35 AM
Last Visited ‎06-05-2025 02:03 PM
Posts 3,676
Kudos received 627