Member since
06-06-2019
81
Posts
58
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1420 | 10-04-2019 07:24 AM | |
1751 | 12-12-2016 03:07 PM | |
3856 | 12-07-2016 03:41 PM | |
3966 | 07-12-2016 02:49 PM | |
1375 | 03-04-2016 02:35 PM |
10-04-2019
07:24 AM
2 Kudos
The Ambari installation is probably attempting to update some of the Ubuntu packages due to the installation dependencies of Ambari and the HDP components. We usually have a local repo for the O/S available installed when a fully disconnected HDP installation is performed.
... View more
04-12-2017
01:24 AM
4 Kudos
This article describes the setup of two separate KDCs in a Master/Slave configuration. This setup will allow two clusters to share a single Kerberos realm, which allows the principals to be recognized between clusters. A use case for this configuration is when a Disaster Recovery cluster is used as a warm standby. The high level information for the article was found at https://web.mit.edu/kerberos/krb5-1.13/doc/admin/install_kdc.html, while the details were worked out through sweat and tears. Execute the following command to install the Master and Slave KDC if the KDC is not already installed: yum install krb5-server The following defines the KDC configuration for both clusters. This file, /etc/krb5.conf, must be copied to each node in the cluster. [libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = CUSTOMER.HDP
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
udp_preference_limit=1
[domain_realm]
customer.com = CUSTOMER.HDP
.customer.com = CUSTOMER.HDP
[logging]
default = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
kdc = FILE:/var/log/krb5kdc.log
[realms]
CUSTOMER.HDP = {
admin_server = master-kdc.customer.com
kdc = master-kdc.customer.com
kdc = slave-kdc.customer.com
} Contents of /var/kerberos/krb5kdc/kadm5.acl: */admin@CUSTOMER.HDP * Contents of the /var/kerberos/krb5kdc/kdc.conf: [kdcdefaults]
kdc_ports = 88,750
kdc_tcp_ports = 88,750
[realms]
CUSTOMER.HDP = {
kadmind_port = 749
max_life = 12h 0m 0s
max_renewable_life = 7d 0h 0m 0s
master_key_type = aes256-cts
supported_enctypes = aes256-cts aes128-cts des-hmac-sha1 des-cbc-md5 arcfour-hmac des-cbc-md5
} Contents of /var/kerberos/krb5kdc/kpropd.acl: host/master-kdc.customer.com@CUSTOMER.HDP
host/slave-kdc.customer.com@CUSTOMER.HDP Now start the KDC and kadmin processes on the Master KDC only: shell% systemctl enable krb5kdc
shell% systemctl start krb5kdc
shell% systemctl enable kadmin
shell% systemctl start kadmin The KDC database is then initialized with the following command, executed from the Master KDC: shell% kdb5_util create -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'CUSTOMER.HDP',
master key name 'K/M@CUSTOMER.HDP'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key: <db_password>
Re-enter KDC database master key to verify: <db_password> An administrator must be created to manage the Kerberos realm. The following command is used to create the administration principal from the Master KDC: shell% kadmin.local -q "addprinc admin/admin"
Authenticating as principal root/admin@CUSTOMER.HDP with password.
WARNING: no policy specified for admin/admin@CUSTOMER.HDP; defaulting to no policy
Enter password for principal "admin/admin@CUSTOMER.HDP": <admin_password>
Re-enter password for principal "admin/admin@CUSTOMER.HDP": <admin_password>
Principal "admin/admin@CUSTOMER.HDP" created.
Host keytabs must now be created for the SLAVE KDC. Execute the following commands from the Master KDC: shell% kadmin
kadmin: addprinc -randkey host/master-kdc.customer.com
kadmin: addprinc -randkey host/slave-kdc.customer.com Extract the host key for the Slave KDC and store it on the hosts keytab file, /etc/krb5.keytab.slave: kadmin: ktadd –k /etc/krb5.keytab.slave host/slave-kdc.customer.com Copy /etc/krb5.keytab.slave to slave-kdc.customer.com and rename the file to /etc/krb5.keytab Update /etc/services on each KDC host, if not present: krb5_prop 754/tcp # Kerberos slave propagation Install xinetd on the hosts of the Master and Slave KDC, if not already installed, to enable kpropd to execute: yum install xinetd Create the configuration for kpropd on both the Master and Slave KDC hosts: Create /etc/xinetd.d/krb5_prop with the following contents. Create /etc/xinetd.d/krb5_prop with the following contents.
service krb_prop
{
disable = no
socket_type = stream
protocol = tcp
user = root
wait = no
server = /usr/sbin/kpropd
} Configure xinetd to run as a persistent service on both the Master and Slave KDC hosts: systemctl enable xinetd.service
systemctl start xinetd.service Copy the following files from the Master KDC host to the Slave KDC host: /etc/krb5.conf
/var/kerberos/krb5kdc/kadm5.acl
/var/kerberos/krb5kdc/kdc.conf
/var/kerberos/krb5kdc/kpropd.acl
/var/kerberos/krb5kdc/.k5.CUSTOMER.HDP Perform the initial KDC database propagation to the Slave KDC: shell% kdb5_util dump /usr/local/var/krb5kdc/slave_datatrans
shell% kprop -f /usr/local/var/krb5kdc/slave_datatrans slave-kdc.customer.com The Slave KDC may be started at this time: shell% systemctl enable krb5kdc
shell% systemctl start krb5kdc Script to propagate the updates from the Master KDC to the Slave KDC. Create a cron job, or the like, to run this script on a frequent basis. #!/bin/sh
#/var/kerberos/kdc-slave-propogate.sh
kdclist = "slave-kdc.customer.com"
/sbin/kdb5_util dump /usr/local/var/krb5kdc/slave_datatrans
for kdc in $kdclist
do
/sbin/kprop -f /usr/local/var/krb5kdc/slave_datatrans $kdc
done
... View more
12-12-2016
03:07 PM
1 Kudo
@Avijeet Dash HDP Search is the basic Solr package with a tested integration to HDP. Lucidworks, who are the primary contributors to Solr, package the product. The default storage option for Solr uses the server's local disk for storage. You can see that this would cause competition for disk resources if the Solr installation is co-located with an HDP datanode. If you go with the SolrCloud option you can configure HDFS as your Solr data repository. Aside from fault tolerance and high availability, this gives you the option of adding more datanodes to your HDP cluster to handle the expected increase in disk use by SolrCloud.
... View more
12-07-2016
04:43 PM
You are welcome Mahendra. I think you will have to push the complex query to the database and place those results into a different table and then perform the sqoop command on that table. Best of luck.
... View more
12-07-2016
03:41 PM
@Mahendra Dahiya The sqoop import --query option is intended to process a single statement and there are warnings about using complex queries. The facility of using free-form query in the current version of Sqoop
is limited to simple queries where there are no ambiguous projections and
no OR conditions in the WHERE clause. Use of complex queries such as
queries that have sub-queries or joins leading to ambiguous projections can
lead to unexpected results. A few more details are at this URL: https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_free_form_query_imports
... View more
09-16-2016
08:13 PM
1 Kudo
@Eric Brosch Can you downgrade to the earlier version with this command? yum downgrade cups-libs-1.4.2-50.el6_4.5.i686
... View more
07-12-2016
02:49 PM
1 Kudo
Hey @Kit Menke Have you tried turning off UDP? If not, add the property and value: udp_preference_limit = 1 to your /etc/krb5.conf and see if that solves your problem. If you have Ambari managing the krb5.conf, go to Kerberos -> Advanced krb5.conf -> krb5-conf template and add the property to the [libdefaults] section. Save the changes and let Ambari push out the changes to the hosts. If you are managing the /etc/krb5.conf file, then you will have to add the property and push out changes yourself.
... View more
06-28-2016
07:45 PM
Did you finalize the upgrade from 2.2.6.0-2800 to 2.4.0.0-169? You can look at directory on your namenode where your fsimage file and the edits are stored and see if you are keeping info for current and the old version. The command to finalize the upgrade is: hdfs dfsadmin -finalizeUpgrade Hope this helps.
... View more
06-13-2016
02:19 PM
1 Kudo
@Randy Gelhausen S3 does not have a directory hierarchy per se but S3 does allow the Object Keys to contain the "/" character. You are dealing with a key-value store and one object can have a key of /my/fake/directory/a/file and another can have a key value of /my/fake/directory/b/file. The objects are named similarly, and most tools that speak S3 will display the object as if they were files in a directory hierarchy, but there is no directory structure behind the objects. That is the key takeaway when dealing with S3. When you store, or retrieve, an object with S3 you have to reference the entire key for the object and the bucket that contains the key. The paradigm of directory and file are just an illusion. Use the Object Key in the method call as @jfrazee said and you should be good to go.
... View more
06-03-2016
01:54 PM
@c pat You need to first find out why the connection to the Namenode is being refused. Is the Namenode process up? You can do a quick check with ps aux | grep -i namenode If the Namenode process is up, then look at the logs in /var/log/hadoop/hdfs. You will want to look at the file with that looks like hadoop-hdfs-namenode-*.log. This should help you narrow down the cause a bit.
... View more