Member since 
    
	
		
		
		06-06-2019
	
	
	
	
	
	
	
	
	
	
	
	
	
	
			
      
                81
            
            
                Posts
            
        
                58
            
            
                Kudos Received
            
        
                11
            
            
                Solutions
            
        My Accepted Solutions
| Title | Views | Posted | 
|---|---|---|
| 1998 | 10-04-2019 07:24 AM | |
| 2397 | 12-12-2016 03:07 PM | |
| 4639 | 12-07-2016 03:41 PM | |
| 5117 | 07-12-2016 02:49 PM | |
| 1924 | 03-04-2016 02:35 PM | 
			
    
	
		
		
		10-04-2019
	
		
		07:24 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		2 Kudos
		
	
				
		
	
		
					
							 The Ambari installation is probably attempting to update some of the Ubuntu packages due to the installation dependencies of Ambari and the HDP components. We usually have a local repo for the O/S available installed when a fully disconnected HDP installation is performed.  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		04-12-2017
	
		
		01:24 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		4 Kudos
		
	
				
		
	
		
					
							 This article describes the setup of two separate KDCs in a Master/Slave configuration. This setup will allow two clusters to share a single Kerberos realm, which allows the principals to be recognized between clusters. A use case for this configuration is when a Disaster Recovery cluster is used as a warm standby. The high level information for the article was found at https://web.mit.edu/kerberos/krb5-1.13/doc/admin/install_kdc.html, while the details were worked out through sweat and tears.  Execute the following command to install the Master and Slave KDC if the KDC is not already installed:   yum install krb5-server   The following defines the KDC configuration for both clusters. This file, /etc/krb5.conf, must be copied to each node in the cluster.  [libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = CUSTOMER.HDP
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  udp_preference_limit=1
[domain_realm]
  customer.com = CUSTOMER.HDP
  .customer.com = CUSTOMER.HDP
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  CUSTOMER.HDP = {
    admin_server = master-kdc.customer.com
    kdc = master-kdc.customer.com
    kdc = slave-kdc.customer.com
  }  Contents of /var/kerberos/krb5kdc/kadm5.acl:  */admin@CUSTOMER.HDP *  Contents of the /var/kerberos/krb5kdc/kdc.conf:  [kdcdefaults]
 kdc_ports = 88,750
 kdc_tcp_ports = 88,750
[realms]
    CUSTOMER.HDP = {
        kadmind_port = 749
        max_life = 12h 0m 0s
        max_renewable_life = 7d 0h 0m 0s
        master_key_type = aes256-cts
       supported_enctypes = aes256-cts aes128-cts des-hmac-sha1 des-cbc-md5 arcfour-hmac des-cbc-md5
    }  Contents of /var/kerberos/krb5kdc/kpropd.acl:  host/master-kdc.customer.com@CUSTOMER.HDP
host/slave-kdc.customer.com@CUSTOMER.HDP  Now start the KDC and kadmin processes on the Master KDC only:  shell% systemctl enable krb5kdc 
shell% systemctl start krb5kdc 
shell% systemctl enable kadmin 
shell% systemctl start kadmin    The KDC database is then initialized with the following command, executed from the Master KDC:  shell% kdb5_util create -s 
Loading random data 
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'CUSTOMER.HDP', 
master key name 'K/M@CUSTOMER.HDP' 
You will be prompted for the database Master Password. 
It is important that you NOT FORGET this password. 
Enter KDC database master key: <db_password>
Re-enter KDC database master key to verify: <db_password>  An administrator must be created to manage the Kerberos realm.  The following command is used to create the administration principal from the Master KDC:  shell% kadmin.local -q "addprinc admin/admin" 
Authenticating as principal root/admin@CUSTOMER.HDP with password. 
WARNING: no policy specified for admin/admin@CUSTOMER.HDP; defaulting to no policy 
Enter password for principal "admin/admin@CUSTOMER.HDP": <admin_password>
Re-enter password for principal "admin/admin@CUSTOMER.HDP": <admin_password>
Principal "admin/admin@CUSTOMER.HDP" created. 
  Host keytabs must now be created for the SLAVE KDC. Execute the following commands from the Master KDC:  shell% kadmin
kadmin: addprinc -randkey host/master-kdc.customer.com
kadmin: addprinc -randkey host/slave-kdc.customer.com  Extract the host key for the Slave KDC and store it on the hosts keytab file, /etc/krb5.keytab.slave:  kadmin: ktadd –k /etc/krb5.keytab.slave host/slave-kdc.customer.com  Copy /etc/krb5.keytab.slave to slave-kdc.customer.com and rename the file to /etc/krb5.keytab  Update /etc/services on each KDC host, if not present:  krb5_prop       754/tcp               # Kerberos slave propagation  Install xinetd on the hosts of the Master and Slave KDC, if not already installed, to enable kpropd to execute:  yum install xinetd  Create the configuration for kpropd on both the Master and Slave KDC hosts:  Create /etc/xinetd.d/krb5_prop with the following contents.  Create /etc/xinetd.d/krb5_prop with the following contents.
service krb_prop
{
        disable         = no
        socket_type     = stream
        protocol        = tcp
        user            = root
        wait            = no
        server          = /usr/sbin/kpropd
}  Configure xinetd to run as a persistent service on both the Master and Slave KDC hosts:  systemctl enable xinetd.service
systemctl start xinetd.service  Copy the following files from the Master KDC host to the Slave KDC host:  /etc/krb5.conf 
/var/kerberos/krb5kdc/kadm5.acl 
/var/kerberos/krb5kdc/kdc.conf
/var/kerberos/krb5kdc/kpropd.acl
/var/kerberos/krb5kdc/.k5.CUSTOMER.HDP  Perform the initial KDC database propagation to the Slave KDC:  shell% kdb5_util dump /usr/local/var/krb5kdc/slave_datatrans
shell% kprop -f /usr/local/var/krb5kdc/slave_datatrans slave-kdc.customer.com  The Slave KDC may be started at this time:   shell% systemctl enable krb5kdc 
shell% systemctl start krb5kdc   Script to propagate the updates from the Master KDC to the Slave KDC. Create a cron job, or the like, to run this script on a frequent basis.  #!/bin/sh
#/var/kerberos/kdc-slave-propogate.sh
kdclist = "slave-kdc.customer.com"
/sbin/kdb5_util dump /usr/local/var/krb5kdc/slave_datatrans
for kdc in $kdclist
do
    /sbin/kprop -f /usr/local/var/krb5kdc/slave_datatrans $kdc
done 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		12-12-2016
	
		
		03:07 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							 @Avijeet Dash   HDP Search is the basic Solr package with a tested integration to HDP.  Lucidworks, who are the primary contributors to Solr, package the product.   The default storage option for Solr uses the server's local disk for storage.  You can see that this would cause competition for disk resources if the Solr installation is co-located with an HDP datanode.  If you go with the SolrCloud option you can configure HDFS as your Solr data repository. Aside from fault tolerance and high availability, this gives you the option of adding more datanodes to your HDP cluster to handle the expected increase in disk use by SolrCloud. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		12-07-2016
	
		
		04:43 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 You are welcome Mahendra. I think you will have to push the complex query to the database and place those results into a different table and then perform the sqoop command on that table.  Best of luck. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		12-07-2016
	
		
		03:41 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @Mahendra Dahiya  The sqoop import --query option is intended to process a single statement and there are warnings about using complex queries.      The facility of using free-form query in the current version of Sqoop
is limited to simple queries where there are no ambiguous projections and
no OR conditions in the WHERE clause. Use of complex queries such as
queries that have sub-queries or joins leading to ambiguous projections can
lead to unexpected results.  A few more details are at this URL:   https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_free_form_query_imports 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		09-16-2016
	
		
		08:13 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							@Eric Brosch Can you downgrade to the earlier version with this command?   yum downgrade cups-libs-1.4.2-50.el6_4.5.i686  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		07-12-2016
	
		
		02:49 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							 Hey @Kit Menke  Have you tried turning off UDP? If not, add the property and value:  udp_preference_limit = 1   to your /etc/krb5.conf  and see if that solves your problem.    If you have Ambari managing the krb5.conf, go to Kerberos -> Advanced krb5.conf -> krb5-conf template and add the property to the [libdefaults] section. Save the changes and let Ambari push out the changes to the hosts.  If you are managing the /etc/krb5.conf file, then you will have to add the property and push out changes yourself. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-28-2016
	
		
		07:45 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Did you finalize the upgrade from 2.2.6.0-2800 to 2.4.0.0-169?  You can look at directory on your namenode where your fsimage file and the edits are stored and see if you are keeping info for current and the old version.  The command to finalize the upgrade is: hdfs dfsadmin -finalizeUpgrade  Hope this helps. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-13-2016
	
		
		02:19 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							  @Randy Gelhausen   S3 does not have a directory hierarchy per se but S3 does allow the Object Keys to contain the "/" character. You are dealing with a key-value store and one object can have a key of /my/fake/directory/a/file and another can have a key value of /my/fake/directory/b/file.  The objects are named similarly, and most tools that speak S3 will display the object as if they were files in a directory hierarchy, but there is no directory structure behind the objects. That is the key takeaway when dealing with S3.  When you store, or retrieve, an object with S3 you have to reference the entire key for the object and the bucket that contains the key.  The paradigm of directory and file are just an illusion.  Use the Object Key in the method call as @jfrazee said and you should be good to go. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-03-2016
	
		
		01:54 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @c pat   You need to first find out why the connection to the Namenode is being refused.  Is the Namenode process up?  You can do a quick check with ps aux | grep -i namenode  If the Namenode process is up, then look at the logs in /var/log/hadoop/hdfs.  You will want to look at the file with that looks like hadoop-hdfs-namenode-*.log.  This should help you narrow down the cause a bit. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		 
        













