Member since 
    
	
		
		
		09-24-2015
	
	
	
	
	
	
	
	
	
	
	
	
	
	
			
      
                178
            
            
                Posts
            
        
                113
            
            
                Kudos Received
            
        
                28
            
            
                Solutions
            
        My Accepted Solutions
| Title | Views | Posted | 
|---|---|---|
| 4647 | 05-25-2016 02:39 AM | |
| 4590 | 05-03-2016 01:27 PM | |
| 1195 | 04-26-2016 07:59 PM | |
| 16767 | 03-24-2016 04:10 PM | |
| 3136 | 02-02-2016 11:50 PM | 
			
    
	
		
		
		10-23-2015
	
		
		06:16 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							 Ulimit does not depend on the size of the cluster but more on the individual node, workloads and user concurrency. I set these values as best practice, which BTW is a large enough value and is probably never reached.   * - nofile 32768
* - nproc 65536  Ambari actually changes the limit of the core files created as part of the commands executed like in this case -      
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		10-23-2015
	
		
		03:07 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 What precautions & extra configurations, if any, are needed when adding worker nodes with different capacity to a cluster? My understanding is that YARN will be able to just manage the nodes without anything special.   For e.g - Any issues with adding 3 nodes with following config to an existing POC cluster that has similar nodes 8 cores, 32 Gigs, 3 TB DAS for data-   node1 - 8 cores, 64GB RAM, NO STORAGE 
node2 - 8 cores, 64GB RAM, 2 TB
node3 - 8 cores, 64GB RAM, 2TB  Also, how do you configure YARN to utilize different amount of memory on these heterogeneous boxes?  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
- 
						
							
		
			Apache YARN
			
    
	
		
		
		10-22-2015
	
		
		09:16 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Vel, thanks for the answer. Just because Neeraj has the screen snapshots (which will make it clearer for the next guys), I will select his answer as correct but I know you are the source for his answer too 🙂  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		10-22-2015
	
		
		08:24 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		3 Kudos
		
	
				
		
	
		
					
							 Changing Ranger UI password causes Namenode to stop because it seems that the password needs to be updated in HDFS plugin within HDFS config. However, I am not sure which config needs to be updated in HDFS plugin. Please advise.  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
- 
						
							
		
			Apache Ranger
			
    
	
		
		
		10-22-2015
	
		
		08:20 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		2 Kudos
		
	
				
		
	
		
					
							 In HA environment, we can access HDFS by referring the active namenode directory but I am interested in finding if there is a way to access HDFS using nameservice id such that if and when the HDFS fails over to the passive namenode, then the client can just continue to use HDFS without manually changing the configuration. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
- 
						
							
		
			Apache Hadoop
			
    
	
		
		
		10-22-2015
	
		
		07:33 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 When a certain amount of memory is given to ResourceManager (Memory allocated for all YARN containers on a node), is it immediately blocked or gradually/progressively used on as-needed basis until that capacity is reached?  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
- 
						
							
		
			Apache YARN
			
    
	
		
		
		10-22-2015
	
		
		07:18 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Knox provides solution for perimeter security and like any security component, (encryption or authorization) does add overhead to the processing time. With that said, in order to determine whether a security tool is needed or not, performance is not considered as a deciding factor. We can load balance the traffic across multiple knox instances to distributed the load to avoid too much degradation in performance. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		10-21-2015
	
		
		04:17 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Does that mean Ranger itself cannot be HA but it can work with HA enabled services? And if we the repo has to be changed manually, what is it that Ranger is doing for hA?  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		10-21-2015
	
		
		02:27 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							 +1 on Nifi. If customer does not want to go that route (which they should because it is probably the most elegant solution) the other option would be to export the data from oracle using export utility to the local file system. Configure Flume agent to listen to the export directory and use Kafka Sink to place the data on Kafka. Depending on the downstream processing, Flume can chunk the data into appropriate sizes (1 row, 10 rows, n rows) 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		10-21-2015
	
		
		02:18 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		4 Kudos
		
	
				
		
	
		
					
							 Hi Wes - It is possible. I used the following knox config in past to do it.   For 2 - You would need a different knox topology.       <provider>
            <role>authentication</role>
            <name>ShiroProvider</name>
            <enabled>true</enabled>
            <param>
                <!--
                session timeout in minutes,  this is really idle timeout,
                defaults to 30mins, if the property value is not defined,,
                current client authentication would expire if client idles contiuosly for more than this value
                -->
                <name>sessionTimeout</name>
                <value>30</value>
            </param>            
            <param>
                <name>main.ldapRealm</name>
                <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
            </param>
            <param>
                <name>main.ldapContextFactory</name>
                <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory</name>
                <value>$ldapContextFactory</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory.url</name>
                <!-- ADJUST host, port for your AD setup-->
                <value>ldap://ad.client.com:389</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
                <value>simple</value>
            </param>
            <!-- Param below is ignored-->
            <param>
                <name>main.ldapRealm.userDnTemplate</name>
                <value>cn={0},ou=hadoop,ou=personal,ou=accounts,dc=ad,dc=client,dc=com</value>
            </param>
            <!-- Param above is ignored-->            
			<param>
				<name>main.ldapRealm.userSearchAttributeName</name>
				<value>sAMAccountName</value>
			</param>
			<param>
				<name>main.ldapRealm.userObjectClass</name>
				<value>person</value>
			</param>
			<param>
				<name>main.ldapRealm.contextFactory.systemUsername</name>
				<value>cn=hadoop_knox_id,ou=process,ou=accounts,dc=ad,dc=client,dc=com</value>
			</param>
			<param>
				<name>main.ldapRealm.contextFactory.systemPassword</name>
				<value>passwd_4_hadoop_knox_id</value>
			</param>
			
			<!-- search base used to search for user bind DN and groups -->
			<param>
				<name>main.ldapRealm.searchBase</name>
				<value>ou=personal,ou=accounts,dc=ad,dc=client,dc=com</value>
			</param>
			<!-- search base used to search for user bind DN.
				 Defaults to the value of main.ldapRealm.searchBase. 
				 If main.ldapRealm.userSearchAttributeName is defined, 
				 vlaue for main.ldapRealm.searchBase  or main.ldapRealm.userSearchBase 
				 should be defined -->
			<param>
				<name>main.ldapRealm.userSearchBase</name>
				<value>ou=personal,ou=accounts,dc=ad,dc=client,dc=com</value>
			</param>			
            <param>
                <name>urls./**</name>
                <value>authcBasic</value>
            </param>
        </provider>
        <provider>
            <role>identity-assertion</role>
            <name>Default</name>
            <enabled>true</enabled>
        </provider>
 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		 
         
					
				













