Member since 
    
	
		
		
		01-03-2017
	
	
	
	
	
	
	
	
	
	
	
	
	
	
			
      
                181
            
            
                Posts
            
        
                44
            
            
                Kudos Received
            
        
                24
            
            
                Solutions
            
        My Accepted Solutions
| Title | Views | Posted | 
|---|---|---|
| 2269 | 12-02-2018 11:49 PM | |
| 3124 | 04-13-2018 06:41 AM | |
| 2679 | 04-06-2018 01:52 AM | |
| 2967 | 01-07-2018 09:04 PM | |
| 6510 | 12-20-2017 10:58 PM | 
			
    
	
		
		
		03-07-2018
	
		
		02:04 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hello Matt,   Sorry for the delay, but I did not see your answer before.   I needed to support server and client Authentification.   I ended up using a different keystore for storing the various key I used.  Best regards  Abdou 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		12-21-2017
	
		
		12:24 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							 To secure
the Spark Thrift server first we need to change the mode from binary to http
then secure the channel with the certificates.  Login to Ambari-> Spark(2)-> Configs -> Custom
spark-hive-site-override:  Set the following parameters :  hive.server2.transport.mode : http 
hive.server2.thrift.http.port : 10015 /  10016 ( in case of spark 2)
hive.server2.http.endpoint : cliservice  #Enabling the SSL mode  hive.server2.use.SSL : true 
hive.server2.keystore.path : </path/to/your/keystore/jks>
hive.server2.keystore.password : <keystorepassword>  in case of
server certs are not available process to create self-signed certs (from Hive
Wiki page)  Setting
up SSL with self-signed certificates  Use the
following steps to create and verify self-signed SSL certificates for use with
HiveServer2:  
 
 Create the self-signed
     certificate and add it to a keystore file using: keytool -genkey -alias example.com
     -keyalg RSA -keystore keystore.jks -keysize 2048 Ensure the name used in the
     self signed certificate matches the hostname where Thrift server will run.  
 List the keystore entries to
     verify that the certificate was added. Note that a keystore can contain
     multiple such certificates: keytool
     -list -keystore keystore.jks  
 Export this certificate from
     keystore.jks to a certificate file: keytool -export
     -alias example.com -file example.com.crt -keystore
     keystore.jks  
 Add this certificate to the
     client's truststore to establish trust: keytool -import -trustcacerts -alias example.com -file example.com.crt
     -keystore truststore.jks  
 Verify that the certificate
     exists in truststore.jks: keytool
     -list -keystore truststore.jks  
 Then start Spark Thrift server,
     use spark-sql form spark bin or try to connect with beeline using: jdbc:hive2://<host>:<port>/<database>;ssl=true;sslTrustStore=<path-to-truststore>;trustStorePassword=<truststore-password>  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
	
					
			
		
	
	
	
	
				
		
	
	
			
    
	
		
		
		12-21-2017
	
		
		08:52 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @bkosaraju. Your explanations makes sense. Thanks for clarifying! My understanding about threshold was different.   Robert 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		12-22-2017
	
		
		02:54 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @bkosaraju  Thanks a lot, the splitting part works.... but i am still getting only the first match ... how do i get all matches? 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		12-12-2017
	
		
		07:23 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @bkosaraju - .. i re-checked this & the issue seems to be when i include the column - deviceid bigint - in the query  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		12-06-2017
	
		
		06:29 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Thanks @bkosaraju  Strict Host Key Checking is set to False.  If with target hosts, you mean, the SFTP server (our destination) then it is not in the list of /.ssh/known_hosts)  Just tried manual SFTP again and this time received, Connection Reset by Peer. Checking SFTP side in case our Node IP is blacklisted (possibly due to Max Failure condition.)  Update -- 12-07-2017 -- The IP for the NIFI Node was blacklisted on the SFTP Side. We've whitelisted the IP and the processor is now functional. Thanks. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		10-25-2017
	
		
		10:13 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 We have resolved the issue. As usual in hindsight it seems obvious. One of the processes, a transformxml, had the thread count set at 20. This seemed fine as most files going through were about 3m each. Our Java heap was set at 64g. However new files were introduced, that were in the 3g range. Simple math, 3 x 20=60g. used up for that 1 process. Since there was usually about 200k files flowing through at any one time, memory quickly got exhausted. We put a route attribute, to route these large files to a process that only has a couple of threads, while the smaller files could still filter through the process with 20 threads. Thanks for all the help suggestions. @Hans Feldmann 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		10-14-2017
	
		
		08:19 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							@Mamta Chawla You cannot use the MIT Kerberos kadmin to create accounts in an Active Directory. That tool is only for use with the MIT KDC.  To create accounts in an Active Directory, you will need to use Active Directory-specific tools. However, you can use a similar process that Ambari uses to create accounts in an Active Directory and then manually build the keytab files.  This requires the use of the OpenLDAP ldapadd and ldapmodify tools as well as the ktutil command.   See https://community.hortonworks.com/articles/82544/how-to-create-ad-principal-accounts-using-openldap.html.  On top of this, you will need to make sure your krb5.conf file is correct in order for you to test it out. The krb5.conf file is not needed to create the AD account if you are using LDAP to do the work.  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		10-14-2017
	
		
		09:50 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hi @bkosaraju,  Thanks a lot for help.  I have checked points
you suggested. Unfortunately it didn't help as number of driver cores is still reported to be 0. Is my expectation right that there should be exact number of
cores as provided in driver.cores property?    I am sure that dynamic allocation is on, as I can see
executors removed during application runtime.  I attached a list of spark properties for ma application taken from Spark UI.   spark-properties.txt   
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		10-11-2017
	
		
		11:39 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @Sumit Sharma,   Use UpdateAttribute processor before PutFile processor with below configurations.  Add new property to the processor by clicking + sign   filename as ${UUID()}  So this will replace the filename of the flowfile to UUID which is unique value all the time, it won't replace the file in your directory.      
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		- « Previous
 - 
						
- 1
 - 2
 
 - Next »