Member since
01-03-2017
181
Posts
44
Kudos Received
24
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2233 | 12-02-2018 11:49 PM | |
3065 | 04-13-2018 06:41 AM | |
2638 | 04-06-2018 01:52 AM | |
2924 | 01-07-2018 09:04 PM | |
6444 | 12-20-2017 10:58 PM |
03-07-2018
02:04 PM
Hello Matt, Sorry for the delay, but I did not see your answer before. I needed to support server and client Authentification. I ended up using a different keystore for storing the various key I used. Best regards Abdou
... View more
12-21-2017
12:24 AM
1 Kudo
To secure
the Spark Thrift server first we need to change the mode from binary to http
then secure the channel with the certificates. Login to Ambari-> Spark(2)-> Configs -> Custom
spark-hive-site-override: Set the following parameters : hive.server2.transport.mode : http
hive.server2.thrift.http.port : 10015 / 10016 ( in case of spark 2)
hive.server2.http.endpoint : cliservice #Enabling the SSL mode hive.server2.use.SSL : true
hive.server2.keystore.path : </path/to/your/keystore/jks>
hive.server2.keystore.password : <keystorepassword> in case of
server certs are not available process to create self-signed certs (from Hive
Wiki page) Setting
up SSL with self-signed certificates Use the
following steps to create and verify self-signed SSL certificates for use with
HiveServer2:
Create the self-signed
certificate and add it to a keystore file using: keytool -genkey -alias example.com
-keyalg RSA -keystore keystore.jks -keysize 2048 Ensure the name used in the
self signed certificate matches the hostname where Thrift server will run.
List the keystore entries to
verify that the certificate was added. Note that a keystore can contain
multiple such certificates: keytool
-list -keystore keystore.jks
Export this certificate from
keystore.jks to a certificate file: keytool -export
-alias example.com -file example.com.crt -keystore
keystore.jks
Add this certificate to the
client's truststore to establish trust: keytool -import -trustcacerts -alias example.com -file example.com.crt
-keystore truststore.jks
Verify that the certificate
exists in truststore.jks: keytool
-list -keystore truststore.jks
Then start Spark Thrift server,
use spark-sql form spark bin or try to connect with beeline using: jdbc:hive2://<host>:<port>/<database>;ssl=true;sslTrustStore=<path-to-truststore>;trustStorePassword=<truststore-password>
... View more
Labels:
12-21-2017
08:52 AM
@bkosaraju. Your explanations makes sense. Thanks for clarifying! My understanding about threshold was different. Robert
... View more
12-22-2017
02:54 AM
@bkosaraju Thanks a lot, the splitting part works.... but i am still getting only the first match ... how do i get all matches?
... View more
12-12-2017
07:23 AM
@bkosaraju - .. i re-checked this & the issue seems to be when i include the column - deviceid bigint - in the query
... View more
12-06-2017
06:29 PM
Thanks @bkosaraju Strict Host Key Checking is set to False. If with target hosts, you mean, the SFTP server (our destination) then it is not in the list of /.ssh/known_hosts) Just tried manual SFTP again and this time received, Connection Reset by Peer. Checking SFTP side in case our Node IP is blacklisted (possibly due to Max Failure condition.) Update -- 12-07-2017 -- The IP for the NIFI Node was blacklisted on the SFTP Side. We've whitelisted the IP and the processor is now functional. Thanks.
... View more
10-25-2017
10:13 PM
We have resolved the issue. As usual in hindsight it seems obvious. One of the processes, a transformxml, had the thread count set at 20. This seemed fine as most files going through were about 3m each. Our Java heap was set at 64g. However new files were introduced, that were in the 3g range. Simple math, 3 x 20=60g. used up for that 1 process. Since there was usually about 200k files flowing through at any one time, memory quickly got exhausted. We put a route attribute, to route these large files to a process that only has a couple of threads, while the smaller files could still filter through the process with 20 threads. Thanks for all the help suggestions. @Hans Feldmann
... View more
10-14-2017
08:19 PM
1 Kudo
@Mamta Chawla You cannot use the MIT Kerberos kadmin to create accounts in an Active Directory. That tool is only for use with the MIT KDC. To create accounts in an Active Directory, you will need to use Active Directory-specific tools. However, you can use a similar process that Ambari uses to create accounts in an Active Directory and then manually build the keytab files. This requires the use of the OpenLDAP ldapadd and ldapmodify tools as well as the ktutil command. See https://community.hortonworks.com/articles/82544/how-to-create-ad-principal-accounts-using-openldap.html. On top of this, you will need to make sure your krb5.conf file is correct in order for you to test it out. The krb5.conf file is not needed to create the AD account if you are using LDAP to do the work.
... View more
10-14-2017
09:50 AM
Hi @bkosaraju, Thanks a lot for help. I have checked points
you suggested. Unfortunately it didn't help as number of driver cores is still reported to be 0. Is my expectation right that there should be exact number of
cores as provided in driver.cores property? I am sure that dynamic allocation is on, as I can see
executors removed during application runtime. I attached a list of spark properties for ma application taken from Spark UI. spark-properties.txt
... View more
10-11-2017
11:39 PM
@Sumit Sharma, Use UpdateAttribute processor before PutFile processor with below configurations. Add new property to the processor by clicking + sign filename as ${UUID()} So this will replace the filename of the flowfile to UUID which is unique value all the time, it won't replace the file in your directory.
... View more
- « Previous
-
- 1
- 2
- Next »