Member since
02-11-2015
10
Posts
0
Kudos Received
0
Solutions
11-15-2018
10:53 AM
Is there anyway where Email Alerts are triggered via Ranger for below scenario? 1. Adding Access for user. => Send an email to user who received the access on the resource. 2. Access Request Denied => Send an email to Admin & User about trying to access the data without permission. 3. When delegated user add any new user => Send an email to Admin & used about new access. Thanks, B
... View more
Labels:
09-11-2018
04:36 AM
Did you got any answer to it?
... View more
10-13-2017
03:41 PM
Hi All, I have setup Hive LDAP Authentication. Locally on OS I have configured ldap.conf, sssd.conf, krb5.conf. I have AD Federation from which each user has ~30 groups to be synced. Important Information. 1. Cluster is not kerberized. 2. LDAP Authentication is enabled for HiveServer. 3. Ranger is syncing data from AD/LDAP. 4. HiveServer2 Authentication with LDAP is enabled and working. 5. Impersonation is on in HiveServer2. 6. Namenode has HA. Question: Hive is taking ~2 minutes to login successfully. But than it give below error in the log & show databases; does not work. "Error getting policies." Hive log says ~~~~ 2017-10-13 14:01:53,976 WARN [Thread-14]: client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(154)) - Error getting policies. secureMode=false, user=hive (auth:SIMPLE), response={"httpStatusCode":400,"statusCode":0}, serviceName=dev_hive 2017-10-13 12:51:03,358 ERROR [HiveServer2-Handler-Pool: Thread-84]: ql.Driver (SessionState.java:printError(993)) - FAILED: HiveAccessControlException Permission denied: user [XXXX] does not have [USE] privilege on [Unknown resource!!]
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException: Permission denied: user [XXXX] does not have [USE] privilege on [Unknown resource!!]
at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.checkPrivileges(RangerHiveAuthorizer.java:460)
at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:856)
at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:644)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:511)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1221)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1215)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:146)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:226)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:264)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:470)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:457)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy40.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:313)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:509)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
10-13-2017
12:38 AM
please correct the repo file path. 2.5.2.0 also works fine with corrected path. Observer ppc in the path correct it for ambari. Than you should be good. I have done this for RHEL 7.
... View more
03-04-2015
11:07 PM
Below is the configuration file which is used to generate certificate for all nodes. CN and DNS.1/2 where modified as per the node. # # OpenSSL example configuration file. # This is mostly being used for generation of certificate requests. # # This definition stops the following lines choking if HOME isn't # defined. HOME = . RANDFILE = $ENV::HOME/.rnd # Extra OBJECT IDENTIFIER info: #oid_file = $ENV::HOME/.oid oid_section = new_oids # To use this configuration file with the "-extfile" option of the # "openssl x509" utility, name here the section containing the # X.509v3 extensions to use: # extensions = # (Alternatively, use a configuration file that has only # X.509v3 extensions in its main [= default] section.) [ new_oids ] # We can add new OIDs in here for use by 'ca', 'req' and 'ts'. # Add a simple OID like this: # testoid1=1.2.3.4 # Or use config file substitution like this: # testoid2=${testoid1}.5.6 # Policies used by the TSA examples. tsa_policy1 = 1.2.3.4.1 tsa_policy2 = 1.2.3.4.5.6 tsa_policy3 = 1.2.3.4.5.7 #################################################################### [ ca ] default_ca = CA_default # The default ca section #################################################################### [ CA_default ] dir = /etc/pki/CA # Where everything is kept certs = $dir/certs # Where the issued certs are kept crl_dir = $dir/crl # Where the issued crl are kept database = $dir/index.txt # database index file. #unique_subject = no # Set to 'no' to allow creation of # several ctificates with same subject. new_certs_dir = $dir/newcerts # default place for new certs. certificate = $dir/cacert.pem # The CA certificate serial = $dir/serial # The current serial number crlnumber = $dir/crlnumber # the current crl number # must be commented out to leave a V1 CRL crl = $dir/crl.pem # The current CRL private_key = $dir/private/cakey.pem# The private key RANDFILE = $dir/private/.rand # private random number file x509_extensions = usr_cert # The extentions to add to the cert # Comment out the following two lines for the "traditional" # (and highly broken) format. name_opt = ca_default # Subject Name options cert_opt = ca_default # Certificate field options # Extension copying option: use with caution. copy_extensions = copy # Extensions to add to a CRL. Note: Netscape communicator chokes on V2 CRLs # so this is commented out by default to leave a V1 CRL. # crlnumber must also be commented out to leave a V1 CRL. # crl_extensions = crl_ext default_days = 365 # how long to certify for default_crl_days= 30 # how long before next CRL default_md = default # use public key default MD preserve = no # keep passed DN ordering # A few difference way of specifying how similar the request should look # For type CA, the listed attributes must be the same, and the optional # and supplied fields are just that 🙂 policy = policy_match # For the CA policy [ policy_match ] countryName = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName = supplied emailAddress = optional # For the 'anything' policy # At this point in time, you must list all acceptable 'object' # types. [ policy_anything ] countryName = optional stateOrProvinceName = optional localityName = optional organizationName = optional organizationalUnitName = optional commonName = supplied emailAddress = optional #################################################################### [ req ] default_bits = 2048 default_md = sha1 default_keyfile = privkey.pem distinguished_name = req_distinguished_name attributes = req_attributes x509_extensions = v3_req # The extentions to add to the self signed cert prompt = no # Passwords for private keys if not present they will be prompted for # input_password = secret # output_password = secret # This sets a mask for permitted string types. There are several options. # default: PrintableString, T61String, BMPString. # pkix : PrintableString, BMPString (PKIX recommendation before 2004) # utf8only: only UTF8Strings (PKIX recommendation after 2004). # nombstr : PrintableString, T61String (no BMPStrings or UTF8Strings). # MASK:XXXX a literal mask value. # WARNING: ancient versions of Netscape crash on BMPStrings or UTF8Strings. string_mask = utf8only # req_extensions = v3_req # The extensions to add to a certificate request [ req_distinguished_name ] C = US ST = California L = San Jose O = Org CN = nn.mydomain.local # SET-ex3 = SET extension number 3 [ req_attributes ] challengePassword = A challenge password challengePassword_min = 4 challengePassword_max = 20 unstructuredName = An optional company name [ usr_cert ] # These extensions are added when 'ca' signs a request. # This goes against PKIX guidelines but some CAs do it and some software # requires this to avoid interpreting an end user certificate as a CA. basicConstraints=CA:FALSE # Here are some examples of the usage of nsCertType. If it is omitted # the certificate can be used for anything *except* object signing. # This is OK for an SSL server. # nsCertType = server # For an object signing certificate this would be used. # nsCertType = objsign # For normal client use this is typical # nsCertType = client, email # and for everything including object signing: # nsCertType = client, email, objsign # This is typical in keyUsage for a client certificate. # keyUsage = nonRepudiation, digitalSignature, keyEncipherment # This will be displayed in Netscape's comment listbox. nsComment = "OpenSSL Generated Certificate" # PKIX recommendations harmless if included in all certificates. subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer # This stuff is for subjectAltName and issuerAltname. # Import the email address. # subjectAltName=email:copy # An alternative to produce certificates that aren't # deprecated according to PKIX. # subjectAltName=email:move # Copy subject details # issuerAltName=issuer:copy #nsCaRevocationUrl = http://www.domain.dom/ca-crl.pem #nsBaseUrl #nsRevocationUrl #nsRenewalUrl #nsCaPolicyUrl #nsSslServerName # This is required for TSA certificates. # extendedKeyUsage = critical,timeStamping [ v3_req ] # Extensions to add to a certificate request basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = nn.mydomain.local DNS.2 = 1.1.1.1 [ v3_ca ] # Extensions for a typical CA # PKIX recommendation. subjectKeyIdentifier=hash authorityKeyIdentifier=keyid:always,issuer # This is what PKIX recommends but some broken software chokes on critical # extensions. #basicConstraints = critical,CA:true # So we do this instead. basicConstraints = CA:true # Key usage: this is typical for a CA certificate. However since it will # prevent it being used as an test self-signed certificate it is best # left out by default. # keyUsage = cRLSign, keyCertSign keyUsage = digitalSignature, keyEncipherment # Some might want this also # nsCertType = sslCA, emailCA # Include email address in subject alt name: another PKIX recommendation # subjectAltName=email:copy # Copy issuer details # issuerAltName=issuer:copy # DER hex encoding of an extension: beware experts only! # obj=DER:02:03 # Where 'obj' is a standard or added object # You can even override a supported extension: # basicConstraints= critical, DER:30:03:01:01:FF [ crl_ext ] # CRL extensions. # Only issuerAltName and authorityKeyIdentifier make any sense in a CRL. # issuerAltName=issuer:copy authorityKeyIdentifier=keyid:always [ proxy_cert_ext ] # These extensions should be added when creating a proxy certificate # This goes against PKIX guidelines but some CAs do it and some software # requires this to avoid interpreting an end user certificate as a CA. basicConstraints=CA:FALSE # Here are some examples of the usage of nsCertType. If it is omitted # the certificate can be used for anything *except* object signing. # This is OK for an SSL server. # nsCertType = server # For an object signing certificate this would be used. # nsCertType = objsign # For normal client use this is typical # nsCertType = client, email # and for everything including object signing: # nsCertType = client, email, objsign # This is typical in keyUsage for a client certificate. # keyUsage = nonRepudiation, digitalSignature, keyEncipherment # This will be displayed in Netscape's comment listbox. nsComment = "OpenSSL Generated Certificate" # PKIX recommendations harmless if included in all certificates. subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer # This stuff is for subjectAltName and issuerAltname. # Import the email address. # subjectAltName=email:copy # An alternative to produce certificates that aren't # deprecated according to PKIX. # subjectAltName=email:move # Copy subject details # issuerAltName=issuer:copy #nsCaRevocationUrl = http://www.domain.dom/ca-crl.pem #nsBaseUrl #nsRevocationUrl #nsRenewalUrl #nsCaPolicyUrl #nsSslServerName # This really needs to be in place for it to be a proxy certificate. proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo #################################################################### [ tsa ] default_tsa = tsa_config1 # the default TSA section [ tsa_config1 ] # These are used by the TSA reply generation only. dir = ./demoCA # TSA root directory serial = $dir/tsaserial # The current serial number (mandatory) crypto_device = builtin # OpenSSL engine to use for signing signer_cert = $dir/tsacert.pem # The TSA signing certificate # (optional) certs = $dir/cacert.pem # Certificate chain to include in reply # (optional) signer_key = $dir/private/tsakey.pem # The TSA private key (optional) default_policy = tsa_policy1 # Policy if request did not specify it # (optional) other_policies = tsa_policy2, tsa_policy3 # acceptable policies (optional) digests = md5, sha1 # Acceptable message digests (mandatory) accuracy = secs:1, millisecs:500, microsecs:100 # (optional) clock_precision_digits = 0 # number of digits after dot. (optional) ordering = yes # Is ordering defined for timestamps? # (optional, default: no) tsa_name = yes # Must the TSA name be included in the reply? # (optional, default: no) ess_cert_id_chain = no # Must the ESS cert id chain be included? # (optional, default: no)
... View more
03-04-2015
10:56 PM
Thanks Ben for looking into it. We have been generating certificate with openssl file. What I have done is I have hdfs.jks keystore. In this store I have added self signed certificates for all the nodes in the cluster and I have truststore cacerts which has all nodes certificates. I have deployed this truststore and keystore on all the nodes. Setup all required details in Cluster for HDFS, Yarn and MapReduce. As soon as I have restarted the server I got earlier error. Now as I see in the logs I have also got below two errors before earlier error. 1:33:16.389 AM INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode Image has changed. Downloading updated image from NN. 1:33:16.389 AM INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage Opening connection to https://nn.mydomain.local:50470/imagetransfer?getimage=1&txid=1040569&storageInfo=-59:2055389451:0:cluster2 1:33:16.409 AM WARN org.apache.hadoop.security.UserGroupInformation PriviledgedActionException as:hdfs/nn.mydomain.local@MYDOMAIN.LOCAL (auth:KERBEROS) cause:java.io.IOException: HTTPS hostname wrong: should be <nn.mydomain.local> Note: This cluster is kerberized and its working fine without this SSL setup enablement. My understanding is: Each node should have self signed certificate of all nodes in keystore and truststore should also have trust certificate from all the nodes. That is what I have done here. Let me know If we are missing somethig. Thanks, Bhavin
... View more
02-12-2015
12:29 AM
I have issued a self signed certificate to my Cloudera Manager node. and copied over all the nodes in the cluster. I am following steps given on this page http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_tls_browser.html I have restarted cloudera manager server and agents on all the nodes. Cloudera manager UI redirected to https://ip:7183 as expected. However when I tried to restart Cloudera Management Service it is not getting restarted and throwing below error. 2015-02-12 03:19:05,632 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: (29 skipped) Failed to send metrics. java.lang.reflect.UndeclaredThrowableException at com.sun.proxy.$Proxy101.writeMetrics(Unknown Source) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:287) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:274) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:129) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88) ... 11 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72) at org.apache.avro.ipc.Requestor.request(Requestor.java:147) at org.apache.avro.ipc.Requestor.request(Requestor.java:101) at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72) ... 11 more Any help appreciated. Am I missing something?
... View more