About Sean464

Sean464 · ‎01-31-2024

I'm not sure if we have example scripts or custom functions, but the general idea to achieve what you're trying to do would be - 1. Referring the HDFS reporting in CM or parse the fsimage (for large files). 2. Write a MapReduce/Spark job to scan files, or run a query in Hive/Impala/SparkSQL to see if data files are mapped to Hive tables. You can also implement access controls through Ranger, enable transparent disk encryption (TDE) using Ranger KMS, tag sensitive hive columns (containing PII, PCI, PHI) using Atlas classification and assign tag-based masking policies from Ranger, implement navencrypt to encrypt the spill files, et. - if you're planning to protect the sensitive data from unauthorised access.

Sean464 · ‎01-30-2024

@phir1 , Can you give us more context on the use-case, please? What type of monitoring you're planning to implement? And, by Hue directory, which path are you referring to?

Sean464 · ‎09-21-2021

@vorraluck, Following is the error I see, which points to the parameter 'SamplePassword123'. You're missing "=" sign in this parameter as the error hints. # openssl req -new -sha256 -nodes -newkey rsa:2048 -config /tmp/openssl.cnf error on line 13 of /tmp/openssl.cnf 140163633919888:error:0E079065:configuration file routines:DEF_LOAD_BIO:missing equal sign:conf_def.c:345:line 13 Change SamplePassword123 to SamplePassword = 123. Once the change is made, both the private key and the CSR file can be created. [root@node1 ~]# openssl req -new -sha256 -nodes -newkey rsa:2048 -config <( > > cat <<-EOF > > [req] > > default_bits = 2048 > > prompt = no > > default_md = sha256 > > req_extensions = req_ext > > distinguished_name = dn > > SamplePassword = 123 > > [ dn ] > > C=TH > > ST=Bangkok > > L=Chaxxx > > O=xtac Public Company Limited > > OU=Enterprise Service Support > > emailAddress=john.doe@xtac.co.th > > CN = rs-xxx-hmb-201.xtac.dev > > > > [ req_ext ] > > subjectAltName = @alt_names > > > > [ alt_names ] > > DNS.1 = rs-xxx-hmb-201 > > DNS.2 = rs-xxx-hmb-201.xtac.dev > > > > EOF > > ) Generating a 2048 bit RSA private key .+++ .....................................................................................................................................................................................................................+++ writing new private key to stdout -----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC0PnXQOdpbKGso wNY7K+pYItm3G4/yL7HYjViUzxQsEZasIypZrYIY8KqF64bndEf002YkaHcEDCFR xr59eBfpmXYWezl89ZeRU2YEL5188ubSrkdo/P0IdBSGAbu1/2WlQ26AyIHTkG96 ppsOa25vbKPdOmPh37ylB0C1ghCk6utib5/HAiIt0X9YqGERfnTVXpo/fRzxSHdb 7eioxFx0CP38Vzo/wXQP2FDwa3bw0MTNNF813ttC0yzRpcFf4ACde2jzFpehgtY0 E3yvq7/gVURibZgg6MSVVrg/dzjlReiRFxQOO4XDcuNn1VLSJ23UOT4wXwszKIe2 e7nNPtb5AgMBAAECggEAEhorzuvgiEM47/DeEzdH4+4sG33DKTmtDOi1OszJY9uo XEVz3WnVpReWqLiM5fYBvA73NTTgryFgv0vMmeFT1Xw6JeveTDSGMmxD1KvLj179 267xMRQnfY/a99J4vcCWvtBU6s44a70X1SOicwJLJxmsI23X1jmYQqLF1vTnIwjh of6e6jQWeE1nyrhHctqlyYfFNat6uK8NfJ21PTb4yfjp5dhefgaIK/9qOB9SKMwk LFXLGVlePBFziadY8spbRl56mcbEC6mgbXff4Zbzy5QhXxQaN38358G5fSGpsgiQ 44rTFrC2PTPzS1HeGpqdAwhqFU5lEXnMVegS43ZeAQKBgQDrngKDgKnXY/HBO9E0 OAzC2j+O6BBalPl65f9l6rhiZo8QwhcJciMyCkgnDxnXb/FkKcBmLx+QJXjYguQ4 zkXMEqCS5Bz+jrwnQrnAh/TIlKyZPxy2qpJUvUJND8DZ7ZMVmBycOPI4gFjoIx/G U0kPPdgHhdRtL/Uv2XsXBM4/4QKBgQDD1ibpDuvAd2x1EKAEWwj3v/Ee41am+Q9Z WjFWDmS7jV8EYPqOrYedKyaY0t5ArOOKUHRiOp0VOsTWF7m7NAzwmvdspV0YeShh qCPuWWOGBATjVOMmPgqKBUn3U67y7pZRX6tpOlcwj8sB6KQRpm9NpLcMSFrDWxEz oCgLHVXaGQKBgQDQirxC1GB/SfCyTVVvWKTC2hUjUXcYFX9zLZsOA+BLB+dct81Y CPPp2HvgRldi/au0MdgfGVpgZSo+yCtjs/7HDz2chda74G3cegyawjsARcc2pEuv ye1Wn2TNfEH/IW3r0QSRqT2KkN8gJ+Z5zUF/AgfxMJzCP45OWbm+t/wtQQKBgCou Yksor5bRIkdEwXKuuQvECAeDKBLm6mtwhdfnWcMb/C9RRCafMeqkdLfOE5kSpCAS nD7cKSF9exAyJAsyducMOebo51hyIESIltSr5EflbbgZfKOsEVEROpFPMQuaYD4+ wQj1S/plvnA2z8ANfUPYCqVWoFYbs2TPRlC+jNNhAoGAPgGiUz+4lDvQL62R6xDP R8OUGhmfB37nQy4NFNaiRxQW9TSdrg09cb3CS04IjyH9QQmetRyhZSXCYCu7H/bf H/rR5S7/e5WKgtgAIpNhYNXg1p6zoac6cXBYUO4ekf/PR70cFXAQdzFt/sQIExiy SV6SL1F3ja/sn9/wUXkvZgc= -----END PRIVATE KEY----- ----- -----BEGIN CERTIFICATE REQUEST----- MIIDTDCCAjQCAQAwgcExCzAJBgNVBAYTAlRIMRAwDgYDVQQIDAdCYW5na29rMQ8w DQYDVQQHDAZDaGF4eHgxJDAiBgNVBAoMG3h0YWMgUHVibGljIENvbXBhbnkgTGlt aXRlZDEjMCEGA1UECwwaRW50ZXJwcmlzZSBTZXJ2aWNlIFN1cHBvcnQxIjAgBgkq hkiG9w0BCQEWE2pvaG4uZG9lQHh0YWMuY28udGgxIDAeBgNVBAMMF3JzLXh4eC1o bWItMjAxLnh0YWMuZGV2MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA tD510DnaWyhrKMDWOyvqWCLZtxuP8i+x2I1YlM8ULBGWrCMqWa2CGPCqheuG53RH 9NNmJGh3BAwhUca+fXgX6Zl2Fns5fPWXkVNmBC+dfPLm0q5HaPz9CHQUhgG7tf9l pUNugMiB05BveqabDmtub2yj3Tpj4d+8pQdAtYIQpOrrYm+fxwIiLdF/WKhhEX50 1V6aP30c8Uh3W+3oqMRcdAj9/Fc6P8F0D9hQ8Gt28NDEzTRfNd7bQtMs0aXBX+AA nXto8xaXoYLWNBN8r6u/4FVEYm2YIOjElVa4P3c45UXokRcUDjuFw3LjZ9VS0idt 1Dk+MF8LMyiHtnu5zT7W+QIDAQABoEUwQwYJKoZIhvcNAQkOMTYwNDAyBgNVHREE KzApgg5ycy14eHgtaG1iLTIwMYIXcnMteHh4LWhtYi0yMDEueHRhYy5kZXYwDQYJ KoZIhvcNAQELBQADggEBAH9kfI4ZCJRDF2lKjCjLgj/UaRpdO9yIcWUsQHU6jpSU meKpjkDgn8zTB8hTJ8C8K5rIp1BbGB8Oe/ppdbgzmIKPLjEKUGxIIkBMdSnPRmAg BUy+nArMmVGrv1weA7F1E/ECzdR1NeXl6eWLYDbmHIwT1/dkpL7evCE8zYufQb9u H1/U/zzyWiiOuneL+GzhCh7srQkQCtjiWrnuhjZnANvRSlR+FIYY7PWlY1e2m7go kL5CRVbSUZaYvknIAgOBJs2fkvyrR+YFlKqzlhN3i9yXdfAC09D0kr0FE9CUedh1 HJsxwMB5S1L2zZXAS0iNqoTTb0wHi3UbKLJIMeQx0RU= -----END CERTIFICATE REQUEST----- [root@node1 ~]# Cheers! Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.

Sean464 · ‎08-07-2021

Hue is unable to handle the slow responses from Hive servers and hence the error. This happens when Hive is unable to return the query handle to Hue in a timely manner. So, the ideal solution here would be to investigate Hive slowness and not Hue. Hue relies on server_conn_timeout parameter in its configuration (default 120s), after which the active sessions with hiveserver2 is teared down. If Hue is using MySQL as the underlying DB, then we can tweak the innodb-lock-wait-timeout to be greater than the server_conn_timeout in Hue's configuration, that way we get fail-fast system and the locked tables are released. Please note that this is only a workaround until the cause of Hive performance issue is found and remediated.

Sean464 · ‎06-07-2021

Hello @Amn_468, That's quite strange, as the service should generally look for the log file before starting. The only possibility that I can think of here is that the log file would have been removed manually while the service was running. Having said that, yes, let's try creating the log file manually with appropriate permissions as suggested by @nthomas, and restart Hue kt_renewer service to see if it is able to write to the newly created file.

Sean464 · ‎06-06-2021

@PrernaU The below exception hints that 'hdfs' user is not authorised to submit jobs in the Yarn queue 'default', and this is due to ACL in place. Caused by: org.apache.hadoop.security.AccessControlException: User hdfs does not have permission to submit application_1622150236559_0013 to queue default Is ACL managed by Ranger please? If so, we will need to add 'hdfs' user to have complete access over Yarn queue 'default' under Yarn policies in Ranger UI.

Sean464 · ‎06-02-2021

Hello @Amn_468 , Are we setting up Hue KT_renewer for the first time? What is the status of Hue kt_renewer service please? stderr logs from Hue kt_renewer can give more insights. The below commands can take us to the directory were stderr log is stored. # export HUE_DIR="/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep hue-KT_RENEWER | tail -1 | awk '{print $9}'`" # cd $HUE_DIR/logs

Sean464 · ‎03-29-2021

Hello @srinikar87 The error indicates that your KDC server requires clients to pre-authenticate themselves before it can issue a Ticket Granting Ticket (TGT). If your KDC is a MIT kerberos, then probably a command line argument was passed by your administrator to require all clients to use pre-authentication. In this case, we can run the following command on your KDC server which will disable pre-authentication only for the impala principal, and the generate missing credentials should work. kadmin.local: modprinc -requires_preauth impala/e2e-02-cdlkc1.nokia.com@NOKIA.COM If your KDC is an AD server, then your AD administrator must enable the “Do not require Kerberos pre-authentication” checkbox in the user properties of the newly created impala principal. Refer the attachment. Please let us know how this goes. Reference: http://web.mit.edu/kerberos/krb5-1.5/krb5-1.5.4/doc/krb5-admin/Adding-or-Modifying-Principals.html Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.

Sean464 · ‎02-26-2021

@nj20200 As @GangWar mentioned, this is more of a yum related issue. Cloudera manager requires openssl, openssl-libs, and openssl-devel packages to be installed. It appears your yum repo server does not contain openssl-devel-1.0.2k-19.el7.x86_64.rpm, and hence the error. Once your OS team or the team that manages the repo server is able to sync or copy the required openssl rpms from satellite server, you should be able to install cloudera manager package without any issue.

Sean464 · ‎12-17-2020

Hello @prabhat10 , Try this - Backup your /etc/krb5.conf on all the hosts Verify the encryption types supported from your Kerberos server (If MIT - Check "supported_enctypes" in /var/kerberos/krb5kdc/) Check the "Kerberos Encryption Types" under CM > Administration > Security > Kerberos Credentials > Configuration. Include the encryption types supported by your KDC. Enable "Manage krb5.conf through Cloudera Manager" from the same configuration page. Select "Deploy Kerberos client configuration" from the drop-down near your cluster. Once deployed, verify if the krb5.conf on the agent nodes have the encryption types included as mentioned in CM. If CM server is running on stale kerberos configuration, copy the krb5.conf from one of the agent nodes to CM server. Regenerate the principals from CM. (If this is success, you should be able to restart CM and CDH services).

Online	Offline
Last Visited	‎09-19-2024 09:29 AM

Member Since	‎03-25-2020 12:38 AM
Last Visited	‎09-19-2024 09:29 AM
Posts	132
Kudos received	4

Cloudera Community

Re: How to secure Kafka Broker Server with a SSL C...

Re: Hue kt log error

Re: DFSIO fails with error "User hdfs does not hav...

Re: Hue Directory Monitoring

Re: Hue Directory Monitoring

Re: How to secure Kafka Broker Server with a SSL C...

Re: You can't execute queries until the end of the...

Re: Hue kt log error

Re: DFSIO fails with error "User hdfs does not hav...

Re: Hue kt log error

Re: Kerberos Error while adding new impala service

Re: Error: Package: 1:openssl-devel-1.0.1e-60.el7....

Re: kinit: KDC has no support for encryption type ...