Member since
04-22-2014
1218
Posts
341
Kudos Received
157
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 26260 | 03-03-2020 08:12 AM | |
| 16421 | 02-28-2020 10:43 AM | |
| 4724 | 12-16-2019 12:59 PM | |
| 4474 | 11-12-2019 03:28 PM | |
| 6678 | 11-01-2019 09:01 AM |
09-18-2018
08:41 AM
4 Kudos
@desind, There are a few ways to enable DEBUG or TRACE depending on what sort of problem you are attempting to troubleshoot. (1) If CM won't start or if there is a problem where you do not have an idea what classes are involved, you can enable DEBUG or TRACE for the whole server. Warning: this can be very very verbose, so it is likely going to be difficult to capture an event. - Edit /usr/sbin/cmf-server in CM 5--- Edit /opt/cloudera/cm/bin/cm-server in CM 6 - Change this: export CMF_ROOT_LOGGER="INFO,LOGFILE" to export CMF_ROOT_LOGGER="DEBUG,LOGFILE" Restart CM to have the change apply. (2) If you know what class or package you want to DEBUG, you can edit /etc/cloudera-scm-server/log4j.properties: Add lines as follows... this is an example of turning on debugging for just ldap classes in SpringFramework (used in LDAP authentication): log4j.logger.org.springframework.ldap=TRACE log4j.logger.org.springframework.security.ldap=TRACE Restart CM to have the changes apply (3) If you want to turn on some debug or trace level logging for just the current session of Cloudera Manager, you can use the debug page: https://cm_host:cm_port/cmf/debug/logLevel - Choose the Logger from the drop-down - Select the level to which you want to change the logging - Click "Submit Query" button to apply The log level you selected will only apply until you restart Cloudera Manager (4) API debugging. You can enable API debugging in the Cloudera Manager interface: - Navigate to: Administration --> Settings - Search for Enable Debugging of API - Check the box next to it and Save API debugging will be written to the /var/log/cloudra-scm-server/cloudera-scm-server.log file without restart. (5) NOTE: If you do enable verbose debugging, you may need to increase the size of log files or the number to be able to review relevant lines. To do so, I believe you can simply edit the following in /etc/cloudera-scm-server/log4j.properties: log4j.appender.LOGFILE.MaxFileSize=10MB log4j.appender.LOGFILE.MaxBackupIndex=10
... View more
09-18-2018
12:08 AM
@bgooley thank you it was what I needed.
... View more
09-14-2018
05:57 PM
@desind, No limit that I know of on the CM side. Please start a new thread and provide your LDAP configuration, what happens in the logs and also the "abc_efg_scd_dfc" user LDIF entry. There are lots of reasons for failures, so it is important we start with what you observe and the items involved.
... View more
09-14-2018
09:25 AM
@TomasTF, (1) Service Monitor will pull query information from the Impala Daemons so first make sure that Service Monitor is not having trouble getting information after the queries complete. View: /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-`hostname -f`.log.out (2) When you query Impala queries via Cloudera Manager, Cloudera Manager will request query information from Service Monitor. So, check Cloudera Manager and Service Monitor logs for any log messages that may be relevant when you are trying to view queries: /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-`hostname -f`.log.out /var/log/cloudera-scm-server/cloudera-scm-server.log (3) Make sure you have enough disk space on the volume where your Service Monitor Storage Directory is located. (4) Lastly, if there are no messages that indicate problems with the above, try increasing Impala Storage from 1GiB to 2 or 3 GiB. Perhaps for some reason you were running out of space, but I don't think that is too likely. Worth a try if nothing else mentioned above gives you any clues.
... View more
09-06-2018
06:47 PM
Hi @bgooley, Thank you for the response. Here are the contents - On CM host below CA cert is generated [root@hostname pki]# ls -ltr total 20 -rw-r--r-- 1 root root 1121 Sep 2 12:27 hostname-server.csr -rw-r--r-- 1 root root 1834 Sep 2 12:27 ca-key -rw-r--r-- 1 root root 1314 Sep 2 12:27 ca-cert -rw-r--r-- 1 root root 4198 Sep 2 12:28 hostname.jks lrwxrwxrwx 1 root root 7 Sep 6 10:21 4ba83cc1.0 -> ca-cert [root@hostname pki]# pwd /opt/cloudera/security/pki -> ca cert is imported to keystore along with jsse certs on CM host and same copied to one of the agent hosts. -> copied the contents for ca-cert generate through - openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650 -> created the sym link for the ca-cert, updated the parameters in /etc/cloudera-scm-agent/config.ini along verify_cert_dir -> Here is the output from verification - [root@hostname1.domain cloudera-scm-agent]# openssl s_client -connect hostname:7183 -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem < /dev/null CONNECTED(00000003) depth=0 C = Unknown, ST = Unknown, L = Unknown, O = Unknown, OU = Unknown, CN = hostname verify return:1 --- Certificate chain 0 s:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname i:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname --- Server certificate -----BEGIN CERTIFICATE----- MIIDmTCCAoGgAwIBAgIEFdAE2jANBgkqhkiG9w0BAQsFADB9MRAwDgYDVQQGEwdV bmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQHEwdVbmtub3duMRAwDgYD VQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMSEwHwYDVQQDExhjc2VraGFy MDIucWU5LmppZ3Nhdy5jb20wHhcNMTgwOTAyMTkyNzAyWhcNMjgwODMwMTkyNzAy WjB9MRAwDgYDVQQGEwdVbmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQH EwdVbmtub3duMRAwDgYDVQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMSEw HwYDVQQDExhjc2VraGFyMDIucWU5LmppZ3Nhdy5jb20wggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQCg4ailB41WoGo6h3fepc6wILw37WaRZ3IApxj0ZTO0 n8po151baw0IEk1X+lYXAkuXloycCKsFUhjiOU2AF2iBbVOYXyA9IB5a9a+b507r EG1NqdJrdN6k4kZrsv+91CizYunjNNheDIgUv/gA7U/sSbvvchXUlN+j4y9LpwKH wcbY/kKLZV9EoNeyrJANccHYiHRQgPLfEcQoHJtDRuWt0TIDqZOR+Xe6fICLfDiC 8WRXqpYKRfFoLVwzFEJuilZXT/J9obzhRgxt9sF7eVV86kFa2EIEZ7aAzZGdIeVT s3RpUO/iSq2X5DxQ7BH8Duc6kgfuaAIOCuANeEGO7trxAgMBAAGjITAfMB0GA1Ud DgQWBBTPmi/hBkF9xJxiN4bvB2QehVuM5TANBgkqhkiG9w0BAQsFAAOCAQEAJEgp 747BfFBIKVPoBCOnxopLM0iJ0jhOTMPo2gld8XoHdH52UOKPlnTsYxqRv9qcU2m8 zjlNxpzT2LMEMGWwk8cP+4ikGrW1vI/bUZOVwFg2pDeho/AE9IOgdn6kd9OQXnL+ D2AB8woJwLWEQ755CEETjvk84kC4BVMMCAB4tDhwAEmzhsehg5B1FzVGfNE5k7fS Ey+ZkNMr/9omWYd4lzHja8pCBCISUCVU7c4ybnEdvLfhyGk/uoNzE2Tf11OsmNtj Fqc4Wbfr9BmcEzDWP6pO5ID6hOu93JaKPmMiW6ofzoJfUdF9Mj/q8pEBc88dqpbf aMvL2RAdfPcFFqSEWA== -----END CERTIFICATE----- subject=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname issuer=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname --- No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 1409 bytes and written 415 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 5B91D501E15D42E800371EBAFE7BF3FD673EEEDA1A10E30CBEC808C2431F2325 Session-ID-ctx: Master-Key: A0DD11CA122343D962AFE236893EC2F00371D37DF3BDC340AB3F1CCDC25C8E48F4DC28255A6CC1926654D1708FE23B9A Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1536283905 Timeout : 300 (sec) Verify return code: 0 (ok) --- DONE restarted CM server and agent Please advise if I'm missing anything.
... View more
09-04-2018
04:08 PM
LOL... just checked the dates. It has been quite a while since Neta Je posted this (09-03-2015!) I just saw your response, @Koti_Karri and followed along ;-).
... View more
08-29-2018
10:35 AM
@DanielWhite, Can you clarify what you mean by "source server?" Really, the answer is No. Your source configuration dictates what NameNode(s) to communicate with. Your source's NameNodes tell clients where to get blocks. Those blocks can be on any DataNode in the source cluster. On the target side where the MapReduce Job runs, the Resource Manager decides on which nodes the Mappers will run.
... View more
08-29-2018
02:40 AM
Thanks for the information!
... View more
08-28-2018
07:24 AM
@bgooley Got it! I'm relatively new to Python development and was trying figure out what was an acceptable form of multiple criteria for a single POST request for the same key. <== Maybe that sentence will help someone find this later on. I used tcpdump as well but could not see the POST data as it was encrypted with SSL. I was not able to turn off SSL to see plain text content as I don't have access to the server. I was exploring using a cert to decrypt but have never done that using tcpdump. I would love to know how, as I have run into trying to debug traffic and could not see the content. Anyway, I was having trouble creating a dict for the form data as it would only accept a single key. I didn't realize that an array(list) with a single dict key would work as well. This line was a problem in that Python would not let me add multiple key entries like "user_ids='4', user_ids='5'". I didn't realize all it needed was a list. form_data = dict(csrfmiddlewaretoken=session.cookies['csrftoken'] ,next='/hue/useradmin/users/', user_ids=['4','5']) I tried every combination I could think of but made the mistake of surrounding the arguments in quotes, making them a single text argument versus a list. What you did for me was verify a few things and put me on the right trail. Thanks to everyone for their contribution!
... View more
08-27-2018
10:45 PM
1 Kudo
@krb, What you provided appears to be an agent log message that indicates an attempt to kinit with the HTTP principal on the host where HTTPFS role runs was not successful. Check on the host where the httpfs role runs and make sure the krb5.conf file is correct. This shoud not impact HDFS as a whole since HTTPFS is a client of HDFS really. Cloudera Manager should merge the HTTP principal automatically, so please run the following to make sure the keytab has the right keys: # klist -kte /run/cloudera-scm-agent/process/1330-hdfs-HTTPFS/httpfs.keytab
... View more