Member since
04-22-2014
1218
Posts
341
Kudos Received
157
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 26310 | 03-03-2020 08:12 AM | |
| 16444 | 02-28-2020 10:43 AM | |
| 4744 | 12-16-2019 12:59 PM | |
| 4491 | 11-12-2019 03:28 PM | |
| 6699 | 11-01-2019 09:01 AM |
09-19-2018
11:02 AM
@xBigDatax Openssl does not use JKS files so if bda.truststore is a JKS file (not PEM) then it won't work for trust. You could do something like this: # openssl s_client -connect myucbpaabdapp03:7182 -CAfile <(keytool -list -rfc -keystore opt/cloudera/security/jks/bda.truststore < /dev/null) < /dev/null Note that your certificate does not appear to be self-signed. Rather, it has a null issuer: 0 s:/C=/ST=/L=/O=/OU=/CN=myucbpaabdapp03 i:/C=/ST=/L=/O=/OU=/CN= If this were self signed, the subject an issuer should match like: 0 s:/C=/ST=/L=/O=/OU=/CN=myucbpaabdapp03 i:/C=/ST=/L=/O=/OU=/CN=myucbpaabdapp03 Also note that you did not add in the Fully-qualified DN which will cause problems for TLS and Kerberos. In general, CM/CDH expects fully-qualified domain names so this could run you into trouble later. Lastly, assuming that this once worked as it was configured, we need to know more about your agent communication TLS config to be able to help: (1) We need to know what you have set for the following in Cloudera Manager (checked or unchecked): In Cloudera Manager (Administration --> Settings) - Use TLS Encryption for Agents - Use TLS Authentication of Agents to Server (2) We need to know what you have configured in the config.ini regarding security on the host that cannot heartbeat: # egrep '(cert|key|tls)' /etc/cloudera-scm-agent/config.ini |grep -v "^#" (3) Verify that all of the configuration items match a working node (one that can heartbeat)
... View more
09-18-2018
05:20 PM
@scratch28, Can you clarify what you are seeing/trying to do? It is unclear...screen shots?
... View more
09-18-2018
12:37 PM
@Tomas79, If Hue is shut down from Cloudera Manager, those health checks should cease automatically. If Hue exited unexpectedly you may get these alerts The agent log will tell us much more about the story since it will show what it knew of the Hue process from a start/stop perspective and also it will show any issues getting results from the Hue UI.
... View more
09-18-2018
11:54 AM
@Tomas79, Thanks, that clears things up a lot. Cloudera Manager does health checks and metrics collection via the Hue UI. The Cloudera Manager Agent that runs on the host where the Hue process is will periodically connect to your Hue instance and request "desktop/debug/is_alive". If the agent does not get a response quickly enough, you can see Cloudera Manager report the bad health. I don't see any "is_alive" requests in your logging. If there was a failure, the agent log should show it. Check your /var/log/cloudera-scm-agent/cloudera-scm-agent.log on the Hue host to see if you can find any failures pertaining to "is_alive" or hue in general. What version of CDH are you on, by the way?
... View more
09-18-2018
11:36 AM
@desind, One of the only things I can think of that could have the results you are seeing is if the func_svc_zab_cdh user belongs to another Domain Controller. You could try looking at the user's LDAP DN as a clue, but if I'm right, you can try using the Global Catalog to see if that allows you and the func_svc_zab_cdh user to authenticate. In your Administration --> Settings search for LDAP URL If you are using ldap, change the port to 3268 If you are using ldaps, change the port to 3269 For instance: change ldap://ad.example.com to ldap://ad.example.com:3268 Save the change Restart Cloudera Manager (service cloudera-scm-server restart) Reasoning: - The exception shows that no result was returned for a search for userPrincipalName=func_svc_zab_cdh@vsp.sas.com in the dc=vsp,dc=sas,dc=com subtree - You see in an Active Directory UI that the func_svc_zab_cdh user does have that UPN value - You can log in All these factors seem to indicate that the ldap search just cannot return the user's object. Global Catalog keeps a copy of user objects that span DCs so if that user is in the domain (which the user appears to be) then a search of the GC may get the results you seek.
... View more
09-18-2018
11:08 AM
@sid2707, Very cool and thanks for sharing your solution!
... View more
09-18-2018
10:49 AM
1 Kudo
LOL... thanks man. I was in the .Mac group (what became iCloud eventually) so I'm sorry to say I didn't know too much about Powerschool. Looks like a pretty cool idea that I will check out more later. I'm just very glad that memory was the only limitation. Thanks for sticking with it trying to get it to work; I think most would have given up, so that says a lot 😉 Feel free to share anything else crazy that happens.
... View more
09-18-2018
10:11 AM
1 Kudo
@pollard, Cool, I worked with Apple for a bit in 2005ish too. 🙂 To business, looks like it was Docker memory default to 2GB that was the cause. I forgot that I noticed my container was sluggish so I bumped my Docker memory to 4GB. I just prooved it by reverting to 2GB and then to 4GB as explained below: Here is my session: (1) $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE <none> <none> 6d2b3c820712 19 hours ago 7GB (2) docker run --hostname=quickstart.cloudera --privileged=true -t -i -p 8888:8888 -p 80:80 -p 7180:7180 6d2b3c820712 /usr/bin/docker-quickstart (3) (in container shell) # sudo -u hdfs hdfs dfs -ls / 18/09/18 16:57:09 WARN ipc.Client: Failed to connect to server: quickstart.cloudera/172.17.0.2:8020: try once and fail. java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) (4) I log into Hue as cloudera:cloudera Check Config shows all the failures you mentioned in your first post (5) start over $ exit (in the terminal launched) * basically, kill the running container (6) In Docker UI configuration, Advanced: Set Memory to 4.0GiB Click the Apply & Restart button (7) Run: $ docker run --hostname=quickstart.cloudera --privileged=true -t -i -p 8888:8888 -p 80:80 -p 7180:7180 6d2b3c820712 /usr/bin/docker-quickstart (8) now everything works: # sudo -u hdfs hdfs dfs -ls / Found 6 items drwxrwxrwx - hdfs supergroup 0 2017-10-23 23:46 /benchmarks drwxr-xr-x - hbase supergroup 0 2018-09-18 17:06 /hbase drwxr-xr-x - solr solr 0 2017-10-23 23:48 /solr drwxrwxrwt - hdfs supergroup 0 2018-09-18 17:06 /tmp drwxr-xr-x - hdfs supergroup 0 2017-10-23 23:48 /user drwxr-xr-x - hdfs supergroup 0 2017-10-23 23:48 /var (9) Hue is fine... can run queries NOTE: the 80 and 7180 ports aren't really necessary for exposure to host OS for this test case. I hope this solves the issue for you too! If so, it is pretty lame that lower memory would somehow let servers start and listen but then cause issues for networking to those listening ports. I guess that is a welcome to docker for me 🙂
... View more
09-18-2018
08:49 AM
@gerasimos, What are your goals for your failover or backup strategy? BDR schedules only replicate on a schedule in one direction. "Active-Active" concepts may not truly apply to CDH depending on what you mean by that.
... View more
09-18-2018
08:41 AM
4 Kudos
@desind, There are a few ways to enable DEBUG or TRACE depending on what sort of problem you are attempting to troubleshoot. (1) If CM won't start or if there is a problem where you do not have an idea what classes are involved, you can enable DEBUG or TRACE for the whole server. Warning: this can be very very verbose, so it is likely going to be difficult to capture an event. - Edit /usr/sbin/cmf-server in CM 5--- Edit /opt/cloudera/cm/bin/cm-server in CM 6 - Change this: export CMF_ROOT_LOGGER="INFO,LOGFILE" to export CMF_ROOT_LOGGER="DEBUG,LOGFILE" Restart CM to have the change apply. (2) If you know what class or package you want to DEBUG, you can edit /etc/cloudera-scm-server/log4j.properties: Add lines as follows... this is an example of turning on debugging for just ldap classes in SpringFramework (used in LDAP authentication): log4j.logger.org.springframework.ldap=TRACE log4j.logger.org.springframework.security.ldap=TRACE Restart CM to have the changes apply (3) If you want to turn on some debug or trace level logging for just the current session of Cloudera Manager, you can use the debug page: https://cm_host:cm_port/cmf/debug/logLevel - Choose the Logger from the drop-down - Select the level to which you want to change the logging - Click "Submit Query" button to apply The log level you selected will only apply until you restart Cloudera Manager (4) API debugging. You can enable API debugging in the Cloudera Manager interface: - Navigate to: Administration --> Settings - Search for Enable Debugging of API - Check the box next to it and Save API debugging will be written to the /var/log/cloudra-scm-server/cloudera-scm-server.log file without restart. (5) NOTE: If you do enable verbose debugging, you may need to increase the size of log files or the number to be able to review relevant lines. To do so, I believe you can simply edit the following in /etc/cloudera-scm-server/log4j.properties: log4j.appender.LOGFILE.MaxFileSize=10MB log4j.appender.LOGFILE.MaxBackupIndex=10
... View more