After disabling TLS/SSL ,cloudera management services are not able to start

[#1]

Hi,

We were trying to enable TLS on our cluster.We have followed cloudera steps ,but i think missed something.After enabling TLS,we were able to login to port 7183,but all  cloudera management services were disabled.

So we reverted back TLS changes from cloudera and removed all keys and certificates.After that given restart to the cloudera manager.

After this process,port 7180 started and login page came.But when i am trying to login then it is not redirecting to the GUI.

when i have check the logs. then these exceptions are showing.

2019-12-17 13:24:29,320 INFO WebServerImpl:com.cloudera.server.web.cmon.JobDetailGatekeeper: ActivityMonitor configured to allow job details for all jobs.
2019-12-17 13:24:30,850 WARN ActionablesProvider-0:com.cloudera.server.web.cmf.StatusProvider: Failed to submit task for getting status from SERVICE_MONITORING
com.cloudera.cmon.MgmtServiceNotRunningException: SERVICE_MONITORING is not running

2019-12-17 13:24:50,238 WARN EventStorePublisherWithRetry-0:com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={CATEGORY=[AUDIT_EVENT], SEVERITY=[INFORMATIONAL], SERVICE=[ClouderaManager], SERVICE_TYPE=[ManagerServer], USER=[admin], EVENTCODE=[EV_LOGIN_SUCCESS], MESSAGE_CODES=[LOGIN_SUCCESS]}, content=User admin logged in successfully., timestamp=1576569290230}
2019-12-17 13:24:52,250 INFO 1267052060@scm-web-0:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '' from 172.26.194.114
2019-12-17 13:25:05,877 INFO 1267052060@scm-web-0:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.26.194.114
2019-12-17 13:25:11,032 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy122.writeMetrics(Unknown Source)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
... 11 more
Caused by: java.net.ConnectException: Connection refused

Do anyone have any idea on that.Please suggest. 

Thanks.

Devendra

[#2]

@pdev have you disabled TLS in the management services as well? After you disable/enable TLS, you will have restart the management services so they communicate with CM with the changed port and protocol.

[#3]

Hi Rajesh,

We have disabled tls from cloudera admin and removed truststore path and password from cloudera management services.

I hope this what you are asking.if not then please share steps to disable it.

Thanks.

Pdev

[#4 · Accepted solution]

Hi @pdev ,

You may want to check this community thread:

https://community.cloudera.com/t5/Support-Questions/how-to-rollback-cloudera-manager-tls-configurati...

It is possible that some of the data was still in CM database which expects TLS login. Could you please run below SQL command to confirm?

select attr, value from CONFIGS where attr in ('web_tls', 'agent_tls');

If you see any returned value is "true" then you need to update the CM database manually. See below steps:

1. Back up database/table before you makde any further changes
2. Run the following queries in the CM database:
   

   Update TLS for web_tls update CONFIGS set value = 'false' where attr = 'web_tls';
   Update TLS for agent_tls update CONFIGS set value = 'false' where attr = 'agent_tls';

3. In the browser do clear cache including passwords, and cookies before trying to login.

Thanks and hope this helps!

[#5]

thanks  a lot Li Wang.It worked .

pdev

[#6]

Hi @pdev ,

Wonderful to hear that! Thanks for marking this thread as resolved!

Cheers,

Li