Ports used for retrieval of log entries

[#1]

I am confuguring a CDH 5.9.0 cluster in AWS with Security Groups (effectively firewalls) separating various classes of nodes, e.g. CM is separate to the cluster nodes and Metastore service and HS2 are in separate Security Groups.

Cirrently CM is unable to retrieve log file entries and I am unable to"Download Full Log" for various services that are outside of the Security Group within which CM is running.

I always thought these were retrieved via CM Agent and that port 7182 into the SG containing CM would be enough, but clearly this is not the case.

If I open all ports into the SG containing the cluster nodes then CM is able to successfully access the log entries for say the DataNode role, so this is definitely a port/firewall issue.

From the information available on the following two URLs I am unable to determine the specific ports I need to open in order to allow CM to access the DataNode role logs:

• https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_ports_cm.html
• https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_ports_cdh5.html

The issue applies across the board. CM cannot see Role logs for pretty much any service not within its Security Group.

Can someone point out to me which of the ports are used for CMs retrieval of this informaiton, including whether this is the same for all service roles or different for each.

Thanks.

[#2 · Accepted solution]

Thanks to the joy that is AWS Flow Logs I was able to see what was going on. It would appear that the arrow on https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_ports_cm.html for port 9000 is the wrong way around; CM calls into CMA on 9000/TCP not the other way around.

Cloudera: You might like to confirm this and update the documentation accordingly.