Ambari with mysql 5.6

[#1]

I had a tough time with Ambari 1.7 pointing to Mysql 5.6 on remote host. AS is using all the available ports on the source to make a connection to remote host to port 3306. At one time i saw 28000 ports being used by ambari to connect. this is killing us because it even used 50070 at one time, and i had tough time to figure it out. Finally i stopped ambari-server and all is well. Did anyone face this issue? if so any idea on fix? apart from downgrading Mysql or upgrade ambari to 2.1.2?

[#2]

@Raja Sekhar Chintalapati - Sounds like the OS level TCP IP params are not tuned correctly.

What kind of OS are you using? For e.g. on Centos and RHEL, you can use these command to check the values -

sysctl net.ipv4.tcp_keepalive_time

sysctl net.ipv4.tcp_keepalive_probes

sysctl net.ipv4.tcp_keepalive_intvl 

There are other params that can be tuned as well but these 3 are kind of critical. For e.g. if the tcp_keepalive_time is set to a high value then the OS holds onto that port even after the transaction is done (SQL is processed and results are obtained).. This is not Ambari specific, as you can see.

If this was Ambari issue, there should be a much wider impact.

I recommend checking the TCP WAITTIME and other params on both, the source and destination servers.

Here are the recommended values -

net.ipv4.tcp_keepalive_time = 600

net.ipv4.tcp_keepalive_probes = 5

net.ipv4.tcp_keepalive_intvl = 15

[#3]

Note: Question moved to the Operations track.

[#4 · Accepted solution]

@bsaini i think i figured out the issue...it is a bug when we upgrade ambari database from Mysql 5.1 to later...as of now only option is to upgrade ambari to 2.2 or rollback the changes. thanks for taking time in answering my question...appreciate it!!!

https://na9.salesforce.com/articles/en_US/Issue/Ambari-log-reports-You-have-an-error-in-your-SQL-syn...