Platform Engineering

Appearance

Troubleshooting MySQL Too Many Connections ​

Problem ​

MySQL server is rejecting connections due to reaching the maximum connection limit, triggering "too many connections" alerts.

Procedure ​

Step 1: Identify Connection Sources ​

Log into the MySQL server and run the following query to identify which IP addresses are consuming the most connections:

sql
SELECT SUBSTRING_INDEX(HOST, ':', 1) AS ip, COUNT(*) AS connections 
FROM information_schema.PROCESSLIST 
GROUP BY ip 
ORDER BY connections DESC;

Step 2: Export Connection Data ​

Extract the IP addresses to a local file for further analysis:

bash
mysql <connection-args> -e "SELECT SUBSTRING_INDEX(HOST, ':', 1) AS ip, COUNT(*) AS connections FROM information_schema.PROCESSLIST GROUP BY ip ORDER BY connections DESC;" | awk '{print $1}'

Copy the output and save it to a local file named connections (one IP address per line). This file will be used in the next step to match IP addresses with pod information.

Step 3: Identify Problematic Pods ​

Use kubectl to find which pods correspond to the high-connection IP addresses:

bash
kubectl get pods -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,POD_IP:.status.podIP' \
  --no-headers | rg -Ff connections

This command will show you the pods that are consuming the most database connections.

Step 4: Restart Problematic Pods ​

Restart the identified pods to clear their connection pools:

bash
kubectl delete pod <pod-name> -n <namespace>

Verification ​

After restarting the pods, monitor the MySQL connection count to ensure it has returned to normal levels:

sql
SHOW STATUS LIKE 'Threads_connected';

Prevention ​