15 May

Why do threads sometimes stay in 'killed' state in MySQL?

Have you ever tried to kill a query, but rather than just go away, it remained among the running ones for an extended period of time? Or perhaps you have noticed some threads makred with killed showing up from time to time and not actually dying. What are these zombies? Why does MySQL sometimes seem to fail to terminate queries quickly? Is there any way to force the kill command to actually work instantaneously? This article sheds some light on it.

Read More
07 May

Running out of disk space on MySQL partition? A quick rescue.

No space left on device - this can happen to anyone. Sooner or later you may face the situation where a database either has already or is only minutes away from running out of disk space. What many people do in such cases, they just start looking for semi-random things to remove - perhaps a backup, a few older log files, or pretty much anything that seems redundant. However this means acting under a lot of stress and without much thinking, so it would be great if there was a possibility to avoid that. Often there is. Or what if there isn't anything to remove?

Read More
03 May

Interesting behavior of a MySQL benchmark on EC2

I had to benchmark an EC2 instance to see whether a database could be safely moved to it. It is a good practice, which helps avoiding surprises when an instance or its storage are allocated in a noisy neighborhood, where the neighbors use so much resources that it affects the performance of our MySQL database. It is understandable that one can never get very reliable results on EC2, this is a shared environment after all, and that some fluctuations should be expected, however it is still good to know the numbers. I started my benchmarks and everything seemed fine at first, but then sometimes statistics I was getting started looking quite odd.

Read More
01 May

The cost of improved security on a MySQL server

Security-Enhanced Linux or SELinux is a Linux kernel feature that provides a mechanism for supporting access control security policies. It enables a system administrator to create an extra set of rules that define allowed operations for programs even after the standard controls are checked. In other words, SELinux can help improving system security by restricting access of an application to only a few resources it actually needs, which makes it more difficult for an attacker to gain access to the entire system through exploiting any possible vulnerabilities in the application. However as rarely anything in life is free, is there any price we have to pay to use SELinux on a MySQL server?

Read More
23 Apr

Dedicated table for counters

There are a few ways to implement counters. Even though it's not a complex feature, often I see people having problems around it. This post describes how bad implementation can impact both application and MySQL performance and how to improve it. A customer asked me for help with performance problem they were facing. I logged into their database and found many client connections waiting for table locks. Almost all threads were stuck on one, small table called hits. What was the reason?

Read More