Some tips for finding performance issues in Linux

Some tips for finding performance issues in Linux

Sometimes we have some trouble with processes that demands lot of IO. There´s a great tool for that, iotop.

I executed fio, to generate some read stress:

mkdir -p /tmp/data ; fio –runtime=300 –time_based –name=random-read –rw=randread –size=128m –directory=/tmp/data

As you could see, fio is on the top of the iotop’s view, and the io is at 99%, displaying the DISK READ K/s.

If you don´t have iotop, you could do it with a little script. It´s not going to give you all that information that iotop shows, but is really good.

Basicly, this script shows the process in “D” State. Processes that are waiting for I/O are commonly in an “uninterruptible sleep” state or “D”; given this information we can simply find the processes that are constantly in a wait state.

cd /proc ; for pid in [0-9]* ; do awk ‘$2 == “D” {print “The process with PID: ‘${pid}’ is in ‘D’ State”;}’ $pid/status ; done

You could see some detailed information with iostat.

iostat -xdy 2 5 (x= Display extended statistics, d= Display the device utilization and y= Omit first report with statistics since system boot)

We see that, %util is very high. It´s very useful to see the r/s w/s (In this case the problem are the reads)

Sometimes we could reach the limit of “open files”.

Error: “Too many open files (24)”

You could see the total of files opened in the system with a simple shell script or lsof:

# for pid in /proc/[0-9]* ; do echo $(ls $pid/fd | wc -l) ; done | sort -n | awk ‘{ SUM += $1} END { print SUM }’

# lsof -Xn -a -d ^mem -d ^cwd -d ^rtd -d ^txt -d ^DEL | wc -l

If you need, to display the info, only for one user, you will need to pass the argument -u $USER to lsof

If you’ve overpassed the user limit, you will need to change this limit with:


Changing the limit for the user: (Edit your .profile and add it or change it
in limits.conf

* ulimit -Hn $NEW_LIMIT ($HOME/.profile)

* Or maybe you will need to do this change globally for all the users, 
editing /etc/sysctl.conf and modifying the value of fs.file-max = $NEW_GLOBAL_LIMIT

Leave a Reply