Do you have a question? Post it now! No Registration Necessary. Now with pictures!
- Posted on
- Tracking Down Speed Issues
- Jason Carlton
February 5, 2010, 9:15 am
rate this thread
This is an extension of my earlier thread, "Images served on separate
server?" We had really stopped talking about images, and were delving
into speed problems that were the root of the issue.
I installed Munin to keep up with my issue, but now I don't know what
to do with my knowledge. I see that it peaked from 7pm until around
11pm EST... but I already knew that! LOL
The CPU load went up to 61, out of a possible 400. This spike wasn't
during peak hours, though, and during my slow time, the CPU load never
went over around 15. So, this looks OK.
Load Average had a peak of 0.57. With 4 CPUs, this looks OK, too.
Inode Usage stayed consistent all day. I don't know what this is, but
it seems OK.
MySQL Queries ranged from 8 to 11 queries / second during the slow
period. The graph makes it look like the majority of these were
through cache_hits. Is this good or bad?
MySQL Slow Queries shows 0, so this seems OK.
MySQL Threads ranges from 2-5 throughout the day, but then during the
slow period peaks up to 9-10 several times. What does this mean?
Netstat shows a max of 274.73, and outlines when the server was slow,
but I don't know what that means. Does this show that we had 274
server requests / second during that time?
Number of Processes is similar to Netstat, but it reaches 280 during
our slow time and then just plateaus there for several hours. Does
this mean that the server peaked on the number of processes per second
that it could handle?
Last but not least, the chart that looks drastic is Memory Usage, but
unfortunately, I have no idea what this means and the title is so
vague that I can't find a lot of info about it. I have this dark blue
box in the back of the chart, showing a maximum "page_tables" of 2GB.
During the entire slow period, "committed" goes well over this max,
and reaches 2.26GB.
So, what does all this mean? I have no idea. Is it telling me that I
should increase page_tables, or just warning me that I've reached the
max my hardware will allow? Should I do something to increase the
number of processes that it can handle (or lower the number of
processes sent), or again, is it just a warning that I've reached a
Re: Tracking Down Speed Issues
On Fri, 5 Feb 2010 01:15:40 -0800 (PST), Jason Carlton wrote in
Creative snipping ensues.
I really don't know what all that above means, but hopefully by now maybe
you're starting to realize that there's so much you don't know, that if the
info you're getting tells you no story you understand, it may just be time
to get someone who can read your story.
When I don't know something, I hire someone who does and then move on.
You shouldn't be learning on a production platform.
Mama say, everybody little bit gay.
Re: Tracking Down Speed Issues
A few updates for those reading in the future...
The Apache configuration had MaxClients set at 150, and I'm guessing
that this setting was forcing the limit for "Number of Processes".
Once it reached 150, processes were going into memory while waiting
for a free spot.
I increased this to 296 (my ServerLimit max), and it had an immediate
impact. "Number of Processes" jumped up to 368 within seconds, but the
server speed became almost normal.
"Memory Usage" apparently refers to my RAM. Why they didn't just call
it RAM, I don't know, but it does mean that I'm reaching the max that
my hardware can handle.
It looks like my options here are (A) buy more RAM, or (B) find the
memory hog. I suspect Apache, so I'll probably install lighttpd before
buying more RAM.
Re: Tracking Down Speed Issues
I don't know what Munin is, but that I don't know is probably
One thing people having performance issues tend to do is add a lot of
performance-monitoring code that makes everything even slower, which
can be good if it helps find and resolve a bottleneck, bad if the
monitoring stuff is forgotten and left to continure holding the system
back when it's no longer needed.
So whatever your problem is does not seem to be related to processor
As I understand it, inodes are the low-level filesystem blocks that
contain the actual information about files, there's one per real file
and each is kind of like the root of the tree that has all the
symbolic links on it. Without knowing what kind of usage they're
talking about it's difficult to interpret consistency. "Usage" may
refer only to the number of inodes referenced per unit time, but it
doesn't say whether all the references are to a few files or many, the
same files or different ones, it tells nothing about the average size
of read or write requests, time spent waiting for the disk, etc etc.
I think it can be considered a very gross overview of "filesystem
activity". Anyway if "inode usage" is consistent and apparent
performance is not, the slow spots are probably unrelated to the
overall filesystem activity level.
Your operating system (is it unix or linux?) probably handles paging
on a very low level basis using fixed disk areas or stored sector
addresses so that paging would not affect inode usage after system
boot. I would definitely give system-provided paging statistics a
higher level of credence than anything deduced from inode activity.
Cache hits are always better than disk reads, unless caching affects
paging. It's time spent waiting on physical devices that hurts
performance, no matter what causes the waiting.
I'd guess the number of threads is barely relevant, but the emphasis
there is "guess". I know nothing about MySQL except that it's a large
complex black box that I can do without, and by doing without it what
I'm also doing without is any problems caused by upgrades, version
interactions with other components, etc. I know it's commonly used,
and mileage varies. Plenty of folks here in AWW know a lot about it,
I'm not one of them.
It isn't so much the number of processes that's important as what
they're for. If you don't know what they're doing, it doesn't tell
much. They could all be sitting on page storage waiting for work, or
they could all be trying to access the same disk sectors, you just
don't know from a simple number-of-processes.
Sorry, I don't recall your mentioning whether you're on a shared or
dedicated server. If you're on a shared server the issue might not be
yours at all.
No matter how much data you accumulate, or what the gizmos tell you,
keep in mind the basic problem. I didn't pay that much attention to
the previous thread. What is the basic problem? Slow page delivery?
Do you have any statistics on the duration of each page's generation?
Delivery includes generation and transmission. The network is part of
no aluminum siding offers today