SUMMARY: Tracking down system calls on Solaris 9

From: <mark_round_at_ipcmedia.com>
Date: Wed Sep 14 2005 - 05:57:00 EDT
Hi all,

Many thanks to everyone who responded - Aleksander Pavic, francisco, and
Darren Dunham. My original email is attached below, along with the
replies I got - but to summarise : I was seeing a very high sysload on a
Solaris 9 web server, and vmstat confirmed that a large number of system
calls were being generated. I wanted to track these down and find out
what was being called, but couldn't use Dtrace. Yet another argument for
moving to Solaris 10 :)

As Darren said in his response: "The limitations on existing tools like
'truss' are part of what drove
dtrace, so I don't know that there's any magic out there."

He then went on to suggest I analyse all the Apache processes with
truss, send the output to a file and then analyse that. This was also
the path suggested by Aleksander, who quite correctly pointed out that
truss can be made to follow any child processes generated via forking,
so I could therefore truss the main Apache process and follow all it's
children. He also suggested I send the output to a file, and
post-process it with awk or perl. Francisco also suggested the useful
lsof tool to see what files are open, as my original hypothesis was that
there were a large number of file handles being opened and closed.

In the end, I trussed every "httpd" process, and generated a summary
using "truss -c". I let this run for 20 seconds, and saw that there were
a very large number of resolvepath() and open() calls being generated,
roughly half of these calls returned with an error.

I then narrowed my search down, and examined what was actually being
passed as arguments to these calls. This is easily done with "truss -t
open,resolvepath". It turns out that a huge number of the
resolvepath()'s and open()'s were being generated by PHP scripts running
under Apache. They were using an inefficient include_path, and so when
most files were being included, PHP generated many resolvepath() and
open() calls which returned in error before finally finding the correct
location of the file.

We fixed the PHP include_path and also modified some of the scripts to
use an absolute path in include() or require() functions, and as
expected, the number of syscalls being generated halved itself.

There were a number of other code-related problems on that server as
well, but these were unrelated to my original request for help.

Once again, many thanks to those that responded. Problem resolved !

-Mark


---


Original post :

Hi all,

Apologies in advance for the somewhat open-ended question here.

I have a very busy web server, that is experiencing a large number of
system calls - a snip from vmstat 1 produces :

 kthr      memory            page            disk          faults
cpu
 r b w   swap  free  re  mf pi po fr de sr m1 m1 m1 m2   in   sy   cs us
sy id
 0 0 0 6793080 5648456 14 123 0 8  0  0  0  0  0  0  0 1052 70799 1308
51 25 23
 0 0 0 6800984 5650440 195 800 0 8 8  0  0  0  0  0  0 1334 49447 1476
38 23 39
 0 0 0 6808168 5651576 63 238 0 8  8  0  0  1  1  1  0 1225 49429 1686
35 26 40
 0 0 0 6807528 5652248 22 275 0 16 16 0  0  0  0  0  0 1040 41245 1252
46 16 38
 0 0 0 6815480 5654432 50 306 0 16 16 0  0  0  0  0  0 1114 47343 1410
37 21 42
 1 0 0 6825704 5657904 56 148 0 24 8  0  0  0  0  0  0 1388 68366 1648
53 34 13
 7 0 0 6825648 5657784 50 170 0 0  0  0  0  0  0  0  0 1240 53751 1781
42 17 41
 0 0 0 6825704 5657832 17 102 0 8  0  0  0  0  0  0  0  951 58447 1292
41 20 39
 11 0 0 6825704 5657808 33 137 0 0 0  0  0  0  0  0  0 1011 53959 1219
54 23 23
 6 0 0 6824168 5657184 41 173 0 32 8  0  0  1  1  1  0 1400 69368 1381
54 31 15

How can I find out what these calls are ? I have experimented with
lockstat, but this doesn't seem to provide me with the information I'm
after. I know there are many Dtrace scripts that would provide me with
the information I'm after - but I am on Solaris 9/Sparc (V240, 2x1.2Ghz,
8Gb RAM) so that is sadly not an option.  I'd ideally like something
like a system-wide "truss"; In an ideal world, I'd like to be able to
see most frequently called syscalls, and see what arguments are being
passed to them. For instance, I suspect that our web application is
opening a large number of files, but I have no way of proving this...

Any input would be much appreciated. I will summarise any responses I
get.



Responses :

[Darren Dunham]
The limitations on existing tools like 'truss' are part of what drove
dtrace, so I don't know that there's any magic out there.

You can certainly write a shell script that runs 'truss' on all the
server processes and sends output to files.  Then you have to correlate
the output files.  For a large number of 'truss'es, you're going to
affect performance on the box.

For now, that's the path I'd take.  Then you can post-process the output
files to summarize them.


[francisco]
Not an answer to your overall question, but to see what files are open,
lsof (not a standard Solaris command) comes in handy.  You can use that
per uid or process to see if a particular app has a lot of open files.


[Aleksander Pavic]
You can truss the main process of your WebServer. For example:
If you're using apache, then you have something like a >master< process
that spawns all other >slave< processes.
Now truss the main process (with -f)and truss will follow all child
processes. redirect STDERR to a file and analyse it with awk or perl.
-----------------------------------------------------------------------
This E-mail is from IPC Media Ltd whose registered office is at Kings
Reach Tower, Stamford Street, London SE1 9LS, registered number 53626.
The contents and any attachments to it include information that is
private and confidential and should only be read by those persons to
whom they are addressed. IPC Media accepts no liability for any loss or
damage suffered by any person arising from the use of this e-mail.
Neither IPC Media nor the sender accepts any responsibility for viruses
and it is your responsibility to check the email and attachments (if any).
No contracts may be concluded on behalf of IPC Media by means of e-mail
communications. If you have received this e-mail in error, please destroy
and delete the message from your computer. For unbeatable savings on
magazine subscriptions and great gift ideas visit www.giftmags.co.uk
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
Received on Wed Sep 14 05:57:47 2005

This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:43:51 EST