SUMMARY: (partial) NFS /var/mail; mail.local locking

From: Admin Staff (admin1@berkshire.net)
Date: Thu Jun 03 1999 - 08:05:08 CDT


Greetings Sun Managers,

Please accept my most sincere apologies for being late on this summary.
Been really busy ;-).

My original message is posted below for reference. Many, many thanks to:

Kevin Ying <kevin@ns1.interq.or.jp>
Benjamin Cline <benji@hnt.com>
Jochen Bern <bern@uni-trier.de>
Dale Shaw <Dale.Shaw@select.com.au>
David Lee <T.D.Lee@durham.ac.uk>
Aaron Holtz <aholtz@bright.net>

Special thanks to Adrian Otto, our performance consultant whom we hired to
help us get a grip on the various performance problems were experiencing
simultaneously.

I received many helpful replies from the above users, which I will post
below my original message below.

Our original problem (as you can see from my original post below) had to
with a performance problem we were having on a set of large mail servers
that share /var/mail over NFS. At the time of my post, we were running
Sendmail 8.9.1 with ipop3d spawned from inetd. We were also also having
performance problems with mail.local looping endlessly in contention for
mailbox locks when a given user received several messages that were
attempting delivery at the same time. Another concern was NFS performance
due to cacheing being turned off, as you'll see in my original message.
The status of the system at the time was were were experiencing high load
averages, tons of individual sendmail, mail.local and POP3 processes and
the result was many processes were blocked waiting for I/O at
peak times according to vmstat.

It turned out that running the POP3 daemon from inetd was a bad idea. To
get a picture of which of the two mail services (sendmail or pop3) was
consuming more resources, we split POP3 and SMTP between two machines
using our Cisco LocalDirector. It turned out, that with the exception of
periodic high sendmail loads and very occasional mail.local lock
contention, POP3 by far was using the most system resources. Since we
experienced a high average of short-lived POP3 connections, the ensuing
fork() and exec()'s from inetd caused a high load and increased paging
activity enormously. The solution was to get a multithreaded POP3
application.

The POP3 solution we chose was DPOP, part of the Dmail package available
from NetWin Software (http://www.netwinsite.com). For roughly $400 per
server, you can get both their multi-threaded SMTP (Dmail) and their POP3
(DPOP) servers with unlimited user licenses. Aside from being
multi-threaded, DPOP also supports custom external authentication (with
tunable cacheing), easy administration and installation, command line (cgi
ability, too) POP user addition (without requiring UNIX accounts) and
directory hashing support (Ie, /var/mail/u/s/user). It is totally awesome
and it fixed the great majority of our performance problems ;-).

NetWin's Dmail is also a highly configurable, mult-threaded MTA. We have
not installed it yet as we are in the process of doing more research
before abandoning sendmail.

We still have occasional problems with mail.local looping incessantly in
contention for a given user's mailbox when that user receives multiple
inbound messages at the same time. This is rare, but it happens
nonetheless. The number one recommendation I received to alleviate this
problem is to use procmail as the local delivery agent instead of
mail.local. Also, Aaron Holtz pointed out (see below) that it is possible
to achieve directory hashing with procmail, so this makes it even more
attractive for use in conjunction with DPOP's hashing support (10,000+
files in /var/mail has gotta be another big performance problem) . We will
probably install procmail as an intermediate step before migrating to
Dmail.

We also applied some minor sendmail tweaks to alleviate some problems due
to load spikes. Most notably, we limited the number of children spawned
from sendmail (Thanks David Lee for the reminder), and we set sendmail to
queue only when the load average on the machine approaches 60. Here's the
relevant m4 info for these tweaks:

define(`confQUEUE_LA',`60')
define(`confREFUSE_LA',`100')
define(`confMAX_DAEMON_CHILDREN',`120')

NB: This was set a bit high because I believe it effects all children,
including mail.local processes and what not. I'll have to refresh myself
on this one.

David Lee also recommended investigating the following value

    confCONNECTION_RATE_THROTTLE

        ... I remember investigating this in the past, but I don't
        remember why I opted not to use it. Maybe I fell asleep ;-).

With regards to my file locking concerns:

Jochen Bern also pointed out that another good reason for using Procmail
was due to its configurability in the area of file locking (see below).
Note to NetApp users: I may be wrong, but I believe the NetApp may need to
be explicitly configured to pass advisory locks. Jochen also points out
that flock() is local only on SunOS 4.x, but the Solaris 2.6 man page
points out that it has been fixed to work over NFS.

We're currently still using dotlocking which both DPOP and mail.local
support. We are investigating the potential for using flock() or lockf()
and possibly restoring attribute cacheing on our Netapp in an attempt to
curtail the getattr NFS calls that have dominated up to 70% of our NFS
calls in the past. At a minimum, we should be able to turn the cache back
on with dotlocking.

Thanks to all who have replied. Below my original post, please see other
helpful advice provided by the respondents of my original post.

Sincerely,

Jason Hatch
SysAdmin for many places ;-).

---------- Forwarded message ----------
Date: Tue, 16 Feb 1999 09:52:48 -0500 (EST)
From: Admin Staff <admin1@berkshire.net>
To: Sun Managers <sun-managers@sunmanagers.ececs.uc.edu>
Subject: NFS /var/mail; mail.local locking

Greetings Sun Managers,

We are running two SMTP/POP3 mail servers (Ultra10 clones, 300mhz, 1 gig
ram, lotsa swap, Solaris 2.6 patched) that are load balanced with a Cisco
Local Director 410. We are running Sendmail 8.9.1 configured for
mail.local, and for POP3, we're running ipop3d. We wanted CuciPop for it's
NFS awareness and general hackability, but could not get it to compile and
ipop3d was substituted in a pinch. The mail spool is provided by a NetApp
F630. I am having some performance problems, particularly with mail.local
and the contention for file locks.

When we researched running multiple mail servers that share the same
/var/mail (private mqueue) over NFS, we knew that file locking would be a
concern. We researched the Sun Managers archives and FAQs and found that
the general NFS mount options were as follows:

netapp:/export/mail - /var/mail nfs - yes
-O,noac,actimeo=0

Basically, the key was "noac" for no attribute cacheing, and "actime=0",
which is probably a bit redundant. We basically didn't want attribute or
write cacheing enabled. This ensures immediate updates of all information:
file attributes (access and modification times, advisory locking flags,
etc) and writes being immediately available.

The end result is that file locking works well, too well.

One of the immediate things we noticed was mail.local's behavior.
Mail.local, from what we could see with truss, achieves file locking by
writing a lock file (mailbox.lock, I think). If more than one mail.local
process needs to write to the same mailbox, we have a small party going on
that rapidly consumes the machine's resources, like a bunch of lions
fighting over the same piece of meat. It doesn't matter much if these
processes are running on one or both mail server, they seem to behave the
same, in that they are both acting on the NetApp filesystem.

Watching truss, each process is very active, checking for the lock several
times a second. There appears to be no random backoff period, just a bunch
of processes looping as fast as they can until by chance, one of them gets
the prize. Eventually they do, but depending on how many messages are
being delivered to that one user, a mail.local processes could loop for up
to 5 minutes before they score the lock.

Our mail servers, on average, handle about 20 inbound SMTP connections at
any time. I've seen this as high as 300 between 2 servers. What kills us
is not the sendmail processes, or the mail.local processes delivering to
different users, or the POP3 processes. It's when one user gets a ton of
mail at the same time. Woe to us if that mailbox is locked by the POP3
daemon, as the mail.local processes will mercilessly loop until the
mailbox becomes available. To reiterate, we could have 100 mail.local
processes delivering to say 60 users fine, but when we get 100 mail.local
processes delivering to the SAME user, the fireworks fly.

I tested this out by sending myself 100 messages with a simple perl
script. The LocalDirector landed the messages evenly between each mail
server, about 50 each. Each message did arrive (obviously not in order),
but what I witnessed was a feeding frenzy of mail.local processes on the
machines, lasting about 5 minutes and driving the load average above 60.
Once my 100 messages were delivered, the load average dropped to a nice
low level, and responsiveness improved.

I first noticed this when someone ran a KamaKaze mail script on one of our
web servers. This generated a ton of spam being sent from the owner of the
CGI process, www. The lamer had about 1.5 million e-mail addresses in his
possession. The mail was being delivered from the web server, and didn't
really result in a higher than usual load on that machine (in that it's
usually slammed). What alerted is to the whole deal was the BOUNCE
messages coming into user www, some 2000 an hour on the mail servers. The
mail.local contention for the "www" mailbox lock killed our servers and we
eventually had to alias www to /dev/null to preserve resources, as bounces
seemed to be coming in for quite some time after we located and terminated
the user.

Another general side affect we noticed are in NFS statistics:

Version 3: (261887628 calls)
null getattr setattr lookup access
0 0% 173244940 66% 6540510 2% 24642192 9% 10440793 3%
readlink read write create mkdir
484 0% 26472926 10% 3713090 1% 2418652 0% 4251 0%
symlink mknod remove rmdir rename
2238 0% 0 0% 4522609 1% 243 0% 7007 0%
link readdir readdirplus fsstat fsinfo
2193701 0% 752 0% 23004 0% 7660002 2% 171 0%
pathconf commit
63 0% 0 0%

Note how 66% of the traffic is used for "getattr". This statistic is
pretty stable: between 60% and 69%, with these statistics representing
about 18 days worth of traffic. The mount_nfs man page warns of this when
"noac" is used to mount a file system.

One of the things I noticed is that mail.local and ipop3d use the same
lock file mechanism for mailbox locking. This is nice, but the contention
is both time and resource consuming.

So I guess I have a few questions ;-)

        1) Is there a local mailer I could use that has better contention
           (ie, backoff and retry) mechanisms than mail.local?
        2) Is there a better way I could achieve load balanced mail, with
           SMTP and POP3 coming into multiple machines sharing the same
           mailboxes, even if drastic change is required?
        3) Would anyone care to share their methods used to achieving
           this?

In exchange for some BIG JUJU, I promise to write a detailed summary to
this list :-)

-Jason

Responses are below:

>From benji@hnt.com Thu Jun 3 08:43:33 1999
Date: Tue, 16 Feb 1999 11:13:31 -0500 (EST)
From: Benjamin Cline <benji@hnt.com>
To: Admin Staff <admin1@berkshire.net>
Subject: Re: NFS /var/mail; mail.local locking

Just a thought, maybe try using procmail as the local mailer?

        benji

--
Benjamin R. Cline       Harrison & Troxell, Inc.         benji@hnt.com
                     Quis Custodiet Ipsos Custodes?

>From bern@penthesilea.uni-trier.de Thu Jun 3 08:43:33 1999 Date: Tue, 16 Feb 1999 18:18:30 +0100 (MET) From: Jochen Bern <bern@penthesilea.uni-trier.de> Reply-To: bern@uni-trier.de To: admin1@berkshire.net Subject: Re: NFS /var/mail; mail.local locking

> One of the things I noticed is that mail.local and ipop3d use the same > lock file mechanism for mailbox locking. This is nice, but the contention > is both time and resource consuming.

Yet, it is pretty much unavoidable. POP daemons, especially those that come out of a combined popd/imapd package, need to add an ID number to the headers of every new mail (usually this header is named "X-UIDL:" or somesuch), which means that the point where mail.local could start appending to the mail spool keeps moving.

> 1) Is there a local mailer I could use that has better contention > (ie, backoff and retry) mechanisms than mail.local?

I wholeheartedly recommend procmail as local mailer, the major reasons being a) versatility (~/.procmailrc) and b) ability to deal with disk quotas. I didn't inspect the sources, and it's been a while since I installed it, but if I'm not very much mistaken, there is a load of compile-time options for file locking ...

| penthesilea:/home/TI/bern% procmail -v | procmail v3.11pre3 1995/05/17 written and created by Stephen R. van den Berg | berg@pool.informatik.rwth-aachen.de | | Submit questions/answers to the procmail-related mailinglist by sending to: | procmail@informatik.rwth-aachen.de | | And of course, subscription and information requests for this list to: | procmail-request@informatik.rwth-aachen.de | | Locking strategies: dotlocking, flock() | Default rcfile: $HOME/.procmailrc | System mailbox: /usr/spool/mail/$LOGNAME

> 2) Is there a better way I could achieve load balanced mail, with > SMTP and POP3 coming into multiple machines sharing the same > mailboxes, even if drastic change is required?

Pure theory on my part: If you could set up sendmail so that incoming mail *even for local users* would only get queued, to be delivered by queue runs from cron, you'ld have pretty much a guarantee that there will be only *one* instance of the local mailer at any point; Having that one time out sufficiently fast should keep your system basically alive (at the cost of a growing mail queue, if deadlock strikes).

(Variation: Leaving the sendmail setup unchanged while setting a short timeout for the local mailer would still solve the problem as long as SMTP connections to the victimized mailbox don't come in faster than the timeout.)

Really nasty quick-and-dirty idea: Choose a limit for the number of local mailers you want to have waiting for a specific mailbox, check the actual numbers from a cron job regularly, and automatically add/ remove aliases for "overloaded" mailboxes that will communicate a temporary delivery error back to sendmail (which should cause sendmail to have the mail enter the queue).

Regards, J. Bern -- /\ /""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""\ / \/ bern@uni-trier.de/bern@ti.uni-trier.de| P.O. box 1203 | Ham: \/\ / J. \ (Accepting PGP, MIME, SUNAttachments) | D-54202 Trier | DD0KZ/ \ \Bern/ finger bern@informatik.uni-trier.de | Email autoreply \ / \ /\ http://www.informatik.uni-trier.de/~bern/ | on subject '##' /\/ \/ \____________________________________________________________/

>From Dale.Shaw@select.com.au Thu Jun 3 08:43:33 1999 Date: Wed, 17 Feb 1999 09:25:30 +1000 From: Dale Shaw <Dale.Shaw@select.com.au> To: Admin Staff <admin1@berkshire.net> Subject: Re: NFS /var/mail; mail.local locking

Jason,

I have not researched this at all but you might like to take a look at procmail as an alternative to mail.local.

See the README in the sendmail dist for the required .mc setting (I think it's FEATURE(local_procmail) or something.

I think you can get it from ftp.informatik.rwth-aachen.de/pub/packages/procmail/

I believe 3.11pre7 is the latest version.

cheers, Dale.

>From T.D.Lee@durham.ac.uk Thu Jun 3 08:43:33 1999 Date: Wed, 17 Feb 1999 15:14:17 +0000 (GMT) From: David Lee <T.D.Lee@durham.ac.uk> To: Admin Staff <admin1@berkshire.net> Subject: Re: NFS /var/mail; mail.local locking

On Tue, 16 Feb 1999, Admin Staff wrote:

> [...] > > Our mail servers, on average, handle about 20 inbound SMTP connections at > any time. I've seen this as high as 300 between 2 servers. What kills us > is not the sendmail processes, or the mail.local processes delivering to > different users, or the POP3 processes. It's when one user gets a ton of > mail at the same time. Woe to us if that mailbox is locked by the POP3 > daemon, as the mail.local processes will mercilessly loop until the > mailbox becomes available. To reiterate, we could have 100 mail.local > processes delivering to say 60 users fine, but when we get 100 mail.local > processes delivering to the SAME user, the fireworks fly.

Our system is somewhat similar to yours (but our mailspool machines are simply Solaris 2.6 direct, not indirect to a NetApp).

We occasionally (once every few months) see something similar (and indeed we had such an incident today after your message arrived!).

You seem to have looked far deeper into it that I have. I don't have any proven answers, sadly. So please summarise.

But after this morning's incident, I read through the cf/README file in the sendmail 8.9.x source and noted: confMAX_DAEMON_CHILDREN and: confCONNECTION_RATE_THROTTLE

These look worthy of further investigation (even though they are somewhat generalised for handling this per-user type of problem).

> [...] > One of the things I noticed is that mail.local and ipop3d use the same > lock file mechanism for mailbox locking. This is nice, but the contention > is both time and resource consuming.

Nice? Essential I think: both a POP process and an incoming sendmail (mail.local) process may wish to update the file simultaneously. Indeed, a few years ago we had occasional reports of mail going missing: not often enough to be able to debug, but often enough to worry us that the effect was genuine. A subsequent routine POP daemon upgrade mentioned a bug-fix for faulty locking: the reports vanished.

> So I guess I have a few questions ;-) > > 1) Is there a local mailer I could use that has better contention > (ie, backoff and retry) mechanisms than mail.local?

The "conf..." options above may help. (Let me know if they do!)

> 2) Is there a better way I could achieve load balanced mail, with > SMTP and POP3 coming into multiple machines sharing the same > mailboxes, even if drastic change is required?

The received wisdom appears to be "never do NFS locking with e-mail, always do it directly within the machine". "Don't believe claims about perfect NFS locking". Not sure how this relates to the (perhaps slightly different?) case of a NetApp spool.

> 3) Would anyone care to share their methods used to achieving > this?

I would if I had anything to share!

> In exchange for some BIG JUJU, I promise to write a detailed summary to > this list :-)

Yes, please!

Incidentally, are you aware of IMAP (second generation POP)? Many mail user agents now use IMAP4 instead of POP (Pine, latest Netscape, ...).

--

: David Lee I.T. Service : : Systems Programmer Computer Centre : : University of Durham : : Phone: +44 191 374 2882 (ddi) South Road : : Fax: +44 191 374 7759 Durham : : Internet: T.D.Lee@durham.ac.uk U.K. :

>From kevin@ns1.interq.or.jp Thu Jun 3 08:43:33 1999 Date: Wed, 24 Feb 1999 12:11:04 +0900 (JST) From: Kevin Ying <kevin@ns1.interq.or.jp> To: Admin Staff <admin1@berkshire.net> Subject: Re: NFS /var/mail; mail.local locking

On Tue, 23 Feb 1999, Admin Staff wrote:

> Thank you for your response. I'm looking forward to your thoughts. > > On Wed, 17 Feb 1999, Kevin Ying wrote: > > > Jason, we have a very similar configuration, and I wanted to let you know > > that I'd be responding with a detailed response and comments... (doing > > morning e-mail over here). > We're currently using the NetApp F630 Fibre Channel filer. We have two shelves of 14 9GB drives. 12 used for space, 1 parity, and 1 hot spare. So the total capacity is around 100GB. The NetApp holds user data for both mail and web, as well as our /usr/local software.

Then, we have clusters of UltraSPARC AXi servers. We chose the SME AXi servers because of their lower cost than the Sun-branded Ultra5/10/30/60s...

I have 2 dedicated servers that handle incoming mail, 2 dedicated servers for outgoing mail and POP3 service, 3 web servers for pages, 3 cgi servers for user cgis (using CGIWrap so we don't have to audit any CGIs). I plan to add a 3rd incoming mail and outgoing mail/pop server in the next month or so.

We export our /etc/passwd and /etc/shadow files to all the servers that need them for user services. They are mounted read-only to ensure that no changes occur on our generic front-end servers. It's been working pretty well with no user complaints about speed. I guess that's a testament to NetApp's filer speed and Sun's nscd caching.

For the MTA, we are continuing to use sendmail. It's the MTA with which we are most familiar and the access features make it easy to get rid of spammers. We use procmail for local delivery that's been patched to use the Maildir delivery format for safe NFS locks. Truly NFS safe because each message is a separate file. It's murder on inodes but as long as you manage it, it's OK.

For POP3, we use qmail-pop3d. It's a really small binary that runs out of inetd and although I'd like a standalone version eventually, it's working OK and the speed is sufficient.

Anything else I missed?

>From bern@penthesilea.uni-trier.de Thu Jun 3 08:43:33 1999 Date: Wed, 24 Feb 1999 20:18:45 +0100 (MET) From: Jochen Bern <bern@penthesilea.uni-trier.de> Reply-To: bern@uni-trier.de To: admin1@berkshire.net Subject: Re: NFS /var/mail; mail.local locking

> I think the locking I was witnessing from the POP3 daemon was a file lock > in the format of user.lock. This is the same file that mail.local looks > for.

I don't have an authoritative source for that, but I think that everything locking a mail spool SHOULD (RFC-ese for "must, or you're outta business" ;-) do that to claim a lock, and SHOULD respect other locks in ANY form it recognizes, even if there's only one lock set. That way, you have the user.lock file pretty much everything should recognize, but yet may use another locking mechanism (*) for "usual situations".

(*) For example, I think that user.lock locking has a window of overriding - process A checks, sees no lock, process B checks, sees no lock, A puts the lock, B puts another OVERRIDING A's LOCK - which is not the case with kernel-based locking mechanisms.

> Just out of curiosity, what POP3 daemon are you using?

None. :-) I have a working NFS-mount-dat-mail-spool-dir-everywhere (no prominently Windows based users) installation - which is greatly helped by the fact that the author of procmail took great pains do make the file locking both NFS- and mailtool-proof, though the latter needs a special quirk at compile time - and though I've been playing around with popd's to fulfill a special user's requests, that guy ended up running IMAP against a Netscape Messaging Server at another site (no funds to buy that s/w for *this* site - luckily).

> would flock() work over NFS, or is it system function?

>From the flock() manpage (SunOS 4.1.3_U1B):

BUGS Locks obtained through the flock() mechanism are known only within the system on which they were placed. Thus, multiple clients may successfully acquire exclusive locks on the same remote file. If this behavior is not explicitly desired, the fcntl(2V) or lockf(3) system calls should be used instead; these make use of the services of the network lock manager (see lockd(8C)).

(Note, however, that we found that Linux boxes do not have, and possibly won't respect, a lockd for fileservers.)

> Hmmm... I never thought of this... So, when you start a sendmail process > just to process the queue, it will spawn one mail.local process max?

Not 100% sure about that, however, I *am* sure that it will process one mail at a time, which at the very least limits the number of local mailers to the number of local recipients of a piece of mail. (Aren't there sendmail.cf mailer flags to control concurrency? I can't check offhand, I'm offsite today. I'm pretty sure that there's a mailer flag controlling whether a mailer can send to several recipients in one run, not_quite_there (tm) but close.)

Regards, J. Bern -- /\ /""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""\ / \/ bern@uni-trier.de/bern@ti.uni-trier.de| P.O. box 1203 | Ham: \/\ / J. \ (Accepting PGP, MIME, SUNAttachments) | D-54202 Trier | DD0KZ/ \ \Bern/ finger bern@informatik.uni-trier.de | Email autoreply \ / \ /\ http://www.informatik.uni-trier.de/~bern/ | on subject '##' /\/ \/ \____________________________________________________________/

>From aholtz@bright.net Thu Jun 3 08:43:33 1999 Date: Sun, 28 Feb 1999 15:10:45 -0500 (EST) From: Aaron Holtz <aholtz@bright.net> To: Admin Staff <admin1@berkshire.net> Subject: Re: NFS /var/mail; mail.local locking

Jason,

I was searching through the newsgroups and saw this posting. I was quite intrigued by it as I have a similiar situation to contend with. We have 65,000 dialup customers that do an inordinate amount of mail. Our mail machine takes about 400,000 connections per day, if not more. Our current setup:

Ultra Sparc II dual 300mhz processors 2gb RAM CMD 5500 Hardware RAID controller 90gb disk space

/var/mail is a separate wide scsi controller into the sparc. Our mail spool is hashed at /var/mail/letter1/letter2 where the letters are the first two fields of the username. We are using sendmail 8.9.3 and procmail for local delivery (much nicer than mail.local from what I've seen.) We are using cucipop for our popclient (same guy who wrote procmail wrote this.) However, we are getting seriously CPU bound in the evenings. Disk I/O is great even during the horrible peak hours. My solution was this:

Cisco local Director Several Ultra sparcs that nfs mount /var/mail from a central server Eventually get a NetApp unit to be that NFS central server.

My concerns regarding locking appear to be the same as yours. From what I've seen, by using cucipop and procmail compiled to ONLY do dotlocking (you can choose from lockf, flock and fnctl methods too), then you are safe in an NFS environment. From what the author claims regarding his dotlocking method, that it is much less overhead than say what qpopper does. You seem to already be doing almost all of this (except for the procmail/cucipop combo.) How is your performance during peak hours? About how much mail do you do in a day? I'm curious about how cucipop and procmail would play together over NFS using dotlocking. Again, from what I've read, it should be very good. I'm trying to create a solution that will last us for quite a while - the above method looks to be very scalable. I'm just wondering if NFS mounting /var/mail is the answer - I really can't see another way of doing it efficiently. I have read about using maildir instead of mbox format (maildir is where each user gets a directory for new and old mail - each message is then stored in a separate file instead of all in one file like mbox.) The big benefit of maildir is that no locking has to take place since each message is in a separate file, so it is NFS friendly. But there is a big change to take place to get existing mail to that format as well as hacking software to support it (not to mention the extra inodes and disk space used up....) It appears you have a Netapp - can I ask which one and whether you are happy with it or not? Maybe we can come to some conclusions and solutions together. Hope this helps.

-------------------------------------------------------------------------- Aaron Holtz ComNet Inc. Manager, Unix Systems Administration Email: aholtz@bright.net "It's not broken, it just lacks duct tape." --------------------------------------------------------------------------



This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:13:20 CDT