Re: ZMailer under load

[Matti Aarnio <mea@nic.funet.fi>]

On Mon, Dec 04, 2006 at 12:25:57PM +0100, Dawid Kuroczko wrote:
> Hello,
>   I'm writing this mail to broaden my knowledge and learn some neat tricks.
> I have a ZMailer cluster which generally works good but from time to time
> we are hit by a severe load and sometimes system grinds to a halt with
> queues over 800 000 mails and rising.

Which ZMailer version ?

Aside of that smelling of spammer attack (smtpserver input rate-limiter 
was developed to counter that effect for a part), is the scheduler falling
back seriously in the queue scanning ?

It can be looked like this:

 # mailq -sQQQ
 25 entries in /var/postoffice/router/ directory: processing
 1397 messages in /var/postoffice/transport/ directory: working
 Kids: 232  Idle: 38  Msgs: 1397  Thrds: 2082  Rcpnts: 27339  ...
 Msgs in 3544817 out 3532295 stored 1397 Rcpnts ...


If the 'Msgs .. stored ____ ..' value differs much of the second
line, then there is definite scanning contention.

If the numbers match, then the issue is something else.

That something else may be looked into with e.g.:
   mailq -Q | egrep 'R=[0-9]{3,9}'
which shows destination queues with 100 or more messages.

Running with too much in parallel (the Kids: ___  Idle: __ -values) is
also dubious.  The example above is from vger.kernel.org,  which is
happily humming along with about 230 transport agents limit while
it has 4000-5000 separate queues to push out.

It is all about the tuning of queue resources allocated for
each subset of addresses.
Command:

  mailq -QQ

shows you the resource allocation subsetting, how much they have queues
("Threads:") and how much there are processes ("Procs:") processing them.
Oh, and how much there are messages ("Msgs:").



>  The reason was I guess I/O contention.  The scheduler is competing
> for I/O with a bunch of happy other processes and loses.  I guess the
> simplest soultion is 'get more drives' (and it works), but I would
> like to know if I overlooked any other ways to help my system.
> 
>  I'm looking for any kind of hints, like ways to tuning Linux, or perhaps
> even switching to some other OS.  All replies appreciated.

"iostat -k -p 10" tells disk & partition IO activity.
No need to guess.


Split the  POSTOFFICE into two disk sets  ( /postoffice  and 
/postoffice/transport .)   Have third disk set for logs ( /var/log/ ),
and  fourth for the rest of the system.

Note: I didn't say "partition".  Those must be separate disks in order
to be able to run sufficiently in parallel.  Physical movement of disk-
heads does take time..

Run the disks on a hardware raid-controller(s) with battery-backed
write-cache. Preferrence order: RAID10, RAID5, RAID1.

Choose smallest capacity but fastest disks that you can find.
(Disk space is not the problem these days, number of spindles is..)

On Linux the most speedy filesystem for MTA use is probably
ReiserFS (v3).

Otherwise you will likely need additional spindle pair (RAID1) for
the external journal / log.  Possibly external battery-backed
RAM-disk ?  You can even run without journal -- oh, and choosing
carefully of what to journal and what not, that helps too.
(Some journal modes with EXT3 make it very slow..)

>  As a side note I was thinking about decoupling scheduler into two
> separate entities, one doing scheduling (mostly non-blocking) and
> the other doing queue scanning.  Would that be a good idea?

Probably not.  I did add several tricks for scheduler queue assimilation
speedup.  It does (very quickly) scan the entire directory tree into
sort of pre-queue, and then munches messages from there to the actual
queue.  Early versions did scan the directory + assimilate messages
in semi-synchronous manner causing considerable congestion at times.


>   Regards,
>      Dawid
-- 
/Matti Aarnio	<mea@nic.funet.fi>
-
To unsubscribe from this list: send the line "unsubscribe zmailer" in
the body of a message to majordomo@nic.funet.fi