Subject: RE: Zmailer problem

From: "Luke Galea" <[email-masked]>
Date: Tue, 8 Oct 2002 11:09:55 -0400

Ya.. in this specific case that is what we plan on doing. My concern is
that our first indication that the queue was piling up was the max files
issue.. at one point we couldn't even log in. We were piling messages up
in the queue at about 25,000 / hour.. so by the time everyone got in in
the morning it was already time to boot from a floppy to disable zmailer
at start-up. Some sort of warning mechanism and a max simultaneous open
files limit in zmailer would fix this. The warning isn't a problem
(monitor mailq)... but the open files limit I am unsure of.

As for the multiple SMTP streams.. would setting maxTA to 2 for that
route accomplish this?

Thanks for your help.

Luke Galea 
Software Development
BlueCat Networks
905-762-5225

-----Original Message-----
From: Arnt Gulbrandsen [mailto:[email-masked]] 
Sent: October 8, 2002 11:02 AM
To: Luke Galea
Cc: Matti Aarnio; [email-masked]
Subject: Re: Zmailer problem

Luke Galea <[email-masked]>
> As far as I can tell.. it seems that as the queue size increases the
max
> files increases linearly.. so there doesn't appear to be a way I can
get
> zmailer to accept an infinite (or limited by disk space) queue. Is
there
> any way to limit the number of files the scheduler uses?

This sounds suspiciously like a "wrong question". There may be an
answer,
but the answer to the wrong question doesn't make anyone happy.

A large, unlimited queue size implies a lage, unlimited delay in mail
delivery, which is surely not desirable.

If I recall correctly, the basic problem is that you're handling mail
over
to a virus checker, and the virus checker is too slow to keep up with
your
mail volume. Couldn't you buy another of those and have zmailer deliver
to
both in parallel? Or have zmailer deliver to your one virus checker
using
two parallel SMTP streams?

--Arnt
-
To unsubscribe from this list: send the line "unsubscribe zmailer" in
the body of a message to [email-masked]

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 8 Oct 2002 17:32:57 +0200

> (monitor mailq)... but the open files limit I am unsure of.

Yes. Eating all FDs and dying is Bad. Personally I think it's a linux
problem, but maybe zmailer should work around it.

(linux appears to let one process open all the files, thereby denying all
services. Bad.)

> As for the multiple SMTP streams.. would setting maxTA to 2 for that

I don't think so. You need to use the same trick mea used with majordomo
on vger. Should be in the config examples somewhere.

--Arnt
-

---

From: Matti Aarnio <[email-masked]>
Date: Tue, 8 Oct 2002 18:36:55 +0300

> (monitor mailq)... but the open files limit I am unsure of.

  I think the issue is more in the UNIX (Linux) kernel you are using.
  I don't know why, but it seems to overflow the file hash cache, and
  be unable to flush unused ones from it.  Neither does it auto-tune
  this particular system feature...

  Scheduler process uses at most  "ulimit -H -n"  (for bash/sh)
  file descriptors itself, but the system can use a lot more
  internally.

  There increasing the hash-cache size can help you there.

  Oh, also do tech your   updatedb  NOT TO go into POSTOFFICE directory.
  That too appears to trigger this trouble in early morning hours...
  (At my home system anyway, I have kvazillion source directories
   in there, and updatedb has more than once induced death of the
   sshd daemon by running out of openable files... Then I did that
   file-max tuning, below, and the system behaves again.)

  I repeat my suggestion:

    # echo "12000" > /proc/sys/fs/file-max

  Set the value high enough so that your system stays happy.
  Monitor the actual value:

    # cat /proc/sys/fs/file-nr

  You are interested in the middle value.

  That system pseudo-directory has a few other monitor "files" you might
  find interesting to follow.

> As for the multiple SMTP streams.. would setting maxTA to 2 for that

  No.  MaxTA=2 cuts system performance radically down.
  You are looking for  maxTHR=2  in the channel where your virus scanner
  traffic is.  (You have made a channel/host selector pair of its own for
  that ?  Have you ?)

  Possibly MaxTHR=10  would do also.  (For 10 parallel streams..)
  (presuming the checker can gain speed from it.)

  In the current scheduler.conf  prototype file there is longish
  passage of comments telling about these things:

#  MaxTHR:   ( max processes per thread; default: 1 )
#       This limits the number of parallel transport agents within each
#       thread; that is, using higher value than default ``1'' will allow
#       running more than one TA for the jobs at the thread.
#
#       Do note that running more than one TA in parallel may also require
#       lowering OVERFEED value.  (E.g. having a queue of 30 messages will
#       not benefit from more TAs, unless they all get something to process.
#       Having OVERFEED per default at 150 will essentially feed whole queue
#       to one TA, others are not getting any.)

> BlueCat Networks

  /Matti Aarnio	<[email-masked]>
-
To unsubscribe from this list: send the line "unsubscribe zmailer" in
the body of a message to [email-masked]