Re: ZMailer's scheduler and Linux's INOTIFY?

[Matti Aarnio <mea@nic.funet.fi>]

On Mon, Aug 07, 2006 at 11:53:17AM +0200, Dawid Kuroczko wrote:
> Hello!
> 
> I've been thinking about a nice new feature of Linux 2.6.x, that
> is the INOTIFY interface.  Scheduler's job is to periodicaly scan
> whole a lot of directories (2^26+1 assuming SCHEDULERDIRHASH=2).

A wee bit less than that ..  mere: 1 + 26 + 26^2  ( 703 directories,
not  67 108 864 -- difference by a factor of about 100 000. )

Actually the directory scanning is a fallback, and primary is to
use the "notify-channel" (of AF_UNIX kind) to tell at first the
router of new files, and router(s) will then tell the scheduler.

At startups, and latter to handle possible missed/non-reported messages
the scanning will do checking of files in the directories every some
minutes (4 minutes, I think).

> The idea of INOTIFY is that instead of checking directories
> (or files) for changes periodically, one may inform the kernel
> about inodes its interested in (in scheduler's case, these would
> be queue subdirectories), and from time to time request
> a list of changes in these directories seen by the kernel.
> This saves a lot of readdirs, and may improve scheduler's
> scalability on heavily loaded (io-bound) systems.
> 
> http://www.linuxjournal.com/article/8478
> 
> Is it a thing worth pursuing?  How do you think?  Or maybe even
> someone has researched the problem already?

Recognizing that there are changes in some directory is only
half the work, apparently that will also inform of a change in
a given directory (along with actual file name) and there can
be watches in every directory in the sub-tree), but it still
does not remove the need to do occasional ordinary directory-tree
scans to pick things that have been missed for any number of
reasons.

It could replace AF_UNIX type notify-channel quite nicely.
It could even allow notification of message arrival for
submissions by means of NFS mounted $POSTOFFICE/ directory,
where AF_UNIX -type notify channel does not work at all.

So yes, implementing its use has some potential.


>   Regards,
>        Dawid

-- 
/Matti Aarnio	<mea@nic.funet.fi>
-
To unsubscribe from this list: send the line "unsubscribe zmailer" in
the body of a message to majordomo@nic.funet.fi