11.1. Message Submission Areas

In few places inside of ZMailer (in parts of router, and more so in parts of scheduler) the system expects the filenames to be decimal encodings of integers of 31 bits (maybe 63 bits at systems with suitably large 'long'), and those integers (modulo something) are used as keys in several internal database lookups.

The numeric values used in filenames must be unique for the entire lifetime of the spool files.

Message submission is done by writing a temporary file into the directory ($POSTOFFICE/public/), the actual format of the submitted message is described in Appendix E.

When the temporary file is completely written, flushed to disk, and closed, it is then renamed into one of the router input directories, usually into $POSTOFFICE/router/ with a name that is a decimal representation of the spool-file i-node number. This is a way to ensure that the name of the file in the $POSTOFFICE/router/ directory is unique.

The message may also be renamed into alternate router directories, which give lower priorities on which messages to process when.

Sometimes, especially smtpserver built files may be moved into alternate directories. The smtpserver “ETRN” command has two implementations, original one is by moving the built special file to the directory $POSTOFFICE/transport/ without going through the router. The smtpserver may also move newly arrived files into the $POSTOFFICE/freezer/ directory.

11.2. Router Behaviour on Queues

FIXME:
   This description is from era before the router got
   “daemonized” in a sense of having separate instance
   of queue processor (and it also handles logging/log
   rotation) and a worker-farm of routing work processes.

FIXME: The system can have multiple router processes running in parallel and competeting on input files. Multiple processes may make sense when there are multiple processors in the system allowing running them in parallel, but also perhaps for handling cases where one process is handling routing of some large list, and other (hopefully) will get less costly jobs.

The router processes have a few different behaviours when they go over their input directories.

First of all, if there are ROUTERDIRS entries, those are scanned for processing after the primary $POSTOFFICE/router/ directory is found empty.

Within each directory, the router will sort files at first into mod-time order, and then process the oldest message first. (Unless the router has been started with the “-s” option.)

The router acquires a lock on the message (spool file) by means of renaming the file from its previous name to a name created with formatting statement:

  sprintf(buf, "%ld-%d", (long)filestat.st_ino, (int)getpid());

Once the router has been able to acquire a new name for the file, it starts off by creating a temporary file of router routing decisions. The file has a name created with formatting statement:

  sprintf(buf, "..%ld-%d", (long)filestat.st_ino, (int)getpid());

Once the processing has completed successfully, the original input file is moved into the directory $POSTOFFICE/queue/, and the router produced scheduler work-specification file is moved to the $POSTOFFICE/transports/ directory with the same name that the original file has.

If the routing analysis encountered problems, the message may end up moved into the directory $POSTOFFICE/deferred/, from which the command zmailer resubmit is needed to return the messages into processing (The router logs should be consulted for the reason why the message ended up in the /deferred/ area, especially if the command zmailer resubmit is not able to get the messages processed successfully and the files end up back in the /deferred/ area.)

If the original message had errors in its RFC-822 compliance, or some other detail, a copy of the message may end up in the directory $POSTOFFICE/postman/.

See Postmaster Analysis Area on section Section 11.4.

11.3. Scheduler, and Transport Agents

The scheduler work specification files are in the directory $POSTOFFICE/transport/, under which there can be (optionally) one or two levels of subdirectories into which the actual work files will be scattered to lessen the sizes of individual directories in which files reside, and thus to speed up the system implied directory lookups at the transport agents, when they open files, (and also in the scheduler).

When the router has completed message file processing, it places the resulting files into the top level directory of the scheduler; $POSTOFFICE/transport/, and $POSTOFFICE/queue/.

The scheduler (if so configured by “-H” option) will move the messages into “hashed subdirectories,” when it finds the new work specification files, and then start processing them.

The transport agents are run with their CWD in directory $POSTOFFICE/transport/, and they open files relative to that location. Actual message bodies, when needed, are opened with the path prefix “../queue/” to the work specification file name.

Usually it is the transport agent's duty to log different permanent status reports (failures, successes) into the end of the work-specification file. Sometimes the scheduler also logs something at the end of this file. All such operations are attempted without any sort of explicit locking, instead trusting the write(2) system call to behave in an atomic manner while writing to the disk file, and having a single buffer of data to write.

Once the scheduler has had all message recipient addresses processed by the transport agents, it will handle possible diagnostics on the message, and finally it will remove the original spool-file from the $POSTOFFICE/queue/, and the work-specification file from $POSTOFFICE/transport/.

11.4. Postmaster Analysis Area

If the filename in the $POSTOFFICE/postman/ directory has an underscore in it, the reason for the copy is soft, that is, the message has been sent through successfully in spite of being copied into the directory.

If the filename in that directory does not have an underscrore in it, that file has not been processed successfully, and the only copy of the message is now in that directory!

Usually forementioned underscoreless filenames are double-errors, that is, error messages to error messages. There is nowhere else to send them.

The indication of error message is, of course, MAIL FROM:<> per RFC 821.

If the smtpserver receives a message with content that the policy filtering system decides to be dubious, it can move the message into $POSTOFFICE/freezer/ directory with a bit explanatory name of type:

  sprintf(buf, "%ld.%s", (long)filestat.st_ino, causecodestring);

The files in the freezer-area are in the input format to the router, and as of this writing, there are no tools to automatically process them for obvious spams, and leave just those that were falsely triggered.

12.1. smtpserver.conf

If the system has file $MAILSHARE/smtpserver.conf (by default), that file is read for various parameters, which can override most of those possibly issued at the command line.

Example configuration is is in figure Figure 12-2.

#PARAM maxsize          10000000    # Same as -M -option
#PARAM max-error-recipients    3    # More than this is propably SPAM!
#PARAM MaxSameIpSource        10    # Max simultaneous connections from
#                                   # any IP source address
#PARAM ListenQueueSize        10    # listen(2) parameter
#
# Enables of some commands:
#PARAM  DEBUGcmd
#PARAM  EXPNcmd
#PARAM  VRFYcmd
#PARAM  enable-router # This is a security decission for you.
#                   # This is needed for EXPN/VRFY and interactive
#                   # processing of MAIL FROM and RCPT TO addresses.
#                   # However it also may allow external user entrance
#                   # to ZMailer router shell environment with suitably
#                   # pervert input, if quotation rules are broken in
#                   # the scripts.
PARAM help -------------------------------------------------------------
PARAM help  This mail-server is at Yoyodyne Propulsion Inc.
PARAM help  Our telephone number is: +1-234-567-8900, and
....
PARAM help -------------------------------------------------------------

# The policy database:  (NOTE: See  'makedb'  for its default suffixes!)
PARAM  policydb  $DBTYPE  $MAILVAR/db/smtp-policy

# External program for received message content analysis:
#PARAM  contentfilter   $MAILBIN/smtp-contentfilter
....
#
# TLSv1/SSLv[23] parameters; all must be used for the system to work!
#
# See  doc/guides/openssl,  or:
# http://www.aet.tu-cottbus.de/personen/jaenicke/pfixtls/doc/setup.html
#
#PARAM  use-tls
#PARAM  tls-CAfile      $MAILVAR/db/smtpserver-CAcert.pem
#PARAM  tls-cert-file   $MAILVAR/db/smtpserver-cert.pem
#PARAM  tls-key-file    $MAILVAR/db/smtpserver-key.pe
....
#
# HELO/EHLO-pattern     style-flags
#               [max loadavg]
#
localhost           999 ftveR
some.host.domain    999 !NO EMAIL ACCEPTED FROM YOUR MACHINE

# If the host presents itself as:  HELO [1.2.3.4], be lenient to it..
# The syntax below is due to these patterns being SH-GLOB patterns,
# where brackets are special characters.

\[*\]               999 ve

# Per default demant strict syntactic adherence, including fully
# qualified addresses for  MAIL FROM, and RCPT TO.  To be lenient
# on that detail, remove the "R" from "veR" string below:

*                   999 veR

12.2. Policy Based Relaying Control

The policy database that smtpserver uses is built with policy-builder.sh script, which bundles together a set of policy source files:

If you want, you can modify your smtp-policy.src boilerplate file as well as your installed policy-builder.sh script.

Basically these various source files (if they exist) are used to combine knowledge of valid users around us:

At the “smtp-policy.src” boiler-plate file there is one particular section containing default setting statements. See figure Figure 12-3 for the salient details concerning this.

...
#| ===========================
#|
#| Default handling boilerplates:
#|
#|   "We are not relaying between off-site hosts, except when ..."
#|
#| You MUST uncomment one of these default-defining pairs, or the blocking
#| of relay hijack will not work at all !
#|
#| == 1st alternate: No MX target usage, no DNS existence verify
#|    Will accept for reception only those domains explicitely listed
#|    in ``smtp-policy.mx'' and  ``localnames''  files.  Will not do
#|    verifications on validity/invalidity of source domains:<foo@bar>
#
# .             relaycustomer - relaytarget -
# [0.0.0.0]/0   relaycustomer - relaytarget -
#
#| == 2nd alternate: No MX target usage, DNS existence verify
#|    Like the 1st alternate, except will verify the sender
#|    (MAIL FROM:<..>) address for existence of the DNS MX and/or
#|    A/AAAA data -- e.g. validity.
#|    If RBL parameters are set below, will use them.
#
# .             relaycustomer - relaytarget - senderokwithdns + = _rbl1
# [0.0.0.0]/0   relaycustomer - relaytarget - senderokwithdns + = _rbl0
#
#| == 3rd alternate: MX relay trust, DNS existence verify
#|    For the people who are in deep s*...  That is, those who for some
#|    reason have given open permissions for people to use their server
#|    as MX backup for their clients, but don't know all domains valid
#|    to go thru...  Substitutes accurate data to user's whimsical DNS
#|    maintenance activities.  Vulnerable to inbound MX resource abuse.
#|    If RBL parameters are set below, will use them.

.               relaycustomer - acceptifmx - senderokwithdns + = _rbl1
[0.0.0.0]/0     relaycustomer - acceptifmx - senderokwithdns + = _rbl0

#| == 4th alternate: Sender & recipient DNS existence verify
#|    This is more of an example for the symmetry's sake, verifies that
#|    the source and destination domains are DNS resolvable, but does not
#|    block relaying
#
# .             senderokwithdns - acceptifdns -
# [0.0.0.0]/0   senderokwithdns - acceptifdns -
#
#|
#|  You may also add  ``test-dns-rbl +''  attribute pair to [0.0.0.0]/0
#|  of your choice to use Paul Vixie's  http://maps.vix.com/  MAPS RBL
#|  system.
#|
#| These rules mean that locally accepted hostnames MUST be listed in
#| the database with ``relaytarget +'' attribute.
#|
#| ===========================
#|
#| RBL type test rules:

#| First RBL variant: NONE OF THE RBL TESTS
_rbl0 # Nothing at early phase
_rbl1 # Nothing at late phase

#| Second RBL variant: Early block with RBL+DUL+RSS
#_rbl0 test-dns-rbl      +:dul.maps.vix.com:relays.mail-abuse.org
#_rbl1 # Nothing at late phase

#| Third RBL variant: Late block with RBL+DUL+RSS
#_rbl0 rcpt-dns-rbl      +:dul.maps.vix.com:relays.mail-abuse.org
#_rbl1 test-rcpt-dns-rbl +

#|  (The "+" at the DNS zone defines is treated as shorthand to
#|   "rbl.maps.vix.com")
#|
#|  The Third RBL variant means that all target domains can all by
#|  themselves choose if they use RBL to do source filtering.
#|  The ``= _RBL1'' test *must* be added to all domain instances
#|  where the check is wanted.
#|  (Including the last-resort domain default of ".")
#|  (Or inverting: If some recipient domain is *not* wanting RBL-type
#|   tests, that domain shall have  ``test-rcpt-dns-rbl -'' attribute
#|   pair given for it at the input datasets - consider smtp-policy.mx
#|   file!)
#|
#| These rules mean that locally accepted hostnames MUST be listed in
#| the database with  'relaytarget +' attribute.  ("acceptifmx *" allows
#| reception if the local system is amonst the MXes.)
#|
#| ===========================
#|
#| If your system has ``whoson'' server (see contrib/whoson-*.tgz),
#| you can activate it by adding  'trust-whoson +' attribute pair to
#| the wild-card IP address test:  [0.0.0.0]/0  of your choise.
#|
#| ===========================
#|
#| For outbound relaying control for fixed IP address networks, see
#| comments in file:  smtp-policy.relay
#|
#| ===========================
...

12.3. Content Based Filtering

The ZMailer can do also message content analysis with an external program at the end of DATA-dot phase, and BDAT LAST phase (that is, when the input message is complete, and final acknowledgement is expected by the email sender.)

The program becomes active if PARAM entry “contentfilter” is set:

# External program for received message content analysis:
#PARAM  contentfilter  $MAILBIN/smtp-contentfilter

More details are at the Reference part: Section 17.5.

13.1. Configuration File Programming Language

Whenever the router process starts, its first action is to read its configuration file. The configuration file is a text file which contains statements interpreted immediately when the file is read. Some statements are functions, in which case the function is defined at that point in reading the configuration file. The purpose of the configuration file is to provide a simple way to customize the behavior of routing process of the mailer, and this is primarily achieved by defining the router (at Section 21.4.2), and crossbar (at Section 21.4.3) functions. For these to work properly, some initialization code and auxiliary functions will usually be needed.

At first sight, a configuration file looks like a Bourne shell script. The ideal is to duplicate the functionality, syntax, and to a large degree the semantics, of a shell script. Therefore, the configuration file programming language is defined in terms of its deviation from standard Bourne shell syntax and semantics. The present differences are:

• No repeat statement.

• Functions are allowed, parameter lists are allowed. If not enough arguments are present in a function call to exhaust the parameter list, the so-far unbound parameter variables are bound to "" (the empty string) as local variables. For example, this is the identity address rewriting function:

  null (address) { return $address # surprise! }

  
  
  

• Multiple-value returns are allowed. The return statement can be used to return a non-“” (non-empty string) value from a function. The following are all legal return statements:

  return return $address return $channel ${next_host} ${next_address}

  
  
  

• Variables are dynamically scoped, local variables are the ones in a function's parameter list and those declared with the “local” statement. Only the first value of a multiple-value return may be assigned to a variable. All values are either strings, or lists, so no type information, checking, or declaration, is necessary.

• Quoting is a bit stilted. All quotes (double-, single-, back-), must appear in matching pairs at the beginning and at the end of a word.

  {\bf\large CHECK!} Single quotes are not stripped, double quotes cause the enclosed character sequence to be collected into a quoted-string RFC822 token.

  For example, the statement:

  foo `bar "`baz`"`

  is evaluated as

  (apply 'foo (apply 'bar (baz)))

  
  
  

• In standard shells the IFS guides on how variable expansion results are to be treated. Namely in cases where the expansion happens without being enclosed into double-quotes, the expansion result is at first split with IFS contained characters forming the separation sequences.

  ZMailer's “shell” behaves alike PERL in this regard, and will not do IFS interpolation on the result. However, unlike with PERL, double-quoted evaluation will not have its contents re-evaluated.

  Thus it is equally safe to do assignments like:

  var1='some text here' var2=' more text cat1="$var1$var2" cat2=$var1$var2

  The notable thing at this particular example is that both result variables are catenates of the input strings.

  However! If either of inputs is a list of any kind, then the catenate is not to be done this way! See lappend.

• Due to lack of implicite split by the IFS characters, ZMailer “shell” contains function ifssplit.

• The for construct is even more strange, and classical Bourne script:

  countvar='1 2 3 4 5 6' for x in $countvar; do ... ; done

  Yields only nasty surprise.

  Here are two alternates on how to do it:

  countvar='1 2 3 4 5 6' for x in $(ifssplit $countvar); do ... ; done countvar=(1 2 3 4 5 6) for x in $(elements $countvar); do ... ; done

• Conditional substitution forms are supported:

  ${variable:=value} ${variable:-value} ${variable:+value}

  But appear to be borken ??
  
  

• Patterns (in case labels) are parsed once, the first time they are encountered.

  This is like with PERL's “m/../o” patterns.

• At the end of a case label, the sequentially next case labels of the same case statement will be tried for successful pattern matching (and the corresponding case label body executed). The only exceptions (apart from encountering a return statement) are:

  again

  a function which retries the current case label for a match

  break

  continues execution after the current case statement

  
  
  

• A regular expressions using variant of “case”, with two flavours:

  ssift

  A “String Shift” where the input string is handled as is.

  tsift

  A “Token Shift” where the input string is spliced according to RFC-822 tokenization rules. Especially RFC-822 special characters cause tokens to split.

  With “tsift” the “.” (dot) will match any single “rfc822-token”, that is, input string “foo.bar” has three tokens: “foo” (atom), “.” (dot, special), and “bar” (atom).

  
  
  

  Overall usage of these “sifts” is very much like that of “case”, including the need for matching termination tokens:

  ssift "$invar" in pattern statements ;; tfiss tsift "$invar" in pattern statements ;; tfist

  
  
  

• Various standard Bourne shell functions do not exist built in.

• The general form of function calls in the system is:

  $(funcname arguments)

  
  
  

  It returns a scalar or list object, and the result can be stored into variables at will.

• Relations, and other database lookups are constructed as function calls where the relation name is the function name. More about this later.

There are currently only three entry points (i.e. magic names known to the router code) in the configuration scripts, namely the process, the router, and the crossbar -functions.

• The process() script function is called with a file name as argument. The file is typically located in the $POSTOFFICE/router/ directory. The process() is a protocol switch function which uses the form of the file name to determine how to process different types of messages.

• The router() script function is called with an address as argument, and returns a quad of (channel, host, user, attribs) as three separate values, corresponding to the channel the message should be sent out on (or, the router function can also be called to check on who sent a message), the host or node name for that channel (semantics depend per what channel is in effect), and the address the receiving agent should transmit to. The fourth parameter is “attribute” storage variable name from which a “privilege” value-pair is picked for recipient address security control functions.

• The crossbar() function is in charge of rewriting envelope addresses, selecting message header address munging type (a function to be called with each message header address), and possibly doing per-message logging or enforcing restrictions deemed necessary. It takes a sender-quad and a receiver-quad as arguments (eight parameters altogether). It returns the new values for each element of the two quads, and in addition a function name corresponding to the function to be used to rewrite header addresses for the specific destination. If the destination is to be ignored, returning a null function name will accomplish this.

There is a fourth script entrypoint used by the smtpserver program, namely the server(), which is used to implement smtpserver's realtime support facilities for “EXPN”, and “VRFY” commands, and optionally also to process addresses in “MAIL FROM:<…>”, and “RCPT TO:<…>” commands.

The router has several built in (C coded) functions. Their calling sequence and interface specification is exactly the same as for the functions defined in the configuration file. Some of these functions have special semantics, and they fall into three classes, as follows:

• Functions that are critical to the proper functioning of the configuration file interpreter:

  return

  returns its argument(s) as the value of a function call

  again

  repeats the current case, and *sift label

  break

  exits case, and *sift statements

  
  
  

• Functions that are necessary to complete the capabilities of the interpreter:

  relation

  defines a database to the database lookup mechanism

  sh

  an internal function which runs its arguments as /bin/sh would

  
  
  

• Non-critical but recommended functions:

  echo

  emulates /bin/echo

  exit

  aborts the router with the specified status code

  hostname

  internal function to get and set the local idea about the system name

  trace

  turns on selected debugging output

  untrace

  turns off selected debugging output

  [, test

  emulates a subset of “/bin/test” (a.k.a. “/bin/[”) functionality.

  
  
  

The relation function is described in “Databases”, at section Section 13.2. Functions trace, and untrace are described in connection with debugging.

See Logging and Statistics, section Chapter 16. (This will probably change to Reference/Router/Debugging)

The hostname function requires some further explanation. It is intended to emulate the BSD UNIX /bin/hostname functionality, except that setting the hostname will only set the router's idea of the hostname, not the system's. Doing so will enable generation of “Message-Id:” and “Received:” “trace” headers on all messages processed by the router.

It is done this way since the router needs to know the official domain name of the local host in order to properly generate these headers, and this method is cleaner than reserving a magic variable for the purpose.

The router cannot assume the hostname reported by the system is a properly qualified domain name, so the configuration file may generate it using whichever method it chooses.

If the hostname indeed is a fully qualified domain name, then:

  hostname "hostname"

Finally, note that a symbol can have both a function-value and a string-value. The string value is of course accessed using the “$” prefix convention of the Bourne shell language.

To test the configuration or routing data, proceed as shown in figure Figure 13-2.

sh$ $MAILBIN/router -i         (select interactive mode)
z$ rtrace                      (turn tracing on)
z$ router user@broken.address  (the address that gave you trouble)
z$ router another@address      (and so on)

Old salts can use “/usr/lib/sendmail -bt” instead of “router -i”. Once satisfied that routing works, command:

  zmailer router

You can also run the router directly on a message. Copy your message to someplace other than the postoffice (/tmp/ is usually good), to a numeric file name. If the file name is “123”, you run

  $MAILBIN/router 123

13.2. Databases

FIXME:
 - Intro
 - How 'dbases.conf' file works
 - How the databases are defined in the deep down inside ('relation' function)
 - How lookup works

Many of the decisions and actions taken by configuration file code depend on the specifics of the environment the MTA finds itself in. So, not just the facts that the local host is attached to (say) the UUCP network and a Local Area Network are important, but it is also essential to know the specific hosts that are reachable by this method. Hardcoding large amounts of such information into the configuration file is not practical. It is also undesirable to change what is really a program (the configuration file), when the information (the data) changes.

The desirable solution to this data abstraction problem is to provide a way for the configuration file programmer to manage such information externally to ZMailer, and access it from within the router. The logical way to do this is to have an interface to externally maintained databases. These databases need not be terribly complicated; after all the simplest kind of information needed is that a string is a member of some collection. This could simply correspond to finding that string as a word in a list of words.

However, there are many ways to organize databases, and the necessary interfaces cannot be known in advance. The router therefore implements a framework that allows flexible interfacing to databases, and easy extension to cover new types of databases.

To use a database, two things are needed: the name of the database, and a way of retrieving the data associated with a particular key from that database. In addition to this knowledge, the needs of an MTA do include some special processing pertinent to its activities and the kind of keys to be looked up.

Specifically, the result of the data lookup can take different forms: one may be interested only in the existence of a datum, not its value, or one may be looking up paths in a pathalias database and need to substitute the proper thing in place of “%s” in the string returned from the database lookup. It should be possible to specify that this kind of postprocessing should be carried out in association with a specific data access. Similarly, there may be a need for search routines that depend on the semantics of keys or the retrieved data. These possibilities have all been taken into consideration in the definition of a relation. A relation maps a key to a value obtained by applying the appropriate lookup and search routines, and perhaps a postprocessing step, applied to a specified database that has a specified access method.

The various attributes that define a relation are largely independent. There will of course be dependencies due to the contents or other semantics of a database. In addition to the features mentioned, each relation may optionally have associated with it a subtype, which is a string value used to tell the lookup routine which table of several in a database one is interested in.

There are no predefined relations in the router. They must all be specified in the configuration file before first use. This is done by calling the special function relation with various options, as indicated by the usage strings printed by the relation function when called the wrong way. See figure Figure 13-3.

The “-t” option specifies one of several predefined database types, each with their specific lookup routine. It determines a template for the set of attributes associated with a particular relation. The predefined database types are:

A subtype is specified by appending it to the database type name separated by a slash, or a comma. For example, specifying bind/mx as the argument to the “-t” option will store “mx” for reference by the access routines whenever a query to that relation is processed. The subtypes must therefore be recognized by either the database-specific access routines (for translation into some other form), or by the database interface itself.

For unordered and ordered database types, the datum corresponding to a particular key may be null. This situation arises if the database is a simple list, with one key per line and nothing else. In this situation, the use of an appropriate post-processor option (e.g. “-b”) is recommended to be able to detect whether or not the lookup succeeded.

The “-f” option specifies the name of the database. This is typically a path that either names the actual (and single) database file, or gives the root path for a number of files comprising the database (e.g. “foo” may refer to the NDBM files “foo.pag” and “foo.dir”). For the hostsfile type of database, the /etc/hosts file is the one used (and since the normal “hosts” file access routines do not allow specifying different file, this cannot be overridden).

The “-s” option specifies the size of the cache. If this value is non-zero (by default it is 10), then an LRU cache of this size is maintained for previous queries to this relation, including both positive and negative results.

The “-e” option specifies the cache data expiration time in seconds.

The “-b” option asks that a postprocessor is applied to the database lookup result, so the empty string is returned from the relation query if the database search failed, and the key itself it returned if the search succeeded. In the latter case, any retrieved data is discarded. The option letter is short for Boolean.

The “-n” option asks that a postprocessor is applied to the database lookup result, so the key string is returned from the relation query if the database search failed, and the retrieved datum string is returned if the search succeeded. The option letter is short for Non-Null.

The “-l” option asks that all keys are converted to lowercase before lookup in the database. This is mutually exclusive with the “-u” option.

The “-u” option asks that all keys are converted to uppercase before lookup in the database. This is mutually exclusive with the “-l” option.

The “-d” option specifies a search routine. Most commonly used argument for this option is “pathalias”, specifying a driver that searches for the key using domain name lookup rules.

The “-C” option specifies a configuration file for the underlying database mechanism. Exact details depend by the database mechanisms.

The “-%” option enables substitution of “%0” thru “%9” patterns in the db lookup results with key, iterated partial key, or positional parameter to lookup of the database. See Reference Section 21.5.50 for more information.

 relation -lmt  $DBTYPE -f $MAILVAR/db/aliases$DBEXT     aliases
 relation -lm%t $DBTYPE -f $MAILVAR/db/fqdnaliases$DBEXT fqdnaliases
 relation -lm%t $DBTYPE -f $MAILVAR/db/routes$DBEXT -d pathalias  routes

 if [ -f /etc/resolv.conf ]; then
   relation -nt bind/cname -s 100 canon # T_CNAME canonicalize hostname
   relation -nt bind/uname uname        # T_UNAME UUCP name
   relation -bt bind/mx neighbour       # T_MX/T_WKS/T_A reachability
   relation -t  bind/mp pathalias       # T_MP pathalias lookup
 else
   relation -nt hostsfile -s 100 canon  # canonicalize hostname
   relation -t unordered -f $MAILBIN/db/hosts.uucp uname
   relation -bt hostsfile neighbour
   relation -t unordered -f /dev/null pathalias
 fi

 #
 # We maintain an aliases database, and may access it via NDBM,
 # or via indirect indexing:
 #
 if [ -f $MAILBIN/db/aliases.dat ]; then
     relation -t ndbm -f $MAILBIN/db/aliases aliases
 else
     relation -it ordered -f $MAILBIN/db/aliases.idx aliases
 fi

 relation -t unordered -f /usr/lib/news/active -b newsgroup
 relation -t unordered -f /usr/lib/uucp/L.sys -b ldotsys
 relation -t ordered -f $MAILBIN/db/hosts.transport -d pathalias transport

The final argument for the relation is not preceeded by an option letter. It specifies the name the relation is known under. Note that it is quite possible for different relations to use the same database (like in case of “bind”).

Some sample relation definitions are in figure Figure 13-4. That fragment defines a set of relations that can be accessed in the same way, using the same names, independent of their actual definition.

CHECK! (-i option!) As the comment says, the relation name aliases has special significance to the router. Although the relation is not special in any other way (i.e. it can be used in the normal fashion), the semantics of the data retrieved are bound by assumptions in the aliasing mechanism. (Or more specifically, actually database compilation in case this isn't “ordered” or “unordered” file will handle this.)

These assumptions are that key strings are local-name's, and the corresponding datum gives a byte offset into another file (the root name of the aliases file, with a “.dat” extention), which contains the actual addresses associated with that alias.

The reason for this indirection is that the number of addresses associated with a particular alias can be very large, and this makes the traditional simple database formats inadequate. For example, quick lookup in a text file is only practical if it is sorted and has a regular structure. A large number of addresses associated with an alias makes structuring a problem. The situation for DBM files and variations have problems too, due to the intrinsic limits of the storage method. The chosen indirection scheme avoids such problems without loss of efficiency.

More examples on figure Figure 13-6, where the first two illustrate convenient coincidences of format, and the last definition shows what might be used if outgoing channel information is maintained in a pathalias-format database (e.g. “bar smtp!bar” means to send mail to “bar” via the SMTP channel).

  uunet    ai.toronto.edu!uunet!%s
  .css.gov ai.toronto.edu!uunet!seismo!%s

  uunet    ai.toronto.edu!uunet!%0
  .css.gov ai.toronto.edu!uunet!seismo!%0

  ai.toronto.edu!uunet
  ai.toronto.edu!uunet!seismo!beno.css.gov

relation -t ndbm -f $MAILBIN/uuDB -d pathalias padb

# pathalias database lookup function
padblookup (name) {
    local path
    path = $(padb "$name")
    tsift "$path" in
    ((.+)!)?([^!]+)!%s
        if [ "$3" == "$name" ]; then
            path = "$2!$3"
        else
            path = "$2!$3!$name"
        fi
        ;;
    .*%s.*
        echo "illegal route in pathalias db: $path"
        ;;
    tfist
    return "$path"
}

  people: :include:/usr/lib/mail/lists/people

  people: ":include:/usr/lib/mail/lists/people"

  prog-pipe: "|/path/to/program -args"

  file-path: "/path/to/file"

  included-list: ":include:/path/to/address/file"

14.1. Principiles of scheduling: Threads

The router produces recipients “address quads” which consists of four components:

• Channel

• Host

• User

• Privilege

Of these, Channel, and Host parts are used by the scheduler to classify message recipients and to choose to what Transport Agents to use, and how to use them.

To control how messages are sent out, the scheduler uses catenation of: Channel/Host to group recipients. All recipients with same values there are groupped together in what is called thread.

Another related thing is so called thread group, which is the collection of all threads within same scheduler Selector Clause. See Section 14.3.

In normal case the scheduling is done by running single transport agent for the thread, which delivers messages one at the time. This means that a single destination with thousands of messages does not block the system significantly more than some other destination with a single message.

Also in normal cases, when messages in given thread are all delivered, or otherwise determined that nothing can be done, the transport agent can be switched to another thread within the same thread group.

14.2. Scheduler Resource Control

FIXME! TO BE WRITTEN!

Note, there are three kinds of resource-pool limitation parameters which control when a given channel+host pair (thread) is NOT taken into processing:

14.3. The scheduler.conf file

Any line starting with a “#” character is assumed to be a comment line, and is ignored, as are empty lines. All other lines must follow a rigid format.

The scheduler configuration file consists of a set of clauses. Each clause is selected by the pattern it starts with. The patterns for the clauses are matched, in sequence, with the channel/host string for each recipient address. When a clause pattern matches an address, the parameters set in the clause will be applied to the scheduler's processing of that address. If the clause specifies a command, the clause pattern matching sequence is terminated. Example of the clause can be seen in figure Figure 14-3.

local/*
        interval=10s
        expiry=3h
        # want 20 channel slots in case of blockage on one
        maxchannel=20
        # want 20 thread-ring slots
        maxring=20
        command="mailbox -8"

A clause consists of:

• A selection pattern (in shell style) that is matched against the channel/host string for an address.

• 0 or more variable assignments or keywords (described below).

If the selection pattern does not contain a '/', it is assumed to be a channel pattern and the host pattern is assumed to be the wildcard '*'.

The components of a clause are separated by whitespace. The pattern introducing a clause must start in the first column of a line, and the variable assignments or keywords inside a clause must not start in the first column of a line. This means a clause may be written both compactly all on one line, or spread out with an assignment or keyword per line.

If the clause is empty (i.e., consists only of a pattern), then the contents of the next non-empty clause will be used.

The typical configuration file will contain the following clauses:

• a clause matching all addresses (using the pattern */*) that sets up default values.

• a clause matching the local delivery channel (usually local).

• a clause matching the deferred delivery channel (usually hold).

• a clause matching the error reporting channel (usually error).

• clauses specific to the other channels known by the router, for example: smtp and uucp.

The actual names of these channels are completely controlled by the router configuration file.

Empty lines, and lines whose first non-whitespace character is “#”, are ignored.

Variable values may be unquoted words or values or double quoted strings. Intervals (delta time) are specified using a concatenation of numbers suffixed with 's', 'm', 'h', or 'd'; modifiers designating the number as a second, minute, hour, or day value. For example: 1h5m20s.

The known variables and keywords, and their typical values and semantics are:

An example of full scheduler.conf file is in figure Figure 14-4.

# Default values
*/*       interval=1m expiry=3d retries="1 1 2 3 5 8 13 21 34"
          maxring=0 maxta=0 skew=5 user=root group=daemon
# Boilerplate parameters for local delivery and service channels
local/*   interval=10s expiry=3h maxchannel=2 command=mailbox
error     interval=5m maxchannel=10 command=errormail
hold/*    interval=5m maxchannel=1 command=hold

# Miscellaneous channels supported by router configuration
smtp/*.toronto.edu
smtp/*.utoronto.ca maxchannel=10 maxring=2
          command="smtp -srl ${LOGDIR}/smtp"
smtp      maxchannel=10 maxring=5
          command="smtp -esrl ${LOGDIR}/smtp"

uucp/*    maxchannel=5 command="sm -c ${channel} uucp"

The first clause (*/*) sets up default values for all addresses. There is no command specification, so clause matching will continue after address have picked up the parameters set here.

The third clause (error) has an implicit host wildcard of “*”, so it would match the same as specifying error/* would have.

The fifth clause (smtp/*.toronto.edu) has no further components so it selects the components of the following nonempty clause (the sixth).

Both the fifth and sixth clauses are specific to address destinations within the TORONTO.EDU and UTORONTO.CA organization (the two are parallel domains). At most 10 deliveries to the smtp channel may be concurrently active, and at most 2 for all possible hosts within TORONTO.EDU. If “$host” is mentioned in the command specification, the transport agent will only be told about the message control files that indicate SMTP delivery to a particular host. The actual host is picked at random from the current choices, to avoid systematic errors leading to a deadlock of any queue.

14.4. Scheduler's Mailq

FIXME! TO BE WRITTEN!

14.5. Scheduler's scheduler.auth control file

FIXME! TO BE WRITTEN!

14.6. manual-expirer

FIXME! TO BE WRITTEN!

14.7. Scheduler's Diagnostics Reporting

FIXME! TO BE WRITTEN!

15.1. Sm Transport Agent

sm is a ZMailer's sendmail(8) compatible transport agent to deliver messages by invoking a program with facilities and in a way compatible with a sendmail(8) MTA.

The program scans the message control files named on stdin for addresses destined for the channel and/or the host given on the command line. If any are found, all matching addresses and messages are processed according to the specifications for the mailer in the configuration file.

The exit status of a mailer should be one of the standard values specified in #include <sysexits.h>. Of these, EX_OK indicates successful deliver, and EX_DATAERR, EX_NOUSER, EX_NOHOST, EX_UNAVAILABLE, and EX_NOPERM indicate permanent failure. All other exit codes will be treated as a temporary failure and the delivery will be retried.

Usage:

sm [-8] [-H] [-Q] [-V] [-f configfile] -c channel -h host mailer

Configuration:

The configuration file $MAILSHARE/sm.conf associates the mailer keyword from the command line with a specification of a delivery program. This is very similar to the way the definition of a “mailer” in sendmail(8). It requires flags, a program name, and a command line specification. These are in fact the fields of the entries of the configuration file. Lines starting with whitespace or a “#” are ignored, and all others are assumed to follow format shown in figure Figure 15-2.

#mailer flags program        argument list
#====== ===== ============== ================================
local   mS    sm/localm      localm -r $g $u
prog    -     /bin/sh        sh -c $u
tty     rs    /usr/local/to  to $u
uucp    U     /usr/bin/uux   uux - -r -a$g -gC $h!rmail ($u)
usenet  m     sm/usenet      usenet $u
test    n     sm/test        test $u

The mailer field extends from the beginning of the line to the first whitespace. It is used simply as a key index to the configuration file contents. One or more whitespace is used as the field separator for all the fields.

The flags field contains a concatenation of one-letter flags. If no flags are desired, a “-” character should be used to indicate presence of the field. All normal sendmail (of 8.11(.0)) flags are recognized, but the ones that do not make sense in the context of ZMailer will produce an error (or some are ignored). The flags that change the behaviour of sm are:

The path field specifies the location of the delivery program. Relative pathnames are allowed and are relative to the $MAILBIN/ directory.

The arguments field extends to the end of the line. It contains whitespace separated argv parameters which may contain one of the following sequences: