1.1.1. The Messaging Model

In addition to users, the basic building blocks of messaging are User Agents (UAs) and Message Submission Agent (MSA) is newer term for specific sub-task of Internet email, namely authenticated message submission to first MTA of MTS system. Message Transfer Agents (MTAs). User agents are the interface through which a human user interacts with the messaging system. On non-interpersonal messaging user agents may be built directly into applications. MTAs are used to transport messages from one computer system to another. An example of a good MTA is ZMailer. Access Units (AUs) can be used for accessing telematic services, for example telefax. (Or in general act on user behalf somehow, e.g. automated scripts.) Message Stores (MSs) can be used between MTAs and UAs. They are used for storing messages before and while UAs are used to access and manipulate them. Message Delivery Agent (MDA) is newer term intended to cover specific sub-task of moving the message from MTA system to the care of MS, UA or AU. Some new MTA suites even do parts of the UA functionality (“.forward” -processing) in the MDA. Gateways (GWs) are used in between two different types of messaging systems, for example between SMTP and X.400 mail. Gateways are conceptually made of two MTAs and a UA that interconnects them.

There are two more acronyms worth looking at in the messaging model, namely MTS (Message Transport System) and MHS (Message Handling System). MTS is the collection of MTAs (and GWs and MSAs and MDAs), while MHS includes MTS and UA functionality (UAs, MSs and AUs).

All in all, X.400 terms are not a complete match on how things are done in Internet email, nor should they be considered as normative, merely giving you a reasonable frame of reference that isn't very wrong.

A graphical example of the messaging model is shown in figure Figure 1-1. It shows the relationships between different elements of the model.

As can be seen, a user may use more than one UA, and a UA can be connected to more than one MTA.

Although it is important to understand the relationships between different entities in the messaging model, it is even more important to understand the nature of a message and the way UAs and MTAs deal with it.

A message consists of a body and headers. In case of messages with more than one body-part (for example some MIME messages) the different body parts are all part of the outermost body-part.

  Headers
    From:    The Manager
    To:      One Bright Employee
    CC:      secretary
    Subject: Salary raise
    Date:    17 May 1997

  Body
    Dear Employee,
    ...
    The Manager

  Headers
  Body
    Headers
    Body
      Headers
      Body
      Headers
      Body
    Headers
    Body

As can be seen, there is always just one outermost body that contains all other body-parts. In some cases, for example X.400 the protocol seems to violate this by leaving out the outermost body-part. However, even on those cases we must assume, at the abstract level, that the outermost body-part is there.

However, this is not all that there is to the structure of a message. When a message is in transit, being handled by MTAs it is put inside an envelope, just like a normal letter is inside an envelope while the postal service is carrying it. Just like the postal service is not permitted to look inside the envelope, neither are MTAs permitted to look inside. Whenever there is a need to look inside the envelope, it is always a UA function, and done on behalf, and on the authority of, a UA.

There are some violations of this. When using the SMTP protocol, the Received: lines are put in the headers by MTAs. This is bad engineering, but as the process of adding a new header line is fairly straightforward, it doesn't cause too much pain. In some cases MTAs, and especially the GWs modify the header even more, and sometimes they even mess with the body. This is a sure recipe for trouble.

Graphically, the way a message should be dealt with is shown in figure Figure 1-4.

The user creates the message with the help of a UA. How the interaction is arranged is a local matter. Once the message has been prepared, it is passed to a nearby MTA together with necessary envelope information and put into an envelope. The MTA puts its stamp on the envelope to show that it has received the message. The first MTA passes the message to the second MTA. The second MTA puts its stamp to the message and passes it to the third MTA, and so on. The final MTA passes the message to a UA, and the envelope is removed.

There are at least three ways to pass the message from an MTA to a UA. The message may be pushed to a (running) UA, a UA may pull it from an MTA, or an MTA may pass the message to an MS from which a UA will receive it at a convenient time.

The normal UNIX way of delivering mail (/usr/spool/mail/user) can be seen as any of the above three mechanisms, but should normally be seen as a UA pulling a message. The reason for this confusion is with the de-facto SMTP standard MTA, Sendmail.

Although Sendmail is in many ways a very clever piece of software, it is also the reason for many problems, as it has blurred the line between MTA and UA. Sendmail is clearly an MTA, but it also performs many of the UA level functions, like handling of “.forward” files. This behaviour has become the de-facto standard way for UNIX MTAs to behave, making it necessary for any MTA, including ZMailer, to behave the same way.

1.1.2. Routing And Delivering Messages

As MTAs only deal with envelopes, all routing and delivery decisions have to be based on information available on the envelope. It follows from this that the envelope and headers may contain conflicting information. This is normal, and is not a cause for worry.

MTAs may, and often do modify addresses present in the envelope. This might include changing addresses to a format more suitable for mail delivery and alias expansion.

It is important to make a distinction between aliasing and forwarding mail. Aliasing is an MTA function, in which an MTA effectively knows that to reach a seemingly local user, mail should be sent to a different address. To accomplish this, the MTA changes the recipient information on the envelope. Forwarding is a UA function. When forwarding, the mail message is received by the original, intended recipient, and re-sent to another address. Although forwarding is a UA function, it doesn't have to result in a change to body or headers, but on the envelope, both sender and recipient should be changed. Sendmail bluntly violates this, and makes most other MTAs violate it as well.

Most mailing lists today are just alias expansions, on which one recipient address on the envelope is replaced with multiple addresses. In many cases this is a reasonable approach. However, all major mailing lists should be set up as a UA function. This involves changing not only the recipient address but also the sender address in the envelope. In this way, undeliverable messages are sent to the owner of the list, who can deal with the problem, and not to the sender of the message, who can do nothing to remedy the situation. (The ZMailer has some built-in facilities for this, see about Mailing Lists… on section Section 13.2.2.)

Error messages must be sent to the envelope sender address, as this is a MTA function.

Replies to messages should be sent to header addresses, because replying is a UA function.

2.1.1. Design Summary

ZMailer is a multi-process mailer, using three daemon processes to manipulate messages. Used technologies are as simple as possible, e.g. while networking stuff is as advanced as possible (with fallbacks to simplest basic behaviour), some of other modern things (like threads) are not used in favor of simpler approaches

• Message arrive in via sendmail program "API" for internally originated messages, or via Smtpserver subsystem, which is system front-door for messages coming in from the external network. The Smtpserver is in reality a cluster of auxiliary programs providing efficient low-overhead support for various analysis things needed while messages are coming in. Second one of these processes is a Router, and makes all decisions about what should happen to a message; routing and possibly message header visible things rewriting. The third daemon is a message queue manager, Scheduler, used to schedule the delivery of messages. The Router uses a configuration file that closely follows Bourne shell script syntax and semantics, with minimal magic. Message files are moved around in a series of directories, and the Scheduler and its Transport Agents run off of control files created by the Router.

• The Sendmail is very simple "plug-compatible" message submission agent for system internal message submissions into the ZMailer MTA, and does all its things without any sort of set-uid privilege escalation needs.

• The Smtpserver has evolved into rather complicated animal, as it aims to really efficiently support things that in the early days required truly heavy-weight auxiliary program startups for every incoming SMTP connection -- startup of such auxiliaries has now been shared over a large number of arriving connection and messages by means of having them as Smtpserver's permant auxiliary helpers:

  • First is a slightly less burdened Router for determining if source or destination address domains are possibly known in the system, and primarily being able to reject messages at front door that are destined to nonexistent addresses. (This instance is separate from main Router, and thus is somewhat duplicating main Router's task, but this doesn't e.g. do list expansions and other such expensive things.)

  • There is also Content-Filter that can be used up to how ever complicated message content analysis syncronously with incoming message feed (there is also a possible "input" subdirectory for offline non-synchronous content analysis, however synchronous processing has certain appeal in itself, not the least the ability to tell message sender to go and stuff their SPAMs into...)

  • And third is the Rate-Tracker subsystem, that can keep track of such things as "Non-authenticated customer at IP address N.N.N.N has sent more than 60 messages in past hour, stop that sending until the sliding window allows more to be sent."

  These auxiliary servers do operate so that when smtpserver-subsystem shutdown has been ordered, and last client needing support goes away, they drop away themselves.
  
  

• The main Router subsystem will process messages one at a time (per Router instance), as it finds them in a directory where User Agents submit their outgoing messages. Message Envelope and Message Header information is all kept in the same message file along with the message body, and this file is never modified by any ZMailer program. After parsing the envelope and RFC822 header information, the Router validates the information extracted, and calls functions defined in the configuration file to decide exactly, how to deliver the message and how to transform the embedded addresses. The algorithms that do this are easily re-configurable, since the control flow and address manipulation is specified by familiar(ish) shell script statements. When the Router is finished on a message, it will produce a message control file for use by the delivery processing stage of ZMailer, and move the original message file to another location.

• Once the main Router subsystem has decided what to do with each of the addresses in a message, the Scheduler builds a summary of this information by reading the control file created by the Router. This knowledge is merged with a data structure it maintains that stores which messages are supposed to be sent where, and how. According to a pre-arranged agenda, the Scheduler will execute delivery programs to properly move the message envelope, header, and body, to the immediate destination. These delivery programs are called Transport Agents, and communicate with the Scheduler using a simple protocol that tells them which messages to process and returns status reports to the Scheduler. The Scheduler also manages status reports, taking appropriate action on delivery errors and when all delivery instructions for a message have been processed.

• There are several standard Transport Agents included with the ZMailer distribution. The collection currently includes a local delivery program, an SMTP client implementation, and a Transport Agent that can run Sendmail-M-line-compatible delivery programs.

• A separate mailq utility allows querying the Scheduler for the state of its queues. For existing Sendmail installations, a replacement program is included that simulates most of the Sendmail functionality in the ZMailer environment. This allows ZMailer to replace a Sendmail installation without requiring changes in standard User Agents.

• Several other tools and utilities exist for other specific purposes.

• There are also facilities that allow loosely coupled cluster creation. Each member machine of the cluster is separate entity, but they can keep track of rate-tracking data of all of their neighbours, as well as activate on demand ETRN queue flush in cluster wide setup. None of these is guaranteed in any strict way, which makes their implementation technology considerably simpler, and also occasional limits that they implement are not considered to be upheld rigorously under every possible conditions. A tripple-sigma reliability on e.g. limit enforcement is considered "a plenty good enough".

  In loosely coupled clusters, delivery to system internal (UNIX-style) message store is not clusterized, unless the store in question is some sort of e.g. LMTP connected external Message Store. (Even UNIX-style mailbox files may work, if underneath there is some sort of cluster-wide filesystem, which has working FCNTL locks.) One such way was once to use front-line processing to map arrived message to actual recipient and node into which it was expected to go, and to proxy POP and IMAP services so that users didn't need to know on which node their mailboxes did actually reside.