INTERNALS.md (7000B)
1 1. Overview 2 3 Here's the data flow in the qmail suite: 4 5 ``` 6 qmail-smtpd --- qmail-queue --- qmail-send --- qmail-rspawn --- qmail-remote 7 / | \ 8 qmail-inject _/ qmail-clean \_ qmail-lspawn --- qmail-local 9 ``` 10 11 Every message is added to a central queue directory by qmail-queue. 12 qmail-queue is invoked as needed, usually by qmail-inject for locally 13 generated messages, qmail-smtpd for messages received through SMTP, 14 qmail-local for forwarded messages, or qmail-send for bounce messages. 15 16 Every message is then delivered by qmail-send, in cooperation with 17 qmail-lspawn and qmail-rspawn, and cleaned up by qmail-clean. These four 18 programs are long-running daemons. 19 20 The queue is designed to be crashproof, provided that the underlying 21 filesystem is crashproof. All cleanups are handled by qmail-send and 22 qmail-clean without human intervention. See section 6 for more details. 23 24 25 2. Queue structure 26 27 Each message in the queue is identified by a unique number, let's say 28 457. The queue is organized into several directories, each of which may 29 contain files related to message 457: 30 31 ``` 32 mess/457: the message 33 todo/457: the envelope: where the message came from, where it's going 34 intd/457: the envelope, under construction by qmail-queue 35 info/457: the envelope sender address, after preprocessing 36 local/457: local envelope recipient addresses, after preprocessing 37 remote/457: remote envelope recipient addresses, after preprocessing 38 bounce/457: permanent delivery errors 39 ``` 40 41 Here are all possible states for a message. + means a file exists; - 42 means it does not exist; ? means it may or may not exist. 43 44 ``` 45 S1. -mess -intd -todo -info -local -remote -bounce 46 S2. +mess -intd -todo -info -local -remote -bounce 47 S3. +mess +intd -todo -info -local -remote -bounce 48 S4. +mess ?intd +todo ?info ?local ?remote -bounce (queued) 49 S5. +mess -intd -todo +info ?local ?remote ?bounce (preprocessed) 50 ``` 51 52 Guarantee: If mess/457 exists, it has inode number 457. 53 54 55 3. How messages enter the queue 56 57 To add a message to the queue, qmail-queue first creates a file in a 58 separate directory, pid/, with a unique name. The filesystem assigns 59 that file a unique inode number. qmail-queue looks at that number, say 60 457. By the guarantee above, message 457 must be in state S1. 61 62 qmail-queue renames pid/whatever as mess/457, moving to S2. It writes 63 the message to mess/457. It then creates intd/457, moving to S3, and 64 writes the envelope information to intd/457. 65 66 Finally qmail-queue creates a new link, todo/457, for intd/457, moving 67 to S4. At that instant the message has been successfully queued, and 68 qmail-queue leaves it for further handling by qmail-send. 69 70 qmail-queue starts a 24-hour timer before touching any files, and 71 commits suicide if the timer expires. 72 73 74 4. How queued messages are preprocessed 75 76 Once a message has been queued, qmail-send must decide which recipients 77 are local and which recipients are remote. It may also rewrite some 78 recipient addresses. 79 80 When qmail-send notices todo/457, it knows that message 457 is in S4. It 81 removes info/457, local/457, and remote/457 if they exist. Then it reads 82 through todo/457. It creates info/457, possibly local/457, and possibly 83 remote/457. When it is done, it removes intd/457. The message is still 84 in S4 at this point. Finally qmail-send removes todo/457, moving to S5. 85 At that instant the message has been successfully preprocessed. 86 87 88 5. How preprocessed messages are delivered 89 90 Messages at S5 are handled as follows. Each address in local/457 and 91 remote/457 is marked either NOT DONE or DONE. 92 93 - DONE: The message was successfully delivered, or the last delivery 94 attempt met with permanent failure. Either way, qmail-send 95 should not attempt further delivery to this address. 96 97 - NOT DONE: If there have been any delivery attempts, they have all 98 met with temporary failure. Either way, qmail-send should 99 try delivery in the future. 100 101 qmail-send may at its leisure try to deliver a message to a NOT DONE 102 address. If the message is successfully delivered, qmail-send marks the 103 address as DONE. If the delivery attempt meets with permanent failure, 104 qmail-send first appends a note to bounce/457, creating bounce/457 if 105 necessary; then it marks the address as DONE. Note that bounce/457 is 106 not crashproof. 107 108 qmail-send may handle bounce/457 at any time, as follows: it (1) injects 109 a new bounce message, created from bounce/457 and mess/457; (2) deletes 110 bounce/457. 111 112 When all addresses in local/457 are DONE, qmail-send deletes local/457. 113 Same for remote/457. 114 115 When local/457 and remote/457 are gone, qmail-send eliminates the 116 message, as follows. First, if bounce/457 exists, qmail-send handles it 117 as described above. Once bounce/457 is definitely gone, qmail-send 118 deletes info/457, moving to S2, and finally mess/457, moving to S1. 119 120 121 6. Cleanups 122 123 If the computer crashes while qmail-queue is trying to queue a message, 124 or while qmail-send is eliminating a message, the message may be left in 125 state S2 or S3. 126 127 When qmail-send sees a message in state S2 or S3 -- other than one 128 it is currently eliminating! -- where mess/457 is more than 36 hours old, 129 it deletes intd/457 if that exists, then deletes mess/457. Note that any 130 qmail-queue handling the message must be dead. 131 132 Similarly, when qmail-send sees a file in the pid/ directory that is 133 more than 36 hours old, it deletes it. 134 135 Cleanups are not necessary if the computer crashes while qmail-send is 136 delivering a message. At worst a message may be delivered twice. (There 137 is no way for a distributed mail system to eliminate the possibility of 138 duplication. What if an SMTP connection is broken just before the server 139 acknowledges successful receipt of the message? The client must assume 140 the worst and send the message again. Similarly, if the computer crashes 141 just before qmail-send marks a message as DONE, the new qmail-send must 142 assume the worst and send the message again. The usual solutions in the 143 database literature -- e.g., keeping log files -- amount to saying that 144 it's the recipient's computer's job to discard duplicate messages.) 145 146 147 7. Further notes 148 149 Currently info/457 serves two purposes: first, it records the envelope 150 sender; second, its modification time is used to decide when a message 151 has been in the queue too long. In the future info/457 may store more 152 information. Any non-backwards-compatible changes will be identified by 153 version numbers. 154 155 When qmail-queue has successfully placed a message into the queue, it 156 pulls a trigger offered by qmail-send. Here is the current triggering 157 mechanism: lock/trigger is a named pipe. Before scanning todo/, 158 qmail-send opens lock/trigger O_NDELAY for reading. It then selects for 159 readability on lock/trigger. qmail-queue pulls the trigger by writing a 160 byte O_NDELAY to lock/trigger. This makes lock/trigger readable and 161 wakes up qmail-send. Before scanning todo/ again, qmail-send closes and 162 reopens lock/trigger.