FTS-0004        EchoMail Specification

This document is directly derived from the documentation of

-------------------------------------------------------------------------------

                           The Conference Mail System
                                        
                                       By
                                  Bob Hartman
                       Sysop of FidoNet(tm) node 132/101
                                        
                  (C) Copyright 1986,87, Spark Software, Inc.
                                        
                              427-3 Amherst Street
                              CS  2032, Suite  232
                              Nashua,  N.H.  03061
                                        
                              ALL RIGHTS RESERVED.

-------------------------------------------------------------------------------

version 3.31 of 12 December, 1987.

With Bob Hartman's kind consent, copying for the purpose of technological
research and advancement is allowed.

                                        
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
                                        

     WHAT IS THE CONFERENCE MAIL SYSTEM?

          Conference Mail  is a  technique to  permit several  nodes  on  a
          network to  share a  message base,  similar  in  concept  to  the
          conferences available on many of the computer services, but it is
          most closely related to the Usenet system consisting of more than
          8,000 systems  world wide. All systems sharing a given conference
          see any  messages entered  into the  conference  by  any  of  the
          participating systems.  This can  be implemented in such a way as
          to be  totally transparent  to the users of a particular node. In
          fact, they  may not  even be  aware of  the network being used to
          move their  messages about from node to node! Unfortunately, this
          has its  disadvantages also  - most  users who  are not  educated
          about Conference  Mail do  not realize  the messages  transmitted
          cost MANY  sysops (system  operators) money,  not just  the local
          sysop. This  is an important consideration in Conference Mail and
          should not be taken lightly.  In a conference with 100 systems as
          participants the cost per message can get quite high.

          The Conference  Mail System is designed to operate in conjunction
          with a  FidoNet compatible  mail server.  The currently supported
          mail servers  are Fido(tm),  SEAdog(tm), Opus, and Dutchie. Since
          the mail  server is  a prerequisite  to using the Conference Mail
          System, it  will be  assumed you  already have  your mail  server
          operating correctly  on your   system, and you are connected into
          FidoNet or a compatible network.


     HISTORY OF THE CONFERENCE MAIL SYSTEM

          In late  1985, Jeff  Rush, a  Fido  sysop  in  Dallas,  wanted  a
          convenient means  of sharing  ideas with the other Dallas sysops.
          He created  a system  of programs  he called  Echomail,  and  the
          Dallas sysops' Conference was born.

          Within a  short time  sysops in other areas began hearing of this
          marvelous new  gadget and  Echomail took  on a  life of  its own.
          Today, a  scant year and a half later, the FidoNet public network
          boasts a myriad of conferences varying in size from the dozen-or-
          so participants  in the  FidoNet  Technical  Standards  Committee
          Conference  to   the  Sysops'  Conference  with  several  hundred
          participants. It  is not  uncommon for a node to carry 30 or more
          conferences and share those conferences with 10 or more nodes.


     HOW IT WORKS

          The Conference  Mail System  is functionally  compatible with the
          original Echomail utilities.  In general, the process is:

          1. A  message is  entered into  a designated  area on  a  FidoNet
          compatible system.

          2. This message is "Exported" along with some control information
          to each system "linked" to the conference through the originating
          system.

          3. Each  of the  receiving systems  "Import" the message into the
          proper Conference Mail area.

          4. The receiving systems then "Export" these messages, along with
          additional control  information,  to  each  of  their  conference
          links.

          5. Return to step 3.

          As you  can see,  the method  is quite  simple -  in general.  Of
          course, following  the steps  literally would mean messages would
          never stop being Exported and transmitted to other systems.  This
          obviously would  not be  desired or  the  network  would  quickly
          become overburdened.  The information  contained in  the 'control
          information' section  is used  to prevent  transmitting the  same
          message more than once to a single system.


     CONFERENCE MAIL MESSAGE CONTROL INFORMATION

          There are  five pieces  of control  information associated with a
          Conference  Mail  message.  Some  are  optional,  some  are  not.
          Normally this information is never entered by the person creating
          the  message.   The  following   control  fields   determine  how
          Conference Mail is handled:

          1. Area line

               This is  the first  line of  a conference  mail message. Its
               actual appearance is:

                                     AREA:CONFERENCE

               Where CONFERENCE is the name of the conference. This line is
               added when  a conference  is  being  "Exported"  to  another
               system. It  is based upon information found in the AREAS.BBS
               (configuration) File  for the  designated message area. This
               field is  REQUIRED by  the receiving  system to  "Import"  a
               message into the correct Conference Mail area.

          2. Tear Line

               This line is near the end of a message and consists of three
               dashes (---)  followed by  an  optional  program  specifier.
               This is  used to show the first program used to add Echomail
               compatible control information to the message. The tear line
               generated by Conference Mail looks like:

                                    --- <a small product-specific banner>

               This  field   is  optional   for  most  Echomail  compatible
               processors, and  is added  by the  Conference Mail System to
               ensure complete  compatibility. Some systems will place this
               line in  the message  when it  is first  created, but  it is
               normally added when the message is first "exported."

          3. Origin line

               This line  appears near  the bottom of a message and gives a
               small amount  of  information  about  the  system  where  it
               originated. It looks like:

                      * Origin: The Conference Mail BBS (1:132/101)

               The "  * Origin:  " part  of the  line is  a constant field.
               This is followed by the name of the system as taken from the
               AREAS.BBS file  or a  file named  ORIGIN located  in the DOS
               directory of  the  designated  message  area.  The  complete
               network address  (1:132/101 in  this case)  is added  by the
               program inserting  the line.  This field is generated at the
               same time  as the  tear line,  and therefore  may either  be
               generated at  the time  of  creation  or  during  the  first
               "export"  processing.   Although  the  Origin  line  is  not
               required by  all Echomail  processors, it  is added  by  the
               Conference Mail System to ensure complete compatibility.


          4. Seen-by Lines

               There can  be many  seen-by lines  at the  end of Conference
               Mail messages,  and they  are the real "meat" of the control
               information. They  are used  to  determine  the  systems  to
               receive the exported messages. The format of the line is:

                           SEEN-BY: 132/101 113 136/601 1014/1

               The net/node  numbers correspond  to the net/node numbers of
               the systems having already received the message. In this way
               a message  is never  sent to a system twice. In a conference
               with many  participants the  number of  seen-by lines can be
               very large.   This line is added if it is not already a part
               of the  message, or added to if it already exists, each time
               a message  is exported  to other systems. This is a REQUIRED
               field, and  Conference Mail  will not  function correctly if
               this field  is not put in place by other Echomail compatible
               programs.

          5. PATH Lines

               These are  the last  lines in  a Conference Mail message and
               are a  new addition,  and therefore  is not supported by all
               Echomail processors. It appears as follows:

                                  ^aPATH: 132/101 1014/1

               Where the  ^a stands  for Control-A  (ASCII character 1) and
               the net/nodes  listed correspond  to  those  systems  having
               processed the  message before it reached the current system.
               This is  not the  same as  the seen-by  lines, because those
               lines list  all systems  the message has been sent to, while
               the path line contains all systems having actually processed
               the message.  This is not a required field, and few echomail
               processors currently  support it,  however it  can  be  used
               safely with  any other  system, since  the line(s)  will  be
               ignored. For  a discussion  on how  the  path  line  can  be
               helpful, see the "Advanced Features" section of this manual.


     METHODS OF SENDING CONFERENCE MAIL

          To this  point the  issue of how Conference Mail is actually sent
          has been glossed over entirely. The phrase has been, "the message
          is exported  to another  system."   What exactly  does this mean?
          Well, for starters lets show what is called the "basic" setup:

          In this setup exported mail is placed into the FidoNet mail area.
          Each message   exported  from a  Conference  Mail  area  has  one
          message generated  for each  receiving system.  This mail is then
          sent the  same as any other network mail. When Echomail was first
          created this was the only way mail could be sent.

          The "basic"  method has some disadvantages. First, since Echomail
          has grown so large it is not uncommon to get 200 new messages per
          day imported  into various message bases. It is also not uncommon
          for a  system to  be exporting  messages to 4 or 5 other systems.
          Simple arithmetic  shows 800-1000  messages per day would be sent
          in normal  netmail! This  puts a tremendous strain on any netmail
          system, not  to mention transmission time and the resultant phone
          charges. When this limitation of Echomail was first noticed a lot
          of people started scratching their heads wondering what to do. If
          a  solution  could  not  be  found  it  appeared  Echomail  would
          certainly overrun the capabilities of FidoNet.

          Thom Henderson  (from System Enhancement Associates) came up with
          the original  ARCmail program.  Having previously written the ARC
          file archiving  and compression  program,  he  knew  the  savings
          achievable by  having all  of the netmail messages placed in .ARC
          format for  transmission. As  a byproduct, the messages no longer
          appeared in  the netmail  area,  but  were  included  in  a  file
          attached to  a message  (see your  FidoNet mailer manual for file
          attaches).  In   this  way  the  tremendous  number  of  messages
          generated, and the phone bill problems were both solved.

          Unfortunately, ARCmail  required the  messages to first be placed
          into the  netmail area  before it  could be  run. In  effect,  it
          caused the  messages to  be scanned once when they were exported,
          once during  the ARCmail  phase, once when ARCmail was run at the
          other end  to get  the messages out of .ARC format, and once when
          those messages  were later  imported into  a message  base on the
          receiving system.  The Conference Mail System solves this problem
          by eliminating  the ARCmail  program. Conference  Mail builds the
          ARCmail files during Export, and unpacks them during Import. This
          way  messages   are  exported  directly  to  ARCmail  style  file
          attaches, and imported directly from ARCmail style file attaches.
          The scanning  phases between importing and exporting messages are
          totally removed and processing time is proportionally reduced.

          This is  now the  most common  method for sending Conference Mail
          between systems.  The overhead  involved in  doing it  during the
          importing and exporting phases is much less than what is involved
          if ARCmailing  is not  utilized. This was a primary consideration
          in the  design and  implementation of the Conference Mail System,
          and as  a result  the entire system is optimized for this type of
          use.  Please  refer  to  the  Import  and  Export  functions  for
          specifics on how to use the ARCmailing feature.


     CONFERENCE TOPOLOGY

          The  way   in  which  systems  link  together  for  a  particular
          conference is  called the "conference topology."  It is important
          to know  this structure  for two  reasons: 1)  It is important to
          have a  topology which  is  efficient  in  the  transfer  of  the
          Conference Mail  messages, and  2) It  is  important  to  have  a
          topology which  will not  cause systems  to see the same messages
          more than once.

          Efficiency can  be measured  in a  number  of  ways;  least  time
          involved for all systems to receive a message, least cost for all
          systems to receive a message, and fewest phone calls required for
          all systems  to receive  a message  are all  valid indicators  of
          efficiency. Users  of Echomail compatible systems have determined
          (through trial  and error)  the best  measure of  efficiency is a
          combination  of  all  three  of  the  measurements  given  above.
          Balancing the equation is not trivial, but some guidelines can be
          given:

               1. Never have two systems attempting to send Conference mail
               to each other at the same time. This results in "collisions"
               that will  cause both  systems to  fail. To  avoid this, one
               system should  be responsible  for polling  while the  other
               system is holding mail. This arrangement can alternate based
               upon various  criteria, but  both systems  should  never  be
               attempting to call each other at the same time.

               2. Have  nodes form  "stars" for  distribution of Conference
               Mail. This arrangement has several nodes all receiving their
               Conference Mail from the same system. In general the systems
               on the  "outside"  of  the  star  poll  the  system  on  the
               "inside". The  system on  the "inside"  in turn  polls other
               systems to  receive the Conference Mail that is being passed
               on to the "outside" systems.

               3. Utilize  fully connected  polygons with  a few  vertices.
               Nodes can  be connected in a triangle (A sends to B and C, B
               sends to  A and  C, C sends to A and B) or a fully connected
               square (all  corners of  the square send to all of the other
               corners). This  method is useful for getting Conference Mail
               messages to each node as quickly as possible.


          All of  these efficiency  guidelines have to be tempered with the
          guidelines dealing  with keeping  duplicate messages  from  being
          exported. Duplicates  will occur  in any  topology that  forms  a
          closed polygon  that is not fully connected. Take for example the
          following configuration:

                                      A ----- B
                                      |       |
                                      |       |
                                      C ----- D

          This square  is a  closed polygon that is not fully connected. It
          is capable of generating duplicates as follows:

               1. A message is entered on node A.

               2. Node  A exports  the message to node B and node C placing
               the seen-by for A, B, and C in the message as it does so.

               3. Node  B sees that node D is not listed in the seen-by and
               exports the message to node D.

               4. Node  C sees that node D is not listed in the seen-by and
               exports the message to node D.

          At this  point node  D has  received the  same message  twice - a
          duplicate was  generated. Normally  a "dup-ring"  will not  be as
          simple as  a square.  Generally it  will be caused by a system on
          one end  of a  long chain  accidentally connecting to a system on
          the other end of the chain. This causes the two ends of the chain
          to become connected, forming a polygon.

          In FidoNet  this problem  is reduced somewhat by having "Regional
          Echomail Coordinators"  (RECS) that try to keep track of Echomail
          connections within  their regions  of the  world. A  further rule
          which is  followed is  that only  the RECS  are allowed  to  make
          inter-regional connections for the larger conferences. In return,
          the RECS  have established  a very  efficient topology which gets
          messages from  coast to  coast, and onto over 200 systems in less
          than 24  hours. If  no one were willing to follow the rules, then
          this system  would collapse,  but due to the excellent efficiency
          it has remained intact for over a year.


     Why a PATH line?

          As was  previously mentioned,  the PATH  line is a new concept in
          Echomail. It  stores the  net/node numbers  of each system having
          actually processed  a message.  This  information  is  useful  in
          correcting the  biggest problem  encountered by  nodes running an
          Echomail compatible  system - the problem of finding the cause of
          duplicate messages.  How does  the  PATH  line  help  solve  this
          problem? Take the following path line as an example:

               ^aPATH: 107/6 107/312 132/101

          This  shows  the  message  was  processed  by  system  107/6  and
          transferred to  system 107/312.  It further  shows system 107/312
          transferred the  message to  132/101, and  132/101  processed  it
          again. Now take the following path line as the example:

               ^aPATH: 107/6 107/312 107/528 107/312 132/101

          This shows  the message  having been processed by node 107/312 on
          more than one occasion. Based upon the earlier description of the
          'information control'  fields in  Echomail messages, this clearly
          is an  error in  processing (see  the section  entitled  "How  it
          Works"). This  further shows   node  107/528 as  the  node  which
          apparently processed  the message  incorrectly. In  this case the
          path line  can be  used to quickly locate the source of duplicate
          messages.

          In  a   conference  with  many  participants  it  becomes  almost
          impossible to  determine the  exact topology used. In these cases
          the use of the path line can help a coordinator of the conference
          track any  possible breakdowns in the overall topology, while not
          substantially increasing  the amount  of information transmitted.
          Having this  small amount of information added to the end of each
          message pays  for itself very quickly when it can be used to help
          detect a  topology  problem  causing  duplicate  messages  to  be
          transmitted to each system.

-30-
