Dynamic screen memory for FujiNetChat
-------------------------------------

This *doesn't exist* yet, even if I speak of it in the present tense
in this document!

Goals:

- Maximize use of available memory for scrollback. This means variable
  sized screens, not fixed to a given address (though they *will* be
  fixed to a given bank in extended RAM).

- Support at least 15 screens. Maybe more. There will be *some* limit,
  anyway.

- Don't waste memory: if you only use a few screens, you shouldn't have
  a bunch of memory reserved for the screens you don't use. You'll only
  pay for what you use.

- Compatible with 48K, 64K (using RAM under OS), 130XE, and at least
  256K or 320K upgraded memory. It will be able to use as much RAM as
  you have, up to some limit (U1MB?).

- *Not* require the 130XE's separate ANTIC access mode. While this
  might be helpful, there's a large installed base of expanded 800XLs
  that don't support it. Plus, the ANTIC bit in PORTB is one of the
  bits that might get repurposed as a bank bit, for machines with
  loads of RAM.

- Able to use the extra 6.4K you get if you boot without DOS (straight
  from the FujiNet), or any space between MEMLO and $2000 (if you boot
  a DOS with a small memory footprint). *Maybe* also include whatever
  space there is between the end of the client's code/data and the
  start of cc65's stack (basically, the entire "heap" area). Whether
  or not this happens depends on how large the client code gets: if
  it's less than 1K free, it's not really worth the trouble.

- Able to use 4K of the RAM under the OS in an XL/XE machine. 3K of this
  would be at $D800-$E3FF: the floating point pack and the font. Another
  1K is at $CC00 (the international font).

- A *single* executable that works on all of the above (no special
  fnchatxe.xex for extended RAM), meaning it has to detect the
  amount of RAM and which extended banks exist.

- Adding new text to a screen won't require scrolling all the existing
  text up by moving it in memory, so it'll be *fast*.

- Config options to disable some of the memory. If you're running
  SpartaDOS, you need a way to tell FujiNetChat not to use the extra
  RAM under the OS. If you use a ramdisk that only uses some of your
  extended memory banks, you need a way to tell FujiNetChat to leave
  those banks alone.

Terms:

Bank - Hopefully you already know the concept of bankswitching. In
  this document, I number the banks 0 (for the base 64K) through...
  I suppose up to 255, since I'll use a byte to store the bank
  number. This means we might support up to 4MB of memory, if
  such a thing exists for the Atari. An unexpanded 130XE has
  5 banks, which I number 0 through 4. A Rambo 800XL has 13
  banks, numbered 0 to 12.

Chunk - a 40x23 (or smaller) piece of a screen (can be thought of as a
  "display window"). At any given time, the screen can only be
  displaying one chunk. Normally, this is the bottom-most one, where
  new text is printed as it comes in. The top-most chunk of a screen
  can be fewer than 23 lines (e.g. if there are 30 lines, you get one
  23-line chunk and one 7-line one).

End Marker - a "special" line whose pointer is set to point to itself,
  and whose data is all spaces. This line is shared by all screens, and
  actually be displayed (e.g. if the screen is less than 23 lines,
  they are displayed at the bottom, then the rest of the GR.0 lines
  are all end markers).

Screen - a scrollable (backwards and forwards) area that displays text,
  like FNChat already uses. In this scheme, each screen will have a
  pool number, a line count, a scrollback line count (0 = not scrolled
  up) and a pointer to the first (bottom-most) line. If the pointer
  points to the End Marker line, that means no lines are assigned to
  the screen yet (it was just created and hasn't been written to yet).
  Otherwise, it points to the address of the *bottom-most* line.
  A null pointer (0) would be an error, and should never exist.
  Also the screens will have a title and a status, like the current
  ones do.

Scrollback - as a noun: the part of the screen that's not normally
  visible. As a verb, the act of making that part visible. Scrolling
  will generally be done one chunk at a time, though there's no
  reason there couldn't be a "one line at a time" scrolling mode.

Screen height - the total number of lines in a screen (includes all
  its chunks). The minimum height of a screen, upon creation, is
  actually 0: it has no lines until it's written to.

Line - 42 bytes of memory that store 40 characters (one GR.0 line) of
  text, plus a 2-byte pointer to the next line (in the screen, or in
  the free line list). Lines in a screen are stored in a linked list
  (each points to the next), in reverse order of how they're displayed
  (bottom-most points to the 2nd-to-bottom, etc, and the top one in
  a screen points to the End Marker). Lines in the free list are also
  stored as a linked list, associated with the pool, not any screen.
  Lines can cross 4K boundaries because they're not directly displayed
  by ANTIC.

Free Line - a line that isn't being used by any screen. All the
  free lines in a pool are a linked list: initializing the pool
  sets up the pointers in all the lines. Closing a screen releases
  all its used lines into the pool's free line list.

Pool - A (possibly non-contiguous) region of memory available for lines.
  Each pool has a bank number, a count of unused lines, and a linked
  list of the unused lines in the pool. Pool 0 is in main memory,
  is always at least 16K, and can be up to 26880 bytes in size (using
  4K of under-the-OS RAM for an XL/XE, plus any space from MEMLO
  to $2000 if DOS is not booted, or if a DOS with low MEMLO is
  booted). The other pools consist of entire banks, 16K apiece, one
  per bank. Each screen is created in one pool and cannot be moved to
  another pool.

Initialization:

At startup, FujiNetChat detects the amount of memory (number of
extended banks) and creates a pool for each bank. Bank 0's pool can
include extra memory beyond 16K: whatever's not in use by the client,
or by DOS (if you booted with one, even). Also pool 0 might have some
of the RAM under the OS on XL/XE (because FujiNetChat doesn't use the
two 1K ROM fonts or the 2K math pack, so we get 4K "for free"). All other
pools (1 and up) will be 16K.

At startup, the [server] and [private] screens will be created. Also
autojoin channels/queries will each get a screen created.

When creating screens, they're assigned to pools in round-robin
style. Suppose we have 5 banks of memory (0 through 4), with one pool
each. The first screen is created in bank 0. The second screen will be
created in bank 1, 3rd in bank 3, etc. After all pools have one screen
in them, the next screen creation will use pools 0 again (so now
we have two screens in one pools). This can continue until we reach
whatever the limit is: 15 screens? 20? Maybe calculated based on the
number of banks, so we can guarantee that when all memory is in use,
each screen will have a minimum of 23 lines. With a 130XE, this would
be a stupid amount of screens: 17 per bank for the 4 extended banks,
and at least 17 for bank 0 (so 85 of them, that's too many). Maybe
limit it to 28, which guarantees each screen can be 3 chunks (69
lines) tall?

The reason for the round-robin creation: Suppose you're only going to
use 3 screens (server, private, and one channel). It makes more sense
for each of those 3 to be in its own bank, so each one can grow to
16K (around 390 lines, or ~17 chunks). If we created them all in bank 0,
they'd compete with each other for memory, which is silly when there's
plenty of free RAM in the other (unused) banks.

Writing text to a screen consists of...

- Find a free line in the screen's pool (see below).
- Fill the line with the new text.
- Make the line's 'next' pointer point to the screen's 'head' pointed to.
- Make the screen's 'head' point to the new line.

The lines in a screen are stored as a textbook example of a linked list.

What happens if we're displaying a screen in one bank, and need
to add text to a screen that's in a different bank? Well, we have
to bankswitch to write to the new bank. But doing so will make that
bank replace the screen memory for the screen we were looking at. So,
bankswitching and writing has to take place during the vertical blank
interval, when ANTIC is done displaying the screen and no longer
reading from RAM.

*Careful*, without writing the code I don't yet know if there's enough
time in one VBLANK to write a huge (up to 510 bytes) IRC message in
one go. It'll be OK if it takes more than one frame, but not more than
maybe 4 or 5 (that'll make the app feel sluggish). Assembly optimization
is a must for this. Also, we don't have to wait for the VBLANK interrupt
to happen: we can start after the last visible scanline and work through
until just before the start of the first visible scanline on the next
frame.

Finding a free line:

- See if there's a free line in the pool (if the head of the free lines
  list is not null, and/or if the free lines count is not 0).
- If you find one, add it to the screen (see above), and remove it
  from the free list (make the pool's free_list point to whatever
  the line's 'next' pointed to). Also decrement the free lines count for
  the pool.
- If there isn't a free line, we have to 'steal' one from another
  screen in this pool. For now, just take the one with the most
  lines and steal its top line, and add it to the screen we're writing to.
  This means the screens "compete" and eat each other :)

'Stealing' means the screens will automatically balance, to some degree.
If you have 3 active channels in one bank, during busy periods the 3
screens will tend to be around the same size. If one channel goes quiet,
the other 2 will steal lines from it until it gets down to 23 lines,
then they'll start stealing from each other instead. Maybe the minimum
should be 46 or 69 lines (2 or 3 chunks), to avoid the scenario where
you leave the Atari connected while you sleep, and 2 busy channels ate
all the 6+ hour old text in the other, that came in an hour after you
went to bed?

Closing a screen:

When a screen is closed, its lines are returned to the free lines list
in the pool. Since they're already a linked list, all that's needed is
to add the screen's 'head' to the end of the pool's free lines list,
and add the screen's line count to the pool's free line count.

Displaying the screen:

All the screens share the same display list, which lives in main
memory. This display list is 23 lines of GR.0 text, with an LMS at the
top, and 2 lines at the bottom which are either the edit box (2 GR.0
lines) or the status bar (one GR.0 line, one GR.1). The top LMS points
to a 920-byte (23 line) buffer in main (non-bankable) memory. This
buffer gets line data copied to it from the screen's lines.

Switch to the screen's bank, then...

Starting at the screen's 'head' line, walk the linked list [scroll
height] times (0 if we're not scrolled up, 23 if we're scrolled up
one chunk, etc). This brings us to the first line that should be
displayed.

When we've walked to the first (bottom-most) line to display (which will
be the 'head' one, if we weren't scrolled back), copy its data to the
bottom line of the buffer, then move on to the next line of the screen
and the next line of the buffer.

Repeat until we've done 23 lines. Remember that the last (oldest,
topmost) line of the screen points to the end marker, which points
to itself. If we're scrolled up past the last real line, we just get
copies of the marker line data in the top lines of the buffer (which
will likely consist of an inverse ~ and 39 blanks, at least during
testing, so we can see it).

Scrolling the screen up (or down) is just a matter of setting the
screen's scroll height. It should *never* be set higher than the
screen's height, and probably the UI will increase by 23 for each
press of Start+Up. So if we have 30 lines, counting from 1 (top) to 30
(bottom), we're normally looking at lines 8 to 30. Scrolling up by
one chunk will show lines 1 to 7 at the bottom, then the rest of the
display (the top 2/3s or so) will all be the End Marker line, which
appears blank. At that point, it won't be possible to scroll again:
We're at 23, adding another 23 would exceed the height of 30 lines,
so the attempt is just ignored.

One wrinkle: while writing new text to the current-displayed screen,
when scroll height is 0 (we're not scrolled back), both the screen's
line and the bottom line of the buffer need to get the data written to
them. This avoids recopying the whole 23-line screen to the buffer on
every printed character (which would be painfully slow). We won't have
to recreate the whole buffer until we reach the end of the line.

Refreshing the buffer involves writing 960 bytes to it, so it won't be
all that fast, but it'll be faster than the current fixed-screen code
(which scrolls 50 or 73 lines every time, 2000 or 2920 bytes, even in
screens that aren't currently displayed).

It would potentially be faster to do without the buffer, and just have
a DL with an LMS for each line... but then we'd be displaying screen
memory straight from extended banks, and we'd have to have some fancy
synchronization going on when we're displaying a screen in one bank
and writing to one that lives in another bank (would have to copy data
only during the time ANTIC isn't reading screen memory). I may revisit
this at some point, but using a buffer in main memory is much simpler
(though it costs about 1K).

Memory layouts for typical machine sizes...

- A 48K 800 will only have pool 0, which will be either 16384 bytes
or ~390 lines (if DOS is booted) or 22784 bytes or ~542 lines without
DOS. This is enough to have 7 or 8 screens without about 3 chunks
(69 lines) apiece, which is better than the exising fixed-buffer code
manages.

- A 64K 800XL/1200XL/65XE/XEGS will only have pool 0, which will be
either 20480 bytes (~487 lines) with DOS, or 26880 bytes (~640 lines)
without DOS.

- A 128K 130XE will have pool 0 as the XL does, plus another 4 pools
of 16K each. That's 2048 lines (with DOS), which could be organized
as e.g. 10 screens of 200 lines each, or 16 with 128 lines each, or 20
with 100 lines each.

- A 256K upgraded XL with DOS will have 132K (135168), or about 3200
lines. For 512K, roughly twice that. To really take advantage of 256K,
you'll actually have to create enough screens so that all the banks
are used (16 screens with 16K each except the one in pool 0 gets
more). 512K will allow 32 screens, each in its own pool, with close
to 400 lines of scrollback in each (~17 chunks per screen!)


Milestones: things that will have to happen to make this a reality.

1. First and foremost, FNChat needs to go on a diet! Lots of stuff
   to rewrite in asm, to shrink it down. Currently, it's right at
   21K, but it's really more because rx_buf, tx_buf, and the font eat
   another 2K (in the screen memory area, in lieu of an 8th screen),
   plus all 3 display lists. No point optimizing the existing screen
   code for size, though, it's going to be replaced. See doc/diet.txt
   for details/progress on this.

   Status: ongoing, but done enough for now.

2. Split the code/BSS/etc into high and low segments, so it lives from
   $2000 to $3FFF (low, 8K) and $8000 to $BFFF (high, 16K). This
   puts the primary screen memory area right where the XE bankswitching
   needs it to be, and gives a *total* size of 24K for the client,
   including the font, display lists, and all buffers (except ones
   located in very low memory, $400-$6FF; these are the config and the
   editbox, and can stay where they are).

   The font and buffers have to be moved out of the new screen address
   space (the banking window at $4000). I'm thinking the font will be
   just below the bank area at $3A00, 2 512-byte buffers below that
   at $3600, display lists below that, etc. Leave about 6K for 'low
   code', the cc65 stack, and the BSS.  Everything else will be in a
   'high code' (and possibly 'high BSS') segment, from $8000 to $BFFF.

   Status: DONE!

3. Rip out all the existing screen code and replace it with a simplified
   form of the new scheme. To start with, only 1-5 pools in bank 0, but
   the display list modification code can be completed. The rest of
   the code (especially irc.c) is gonna need changes, because it
   "knows" that there are "always" 7 screens. This stage includes
   defining new hotkeys for screen numbers above 7, and making the
   status bar variable sized (show only the number of screens we
   have enough RAM for, based on the minimum size being 23 or 46
   lines). Better yet, don't even show the inactive or unused (grey
   dot or number) at all, which would free up a color register.
   Figure out some way to handle having more screen numbers than we
   could fit in a GR.1 line. 2 lines? Use a GR.0 line (what, PMG and a
   DLI for the colors? rather not)?

   Status: barely started...

4. Learn more than I currently know about bankswitching (I know the
   130XE, but what's the difference between a Rambo and a Compy Shop
   upgrade? I *think* I know how to detect all the available banks,
   but what about when the self-test and/or BASIC bits are being
   used for bank bits instead? Do I want to even try to support
   the Axlon and Mosaic upgrades for the 800?)

5. Do a version that supports a 130XE (4 extra banks only). Get it
   well tested, fix the inevitable issues that are going to happen.
   This version should also still be usable on 48K/64K. This will
   probably involve adding the memory detection to the config
   segment (along with copying the OS to RAM if possible). It'll
   deposit the pools array somewhere in screen memory, and the
   client will memcpy() that to its pools[] (before scr_init()
   is called). No point keeping all that startup code in memory
   the whole time the client runs.

6. Add support for more banks (detection and use).

TODO: should there be a way to move a screen from one pool to another?
Imagine, user with 128K opens a bunch of screens, then happens to
close all the ones in e.g. bank 2. The screens in banks 0/1/3/4 are
still competing for lines, and we have a whole empty bank we don't
use until another screen gets opened... it would be an expensive
operation, and when exactly would we check for it?

Rest of the file is C structs that define the stuff above. This is
just hypothetical code (final implementation may look different).

/* if each pool is 16K, that's 512K, not bad. however, overhead. maybe
   limit this to something like 20 (128K extended = 16, plus the big
   pool in main bank, plus the potential smaller pools at MEMLO and
   $d800. */
#define MAX_POOLS 32

/* with 512K, we get one screen per pool.
   with 256K, up to 2 screens per pool.
   with 128K, up to 4.
   with 64K, we only get 1 large pool.
   with 48K, we only get 1 pool.
*/
#define MAX_SCREENS 32

/* 42 bytes per line */
typedef struct line_s {
  struct line_s *next;
  char data[40];
} line_t;

/* the end marker line is a line_t, but it lives outside of any pool and
   has a 'next' pointer that points to itself. */

/* sizeof(screen_t) is 33 bytes... */
typedef struct {
  char title[25];
  char status;
  char pool; /* this could be a pool_t * instead */
  int line_count; /* can be above 255 */
  line_t *line_list; /* head of a linked list */
  int scrollback_pos; /* also can be >255 */
} screen_t;

screen_t screens[MAX_SCREENS]; /* array is 1023 bytes */

typedef struct {
  // I don't think we even need these:
  // u16 start; /* 0 = not in use */
  // u16 end;
  char screen_count; /* maybe set to $ff for "not in use" */
  // int line_count; /* maybe better to use line count instead of
                        screen count to decide which pool to create
                        new screens in? */
  u8 bank; /* probably this is just the PORTB value */
  line_t *free_list; /* when this is null, the pool has no free lines */
} pool_t;

/* this array is sizeof(pool) * 9, so 288 bytes */
/* the code that builds this array (detects extended ram too),
   will live in the config segment. */
pool_t pools[MAX_POOLS];

/* this function is responsible for counting the lines and arranging
   them in a linked list. it zeroes the 40 character bytes in each
   line and sets the other 2 as a pointer to the next. */
void add_pool(u8 bank, u16 start, u16 end);
