On open, CLOSED and MODULEs
---------------------------

Author:		Jos Visser <josv@osp.nl>
Version:	0.1 (Thu Dec 19 14:07:26 CET 2002)
OpenComal:	0.2.7-pre1

License
-------

This document is copyright (c) 2002 Jos Visser <josv@osp.nl>. You
are allowed to distribute, link and print this document as you see fit,
provided that you do so in its entirety. Modifications are not allowed.

0. OpenComal
------------

All examples in this document work with OpenComal version 0.2.7-pre1 or
later. For more information on OpenComal see the OpenComal web site at
http://www.josvisser.nl/opencomal.


1. Introduction
---------------

Ever since its inception Comal contained the concepts of closed 
procedures and functions. The purpose of closed procedures and functions
is to separate the global and local variable spaces so as not to pollute
the set of global variables with variables and values that are only
meaningful within a routine's (PROCedure or FUNCtion) context. Closed
routines are Comal's way to escape from Wizard's second rule (the rule 
of unintended results); Comal's conceptual parent (Basic) did not have
an similar concept and many programming errors were caused by
uncoordinated modification of variables.


2. The global variable space
----------------------------

When a Comal program is run it has a global variable space and (unless
changed by the features described in the following sections) all
variable references are resolved to the global variable space:

$ list
       10 a:=1
       20 aap
       30 PRINT a
       40 
       50 PROC aap
       60   PRINT a
       70   a:=2
       80 ENDPROC
$ run
1
2

The results speak for themselves


3. CLOSED
---------

By declaring a routine CLOSED two things happen:

- Variables in the global variable space become invisible
- New variables are created in a local variable space. They are removed
  when the routine ends and their existence does not influence any
  variables in the global (or other local) variable space:

$ list
       10 a:=1
       20 aap
       30 PRINT a
       40 
       50 PROC aap CLOSED
       60   PRINT a
       70   a:=2
       80 ENDPROC
$ run
       60   PRINT a
Error 19: "Unknown identifier a" at line 60
$ del 60
Adding/Modifying/Deleting the current execution line has inhibited CONtinuation
$ run
1

As you can see the reference to "a" in line 60 results in a run time
error. The "a" introduced in line 70 is local and does not modify the
"a" in the global variable space.


4. IMPORT
---------

The complete separation of local and global variable spaces is in some
occasions counterproductice. For instance, because Comal does not know
constants programmers regularly abuse variables to hold constant values
that are global by nature. In other cases certain data just has to be
kept globally because it plays an important role throughout the program.

For these cases Comal provides the IMPORT statement. IMPORT introduces
variables from another (most often the global) variable space into the
local space:

$ list
       10 a:=1
       20 aap
       30 PRINT a
       40 
       50 PROC aap CLOSED
       60   IMPORT a
       70   PRINT a
       80   a:=2
       90 ENDPROC aap
$ run
1
2

In some Comal implementations the CLOSED concept encompasses procedures
and functions too. In these Comals you should IMPORT any procedures and
functions that you want to call from your closed routine. In OpenComal
procedures and functions are global entities that need not be imported
(a warning is given if you do).


5. LOCAL
--------

OpenComal extends the Comal language with the LOCAL keyword. LOCAL
creates a local variable. It may only be used in open routines (since it
is unneccesary in closed routines):

$ list
       10 a:=1
       20 aap
       30 PRINT a
       40 
       50 PROC aap
       60   LOCAL a
       70   PRINT a
       80   a:=2
       90 ENDPROC aap
$ run
0
1

LOCAL can also be used to introduce local string variables and arrays.
Its syntax is just like DIM:

$ list
       10 DIM a$(20) OF 2
       20 a$():="x"
       30 aap
       40 PRINT a$()
       50 
       60 PROC aap
       70   LOCAL a$(10) OF 2
       80   a$():="y"
       90   PRINT a$()
      100 ENDPROC aap
$ run
y y y y y y y y y y 
x x x x x x x x x x x x x x x x x x x x 


6. Calling open routines from closed ones
-----------------------------------------

OpenComal relaxes restrictions found in other Comals related to calling
open routines from closed ones. In OpenComal, one *can* call an open
routine from a closed one:

$ list
       10 a:=1
       20 aap
       30 PRINT a
       40 
       50 PROC aap CLOSED
       60   a:=2
       70   tijger
       80   PRINT a
       90 ENDPROC aap
      100 
      110 PROC tijger
      120   PRINT a
      130   a:=3
      140 ENDPROC tijger
$ run
1
2
3

As this example show, the "a" referred to in procedure "tijger" is the
global one. The "a" in the closed procedure "aap" is not changed by the
assignment in line 130 whereas the global "a" initialised in line 10
is!

Allowing closed routines to call open ones removes an important
restriction from Comal. In my (and thus OpenComal's :-) view routines
are global entities that package a certain functionality. Calling
routines should be mostly context free, that the routine's developer
decided to make a routine closed should not make any difference to the
caller.


7. Variable space stack
-----------------------

When routines start calling each other a stack of variable spaces comes
into existence. For instance take the following case:

$ list
       10 a:=1
       20 aap
       30 
       40 PROC aap CLOSED
       50   a:=2
       60   tijger
       70 ENDPROC aap
       80 
       90 PROC tijger
      100   LOCAL b
      110   b:=2
      120   SYS listvars
      130 ENDPROC tijger

When control reaches line 120 there are three variable spaces into
existence:

1) The global variable space
2) The closed environment of "aap"
3) The local (but essentially open) environment of "tijger"

The internal OpenComal command "SYS listvars" dumps the variable
environments that exist at that point:

$ run
Symbol environment:        90 PROC tijger
  Item: b (is Variable) Value: 2
Symbol environment:        40 PROC aap CLOSED
  Item: a (is Variable) Value: 2
Symbol environment: Global
  Item: a (is Variable) Value: 1


8. Named IMPORT
---------------

Since OpenComal is at all times aware of the actual variable space stack
it can (and does) allow an IMPORT to specify from which environment a
variable should be imported.

Look at the following example:

$ list
       10 a:=1
       20 aap
       30 PRINT a
       40 
       50 PROC aap CLOSED
       60   a:=2
       70   tijger
       80   PRINT a
       90 ENDPROC aap
      100 
      110 PROC tijger
      120   IMPORT aap: a
      130   PRINT a
      140   a:=3
      150 ENDPROC tijger
$ run
2
3
1

The execution of this program proceeds through lines 10, 20, 60, 70 and
120. At that time, two variables "a" exist: one in the global variable
space and one in the closed environment of "aap". Without qualification
an "IMPORT a" would import "a" from the global environment. However,
because the programmer specified "IMPORT aap:a" the "a" IMPORTed is the
one from the (calling) closed environment of "aap". The assignment in
line 140 modifies that instance of "a" as we can see from the result of
the PRINT statement in line 80.

Named IMPORT provides a powerful mechanism to break the CLOSEDness of a
routine. It also introduces a powerful dependency on the calling
sequence of routines. In the example outlined above "tijger" *must* be
called by someone who called (or is) "aap". If you break this implicit
dependency a runtime error occurs:

$ tijger
      120   IMPORT aap: a
Error 30: "Environment not found" at line 120

All these are strong reasons for disallowing named IMPORT. However, as
we will see later, named IMPORT can be necessary for if and when we
start nesting routines.


9. STATIC
---------

A feature that exists in other languages and that I have included in
OpenComal too is that of local variables that retain their value from
call to call. These variables are called static variables and are
introduced in a procedure through the STATIC keyword (which has the same
syntax as LOCAL):

$ list
       10 TRACE on
       20 a:=100
       30 aap
       40 aap
       50 PRINT a
       60 
       70 PROC aap
       80   STATIC a
       90   PRINT a
      100   a:+1
      110   tijger
      120 ENDPROC aap
      130 
      140 PROC tijger
      150   STATIC a
      160   PRINT a
      170   a:+10
      180 ENDPROC tijger

To get a decent idea of what is going on I activated execution line
tracing:

$ run 
       10 TRACE on
       20 a:=100
       30 aap
       80   STATIC a
       90   PRINT a
0
      100   a:+1
      110   tijger
      150   STATIC a
      160   PRINT a
0
      170   a:+10
      180 ENDPROC tijger
      120 ENDPROC aap
       40 aap
       80   STATIC a
       90   PRINT a
1
      100   a:+1
      110   tijger
      150   STATIC a
      160   PRINT a
10
      170   a:+10
      180 ENDPROC tijger
      120 ENDPROC aap
       50 PRINT a
100
       60 
       70 PROC aap
      130 
      140 PROC tijger

STATIC variables are automatically LOCAL to the routine that defines
them. For instance if we change line 150 to "IMPORT a" and run again the
following output is produced:

$ run
0
100
1
110
120

For those that really want to make a mess of things, we can use a named
IMPORT to import a STATIC variable into a local environment:

$ list
       10 a:=100
       20 aap
       30 aap
       40 PRINT a
       50 
       60 PROC aap CLOSED
       70   STATIC a
       80   PRINT a
       90   a:+1
      100   tijger
      110 ENDPROC aap
      120 
      130 PROC tijger
      140   IMPORT aap: a
      150   PRINT a
      160   a:+10
      170 ENDPROC tijger
$ run
0
1
11
12
100

The message here is that features that are introduced for a good cause
can be misused to create chaotic programs (Wizard's Second Rule: the
rule of unintended results). But, as Larry Wall aptly stated: "Good
programmers can write assembler in any language" :-).


10. Nested routines
-------------------

OpenComal allows procedures and functions to be nested. This feature was
introduced in order to allow large and complex procedures to be
modularised internally and then be loaded on demand through an EXTERNAL
procedure or function. For a good example of this look at the "verbaal"
sample programs that use both nested and external routines.

Nesting routines adds extra complexity to the issues of variable spaces
and open/CLOSED/IMPORT. For instance look at this sample program:

$ list
       10 a:=1
       20 aap
       30 PRINT a
       40 
       50 PROC aap CLOSED
       60   
       70   tijger
       80   PRINT a
       90   
      100   PROC tijger
      110     PRINT a
      120     a:=100
      130   ENDPROC tijger
      140   
      150 ENDPROC aap

First of all it should be noticed that nesting changes the rules of 
calling routines. The procedure "tijger" is nested in "aap"; one of the
consequences of this is that "tijger" can only be called from within
"aap"! If you would try and call "tijger" from the main program the
OpenComal program structure scanning engine (invoked before RUN or
through the SCAN command) would complain:

$ 15 tijger
$ scan
       15 tijger
PROCedure tijger not found
$ del 15
$ scan

There are no conceptual limits to nesting of routines. OpenComal does
restrict nesting of routines for reasons of readability (to prevent the
automatic indentation of program listings to go through the roof).

When we run this program, the execution progresses through lines 10, 20,
70 on to 110. The interesting question then becomes whether the "a"
mentioned there is visible or not? On the one hand, "tijger" is an open
procedure (whatever that may mean in a nested routine) but it is part of
the CLOSED procedure "aap"...

Let's run the program and see what happens...

$ trace on
$ run
       10 a:=1
       20 aap
       60   
       70   tijger
      110     PRINT a
      110     PRINT a
Error 19: "Unknown identifier a" at line 110

The OpenComal interpreter declares that it does not know about "a"!

We are know (hopefully) ready to get a better understanding of what open
and CLOSED routines actually entail...

An open variable environment is one that allows unrestricted access to 
variables from its parent environment. In this example the procedure
"tijger" provides a new open environment upon call, which allows access
to variables from its parent environment, which is the procedure "aap".

In contrast, a CLOSED environment is one that does not allow access to 
variables from its parent environment.

So when the OpenComal interpreter finds the reference to "a" in line 110
it first examines the local environment of "tijger". It does not find an
"a" there, thus it progresses to the parent environment ("aap"). It does
not find an "a" there either. Since "aap" is CLOSED the search process
stops there. If "aap" would have been open, the search for "a" would
have progressed into the global (program) variable space. Try it!

One of the nice things of an interpreter is that we can modify the
environment and continue the program. 

$ a=99
$ con
      110     PRINT a
99
      120     a:=100
      130   ENDPROC tijger
       80   PRINT a
100
       90   
      100   PROC tijger
      140   
      150 ENDPROC aap
       30 PRINT a
1
       40 
       50 PROC aap CLOSED

In this example I assign the value 99 to "a" and CONtinue the execution
of the program. The PRINT statement in line 110 now prints the assigned
value and the program continues. Next comes the PRINT statement in line
80. Will it succeed? It apparently does, which raises the next question:
In which variable environment was the "a" assigned to from the command
line (conceptually in the procedure "tijger", since that was where the
interpreter halted) entered?

The (Open)Comal rules for the environment in which new variables are
entered are as follows:

- PROCedure and FUNCtion parameters exist in the local environment of
  the routine.
- LOCAL variables are entered in the routine's local environment (that's
  not much of a surprise I hope :-)
- STATIC variables are entered in a special per-routine static
  environment that survives from one call to another.
- IMPORTed variables are created in the local variable environment with
  a reference to their true home environment.
- Normal variables are entered in the first CLOSED environment found
  when moving outward from the point of assignment.
  The global variable environment is CLOSED by default, so that is
  ultimately where new variables are created.

In this example: Since "tijger" is open, the new "a" is introduced in
the local environment of "aap" (as it lives in the execution stack).
That's why "a" still exists when we return from "tijger"! When we
subsequently return from "aap" the "a" in the global variable space has
not been touched.


11. IMPORT and nested routines
------------------------------

Let's modify the example program somewhat. Suppose we add an "IMPORT a"
to line 65 and run again:

$ 65 import a
$ list
       10 a:=1
       20 aap
       30 PRINT a
       40 
       50 PROC aap CLOSED
       60   
       65   IMPORT a
       70   tijger
       80   PRINT a
       90   
      100   PROC tijger
      110     PRINT a
      120     a:=100
      130   ENDPROC tijger
      140   
      150 ENDPROC aap
$ run
1
100
100

The IMPORT statement introduces the global "a" in the local variable
space of "aap". This makes that "a" reachable for "aap" and all its
nested routines. The PRINT statements in line 110 and 80 refer to the
one global "a" and the assignment statement in line 120 modifies it (as
is shown by the PRINT statement in line 30).

However, what would happen if we would insert an IMPORT statement in
"tijger"?

$ list
       10 a:=1
       20 aap
       30 PRINT a
       40 
       50 PROC aap CLOSED
       60   
       70   tijger
       80   PRINT a
       90   
      100   PROC tijger
      110     IMPORT a
      120     PRINT a
      130     a:=100
      140   ENDPROC tijger
      150   
      160 ENDPROC aap
$ run
      110     IMPORT a
Error 30: "IMPORTable item "a" not found" at line 110

OpenComal tries to IMPORT a variable "a" from the parent environment
(i.e. from "aap"). There is no "a" in "aap", and "aap" itself is closed,
so OpenComal can not find the "a" to be imported.

If we really want to mess things up we can use a named IMPORT to
specifically declare which "a" we are talking about. In that case we
need to specify the special identifier "_program" to instruct OpenComal
to IMPORT from the global variable space:

$ edit 110
      110     IMPORT _program:a
$ run
1
       80   PRINT a
Error 19: "Unknown identifier a" at line 80
$ a=99
$ con
99
100

What happens here?

Line 10 sets the global "a" to 1. Execution then proceeds through lines
20 and 70 to line 110. There the global "a" is IMPORTed in the local
variable space of "tijger". Line 120 prints that global "a" and line 130
sets it to the value 100. We then leave procedure "tijger" and return to
line 80. There we try to print "a", but there is no "a" in the local
variable environment of "aap"! Program execution therefore stops with an
error message. 

If we then manually assign "a=99", an "a" is created in the local 
variable environment of "aap". We then CONtinue the program. The local
"aap" copy of "a" is printed and execution returns to line 30. There we
print the global copy of "a" which has been modified in the nested
routine "tijger" (line 130).

Here, named import provides us with a sort of escape mechanism to break
the strict scoping rules of Comal. Use it with care!


12. MODULEs
-----------

To aid in developing modular programs, UniComal introduced the concepts
of modules into the Comal language. OpenComal supports these modules
too. 

In OpenComal, a module is a package of procedures and functions from
which only selected (exported) procedures and functions can be called
from outside it. Put differently, a routine inside a module can only be
called by someone outside the module if that routine has been explicitly
exported (through the EXPORT keyword).

Modules are introduced with the MODULE keyword. A program can declare
its intention to use the exported routines in a module with the USE
keyword:

$ list
       10 USE aap
       20 f
       30 g
       40 
       50 MODULE aap
       60   
       70   EXPORT f, g
       80   
       90   PROC f
      100     PRINT "Hello from f"
      110   ENDPROC f
      120   
      130   PROC g
      140     PRINT "hello from g"
      150   ENDPROC g
      160   
      170   PROC h
      180     PRINT "hello from h"
      190   ENDPROC h
      200 ENDMODULE aap
$ run
Hello from f
hello from g

Procedure "h" is not exported and can thus not be called. If we try, an
error is thrown:

$ 35 h
$ run
       35 h
PROCedure h not found


13. MODULE variable access
--------------------------

Modules have their own static variable environment that is available to
all routines in the module. All variables that are in this module
environment are available to all routines in the module:

$ list
       10 USE a
       20 f
       30 f
       40 
       50 MODULE a
       60   n:=1
       70   EXPORT f
       80   PROC f
       90     PRINT "In f, n=",n
      100     n:+1
      110   ENDPROC
      120 ENDMODULE
$ run
In f, n=1
In f, n=2

Sort of hidden in this last example is that the initialization code in
line 60 is executed *before* the program executes. In fact, to help in 
initializing a module, all executable statements in the module are 
executed just before the program is run. Look at the following example:

$ list
       10 USE a, b
       20 FOR i:=1 TO 4 DO
       30   f
       40   g
       50 ENDFOR i
       60 
       70 MODULE a
       80   PRINT "Init a"
       90   n:=1
      100   EXPORT f
      110   PROC f
      120     n:+1
      130     PRINT n
      140   ENDPROC f
      150 ENDMODULE a
      160 
      170 MODULE b
      180   PRINT "Init b"
      190   n:=100
      200   EXPORT g
      210   PROC g
      220     n:+1
      230     PRINT n
      240   ENDPROC g
      250 ENDMODULE b
$ run
Init b
Init a
2
101
3
102
4
103
5
104

As you can see, the initialization statements in "a" and "b" are
executed before the main program starts. You will also see that the
order in which they are executed is not easily predicted. It is best not
to depend on the exact order of initialization.

The "n" variables of modules "a" and "b" are completely separate and not
available outside the module. Look at the following example:

$ list
       10 USE a
       20 f
       30 
       40 MODULE a
       50   n:=3
       60   EXPORT f
       70   PROC f
       80     n:+1
       90     g
      100   ENDPROC f
      110 ENDMODULE a
      120 
      130 PROC g
      140   PRINT n
      150 ENDPROC g
$ run
      140   PRINT n
Error 19: "Unknown identifier n" at line 140

This example shows another thing: In OpenComal routines in modules can
call routines from outside the module without any obstructions. You can
have a debate about whether this is advisable (and hence, should be
allowed) or not. My point of view is that it is certainly useful to be 
able to do so (e.g. to have callback routines from a module).
Furthermore (but, I admit, completely beside the point), OpenComal's
internal architecture makes this "feature" more or less automatically
available.

By the way, if we really want to mess things up, we can use a named
IMPORT statement in procedure "g" to get at the internal "n" of module
"a":

$ 135 import a:n
$ run
4

This is not advised, but possible! Preventing people from writing stupid
and hard-to-understand code is unfortunately impossible in a programming
language that is sufficiently powerful.


14. Additional comments on MODULEs
----------------------------------

(This section delves somewhat deeper than usual; you may skip it if you
are not interested).

OpenComal implements MODULEs as a special sort of closed procedures.
When a program is SCANned, all USE statements are collected and a search
is initiated for the MODULEs mentioned. Then all EXPORTs inside the
module are located and all exported functions are added to a global
table.

Then, just before "run", all executable statements in all loaded modules
are executed. The variable environment created by the initialization is
set aside (remembered).

When a procedure/function call is executed, the OpenComal interpreter
starts the search of the routine using the normal visibility rules:

- PROC/FUNC parameter visible in the current variable environment
- Inward->out search for the routine

If the routine is not found then the global table of exported routines
is consulted. If the routine is found there then the static module
environment is pushed onto the call stack and the call is executed.
