This is a supplement to the main Readme file. The reader should consult
that file for more information.


Table of Contents
-----------------

1. Multiprocessor Architectures vs. Workstation Clusters
2. iPSC/860
3. Paragon
4. CM5
5. Shared-memory Systems
	Solaris 2.3
	SGI 5.1
6. IBM SP2


1. Multiprocessor Architectures vs. Workstation Clusters
--------------------------------------------------------

	This software is based on (and compatible with) version 3.3. 

	In the workstation environment, there is usually one (or a few at
the most) task(s) and one PVM Daemon (pvmd) on each host. On a MPP
machine, however, only one pvmd runs on the front-end and it has to
support all the tasks running on the nodes. Node-to-node messages are 
sent directly across and very fast, but packets going to another 
machine (or between tasks not spawned together in the case of the CM5
and IBM SP2) must be relayed by pvmd and that link is slow, especially for
large messages.

	On a MPP machine tasks are always spawned on the nodes. If you
want to start a task on the front-end you have to run it as a Unix
process, which then connects to the pvmd by calling pvm_mytid().
However node processes not loaded by PVM are not allowed to
enrolled.

	The system supports on nodes of some MPP machines are limited.
For example if your program forks on a node, the behavior of the new
process is machine-dependent. In any event it would not be allowed
to become a new PVM task. 

	Inter-host task-to-task direct routing via a TCP socket connection 
has not been implemented on the nodes. Calling pvm_advise() on a node
has no effect. 

	PVM message tags above 999999000 are reserved for internal use.


2. iPSC/860
-----------

	This port was developed on an iPSC/860 with 128 nodes, using an
SRM as its front-end. Compiling and installing on a remote host may
require extensive knowledge of Intel software.


INSTALLATION

	Type "make" on the SRM front-end to build pvmd and libpvm3.a, and 
		make PVM_ARCH=CUBE
to build libpvm3pe.a on a machine (SRM or workstation) that has the 
icc/if77 i860 cross-compilers.

	To make the examples, repeat the above steps in the "examples"
directory but use ../lib/aimk instead of make. (Note that aimk ---
and pvmd, pvmgetarch, etc. --- is a sh script. You have to run them
under sh instead of csh. The i386 frontend running sysV3.2 does not 
understand #!, so you have to put a blank line at the beginning of
the scripts.)

	To build pvmd and libpvm3.a on a remote host (Sun/SGI) instead of
the SRM, try 
		make PVM_ARCH=I860
You may have to add a -L option to the definition of ARCHDLIB and a 
-I option to ARCHCFLAGS in conf/I860SUN4.def (or conf/I860SGI.def) 
to include the path of libhost.a and cube.h .



APPLICATION PROGRAMS

	Host (master) programs should be linked with pvm3/lib/I860/libpvm3.a 
and the SRM system library libsocket.a, node (slave) programs with 
pvm3/lib/ARCH/libpvm3pe.a and libnode.a .  Fortran programs should
also be linked with pvm3/lib/I860/libfpvm3.a (host program) or
pvm3/lib/I860/libfpvm3pe.a (node program).

	The programs in the gexamples directory have NOT been ported. The
user must modify the makefiles to link to the appropriate libraries.


BUGS AND TIPS

	1. On the SRM pvm is slow to start and it may sometimes time out and
print out the message "can't start pvmd", while in fact pvmd has already
been started. When this happens, just run pvm again and it'll connect
to the pvmd. If you start the pvm console on another machine and add the
I860 host to the configuration then this should not occur.

	2. To use PVM on a remote host you have to start "pvm3/lib/I860/pvmd"
manually in the background, because PVM has no way of knowing that your
workstation is being used as a front-end to the hypercube. PVM relies on
the shell script "pvm3/lib/pvmd" to determine the architecture of your
system, you can hack the script to make it do what you want.

	3. If PVM cannot obtain a cube of the size you specify because none
is available, it will return the error "Out of resources".

	4. The function pvm_spawn() will NOT return until all spawned tasks
check in with PVM. Therefore slave tasks should call pvm_mytid() before
starting any heavy duty computation.

	5. The only signal that can be sent to tasks running on the cube is
SIGKILL. Any other signals will be ignored.

	6. All spawned tasks should call pvm_exit() just before they exit.
Pvmd will release the cube after all tasks loaded into the cube have
called pvm_exit(). If any task attempts to continue execution after
it called pvm_exit(), it'll be terminated when the cube is released.

	7. There is a constant TIMEOUT in the file "pvmmimd.h" that controls 
the frequency at which the PVM daemon probes for packets from node tasks.
If you want it to respond more quickly you can reduce this value. 
Currently it is set to 10 millisecond.

	8. DO NOT use any native NX message passing calls in PVM or they
may interfere with pvm_send()/pvm_recv().

	9. Messages printed to "stderr" by node tasks do not end up in
the log file "/tmp/pvml.uid".

	10. The "argv" argument to pvm_spawn() is quietly ignored.



3. Paragon
----------

	This port has been tested on a Paragon XPS5 running R1.3. The support
and information provided by Intel engineers are gratefully acknowledged.


INSTALLATION

	Type "make" in this directory on the Paragon. To build on a cross-
compiler machine, you need to type
		setenv PVM_ARCH PGON
		make -e
The environmental variable PARAGON_XDEV must be set to the proper base 
directory in the later case.

	To make the examples, go to the "examples" directory and repeat the
above procedure but use ../lib/aimk instead of make.

STARTING THE PVMD WITH PEXEC

Many Paragon sites use the program "pexec" to manage partitions for 
application programs. The pvmd startup script has been modified to
use pexec, if it exists.  The environment variable NX_DFLT_SIZE should
be set in the user's .cshrc file and is the size (with a default of 1) 
of the partition that the pvmd will manage. The partition size can also be 
set by invoking the pvmd with the command
       % pvmd [pvmd options] -sz <part_size> 

The "pexec" program should exist in /bin/pexec. If it exists in another
location, modify the  pvm3/lib/pvmd startup script to point at the
site-specific location or have the system administrator make the 
appropriate link.

If a PGON is the first ("master") host in the virtual machine, the
pvmd should be started first, and then the console may be started:
       % pvmd -sz <partsize> & 
       % pvm
Other hosts, including other Paragons, may then be add using the console.
       pvm> add other_pgon
       pvm> add my_workstation
       
APPLICATION PROGRAMS

    Host (master) programs should be linked with pvm3/lib/PGON/libpvm3.a
and the system library librpc.a, node (slave) programs with
pvm3/lib/PGON/libpvm3pe.a, libnx.a, and librpc.a .  Fortran programs should
also be linked with pvm3/lib/PGON/libfpvm3.a .

	The programs in the gexamples directory have NOT been ported. The
user must modify the makefiles to link to the appropriate libraries.

NATIVE MODE COLLECTIVE OPERATIONS
    
    If possible, native mode collective operations such as gsync,
gisum, and gdsum are used for group operations.  The native operations
will be used if all of following hold 
      1) A native collective operation exists on the Paragon
      2) All the nodes in the paragon compute partition are participating
         in the collective operation (pvm_barrier, pvm_reduce, etc.)
      3) The group has been made static with pvm_freezegroup

If all three conditions do not hold, the collective operation still functions
correctly, but does not use native operations. If all the PGON nodes are part
of a larger group, the native operations are used for the PGON part of the 
collective operation. 

BUGS AND CAVEATS

	1. Tasks spawned onto the Paragon run on the compute nodes by default.
Host processes run on the service nodes and should be started from a Unix
prompt.

	2. By default PVM spawns tasks in your default partition. You can use
the NX command-line options such as "-pn partition_name" to force it to
run on a particular partition or "-sz number_of_nodes" to specify the
number of nodes you want it to use. Setting the environmental variable 
NX_DFLT_SIZE would have the same effect. For example starting pvmd with
the following command
		pvmd -pn pvm -sz 33
would force it to run on the partition "pvm" using only 33 nodes (there
must be at least that many nodes in the partition).

	3. The current implementation only allows one task to be spawned on
each node.

	4. There is a constant TIMEOUT in the file "pvmmimd.h" that controls 
the frequency at which the PVM daemon probes for packets from node tasks. 
If you want it to respond more quickly you can reduce this value. 
Currently it is set to 10 millisecond.

	5. DO NOT use any native NX message passing calls in PVM or they
may interfere with pvm_send()/pvm_recv().

	6. Messages printed to "stderr/stdout" by node tasks do not end up in
the log file "/tmp/pvml.uid".

	7. PVM programs compiled for versions earlier than 3.3.8 need to
be recompiled.  A small change in data passed to node tasks on startup
will cause earlier programs to break. 


4. CM5
------

	This port was developed on a 32-node CM5 running CMOST v7.2 and
CMMD3.0. We gratefully acknowledge the supports and information provided
by engineers of Thinking Machines Corp.


INSTALLATION

    Type "make" in this directory.

	To make the examples, go to the "examples" directory and type
		../lib/aimk default


APPLICATION PROGRAMS

    Host (master) programs should be linked with pvm3/lib/CM5/libpvm3.a .
Node (slave) programs need to be linked to pvm3/lib/CM5/libpvm3pe.a and 
joined with pvm3/lib/CM5/pvmhost.o and pvm3/lib/CM5/libpvm3.a using the
CMMD loader "cmmd-ld". Fortran programs should also be linked with 
pvm3/lib/CM5/libfpvm3.a .

    The programs in the gexamples directory have NOT been ported. The
user must modify the makefiles to link to the appropriate libraries.


BUGS

	1. The function pvm_kill() does not behave like a Unix kill: it
will delete the target task from PVM's task queue but doesn't actually
kill the task. 

	2. Signals sent by pvm_sendsig() are simply discarded, except for
SIGTERM and SIGKILL, which are equivalent to calling pvm_kill().

	3. You're not allowed to spawn more tasks than the number of nodes
available in your partition in one call to pvm_spawn().

	4. If you kill the node tasks from PVM (e.g., using pvm_kill), the
node processes will dump core files into your HOME directory. I don't
consider this to be a PVM bug.



5. Shared-memory Systems
------------------------

	These ports achieve better performance by placing the message buffers
in shared-memory. Tasks on the same host exchange messages by mapping
each other's buffers into their own address space.


BUGS

	1. Messages are sent directly, there is no need to call
pvm_setopt(PvmRoute, PvmRouteDirect). The sender will block if the
receiving box is full. It will also block if its own outgoing message 
buffer is full. This could lead to deadlocks.

	2. The buffer size used by PVM, SHMBUFSIZE, is defined in 
pvm3/src/pvmshmem.h, with a default of 1MBytes. Your message must fit
in the buffer. If your program uses large messages, this parameter MUST
be increased accordingly. Note that the default system limit on shared-
memory segment size (1 MB on Sun's) may have to be raised as well.
A large number of small messages can also fill up the buffer if they
are not received in time, and the performance degrades considerably
when this happens. 

	3. You can change the buffer size by setting the environment
variable PVMBUFSIZE before starting PVM. This would have the same
effect as changing SHMBUFSIZE and recompiling PVM.

	4. When a task sends a message to another task, only the pointer
is passed. The receiving task should call pvm_freebuf() to "destroy"
the receive buffer after the data have been unpacked, and allow the
sender to reuse that space.

	5. If you save any PVM buffers (with pvm_setsbuf, for example), 
it is important that you free them before calling pvm_exit(), otherwise
your process will hang.

	6. The socket-based library has been renamed libpvm3s.a; PVM (more
precisely, the daemon) can serve tasks compiled with either library.
If you want to use sockets in your application compile it with the
option "-lpvm3s". Tasks using different libraries can (only) communicate 
through pvmd (i.e. no PvmRouteDirect).

	7. DataInPlace encoding not supported in this release.

Solaris2.3
----------

APPLICATION PROGRAMS

	All PVM programs must be linked with the thread library (-lthread).
Refer to the example Makefile (pvm3/examples/SUNMP/Makefile) for 
details.


BUGS

	1. This software has been tested on a 2-processor Sparc10/512, a
4-processor SparcServer1000, and a 24-processor Cray CS6400. 

	2. There is a system limit on the number of shared-memory segments
a process can attach to. This in turn imposes a limit on the number of
PVM tasks allowed on a single host. According to Sun Microsystem, the
system parameter can be set by a system administrator:

=====================================================================
Here is the information on getting the limits:

One way if this is what you want.
Must be root.
gwhiz# adb -k /dev/ksyms /dev/mem
shminfo,4/D
shminfo:
shminfo:        268435456       200             200             200
 
 
So shmseg is the fourth field which is 200 decimal.
Remember that this driver is only loaded if you use it. So shminfo will
show up symbol undefined unless the driver has been loaded.
 
The default seems to be six but for tpc we up it in "/etc/system".
=======================================================================

	3. Occasionally dead PVM processes can become zombies and haunt
the daemon. When this happens the pvmd3 process must be killed manually
(with kill -9).


SGI 5.1.1.2
-----------

APPLICATION PROGRAMS

	All PVM programs must be linked with the mutex library (-lmutex)
as well as the XDR library (-lsun). Refer to the example Makefile
(pvm3/examples/SGIMP/Makefile) for details.


BUGS

	1. This software has only been tested on a 2-way SGI Power system.



6. IBM SP2
----------

	This implementation is built on top of MPI-F, IBM's version of the
Message Passing Interface. Information provided by Hubertus Franke of
IBM T. J. Watson Research Center is gratefully acknowledged.


INSTALLATION

	Type "make PVM_ARCH=SP2MPI" in this directory. Make sure the mpicc
compiler is in your path.

	To make the examples, go to the "examples" directory, and type
		setenv PVM_ARCH SP2MPI; ../lib/aimk


APPLICATION PROGRAMS

	Host (master) programs should be linked with pvm3/lib/SP2MPI/libpvm3.a .
Node (slave) programs must be compiled with the mpicc compiler for a C
program and mpixlf compiler for a FORTRAN program, and then linked with
pvm3/lib/SP2MPI/libpvm3pe.a . FORTRAN programs should also be linked with 
pvm3/lib/SP2MPI/libfpvm3.a .

	The programs in the gexamples directory have NOT been ported. The
user must modify the makefiles to link to the appropriate libraries.


BUGS AND CAVEATS

	1. The user is required to provide a node file for the SP2 that lists
all the nodes the user intends to run PVM on. This is not to be confused 
with the PVM hostfile. Before starting PVM, the POE environment variable
MP_HOSTFILE should be set to the path of the user's SP2 node file. For
example I have the line "setenv MP_HOSTFILE ~/host.list" in my .cshrc file.
My host.list file contains the names of the switches on all the parallel
nodes.

	2. On a SP2 node only one process can use the switch at any given time. 
To run your program you need one node for each PVM task, plus one for the 
PVM host process that relays messages for pvmd. So if you want to spawn 8 
PVM tasks, for instance, you'll need at least 9 node names in your SP2 
node file. Remember there is a host process for each group of tasks spawned
together. If you spawn 8 tasks in two batches, you'll need 10 nodes instead
of 9.

	3. Signals sent by pvm_sendsig() are simply discarded, except for
SIGTERM and SIGKILL, which are equivalent to calling pvm_kill().

	4. If you kill one PVM task using pvm_kill(), all the tasks spawned
together will also die in sympathy.

	5. To get the best performance for communications that involves pvmd,
for example when you pass messages between two group of tasks spawned
separately, use switch names in the node file instead of the host names,
and start pvm with the -nswitchname option. This would force any socket
communication to go over the high-performance switch rather than Ethernet.

