  Linux SMP FAQ
  David Mentré, David.Mentre@irisa.fr
  v0.47, 20 november 1998

  This FAQ review main issues (and I hope solutions) related to SMP con
  figuration under Linux.
  ______________________________________________________________________

  Table of Contents


  1. Introduction

  2. What's new ?

  3. Questions related to any architectures

     3.1 Kernel side
     3.2 User side
     3.3 SMP Programming
        3.3.1 Parallelization methods
        3.3.2 The C Library
        3.3.3 Languages, Compilers and debuggers
        3.3.4 Other libraries
        3.3.5 Other points about SMP Programming

  4. Intel architecture specific questions

     4.1 Why it doesn't work on my machine?
     4.2 Possible causes of crash
     4.3 Motherboard specific information
        4.3.1 Motherboards with known problems
        4.3.2 Motherboards with NO known problems
     4.4 How to obtain maximum performance ? (not strictly SMP related)
        4.4.1 From disks

  5. Useful pointers

     5.1 Various
     5.2 Multithreaded programs and library
     5.3 SMP specific patches
     5.4 Parallelizing/Optimizing Compilers for 586/686 machines (

  6. Glossary

  7. List of contributors



  ______________________________________________________________________

  1.  Introduction

  Linux can work on SMP (Symetric Multi-Processors) machines. SMP
  support has started with the 2.0 family and has been improved in the
  2.1 (future 2.2) saga.


  FAQ maintained by David Mentré (David.Mentre@irisa.fr). The latest
  edition of this FAQ can be found at

  ·  http://www.irisa.fr/prive/mentre/smp-faq/ (France)

  ·  http://www.phy.duke.edu/brahma/smp-faq/ (USA)


  If you want to contribute to this FAQ, I would prefer a diff against
  the SGML version <http://www.irisa.fr/prive/mentre/smp-faq/smp-
  faq.sgml> of this document, but any remarks (in plain text) will be
  greatly appreciated. If you send me an email about this FAQ, please
  include a tag like [Linux SMP FAQ] in the Subject: field of your e-
  mail. It helps me automatically sort mails (and you will have a faster
  reply ;)).


  This FAQ is an improvement of a first draft made by Chris Pirih.


  All information contained in this FAQ is provided "as is." All
  warranties, expressed, implied or statutory, concerning the accuracy
  of the information of the suitability for any particular use are
  hereby specifically disclaimed. While every effort has been taken to
  ensure the accuracy of the information contained in this FAQ, the
  authors assume(s) no responsibility for errors or omissions, or for
  damages resulting from the use of the information contained herein.


  2.  What's new ?



     v0.47, 20 november 1998

     ·  Added that 2.0.36 as the MTRR patch (related to the BogoMips
        problem)


     v0.46, 10 november 1998

     ·  Update about Epox KP6-LS motherboards


     v0.45, 25 october 1998

     ·  Corrected an error regarding /proc/stat file

     ·  Added a pointer to CESDIS Ethernet Linux Drivers site


     v0.44, 14 october 1998

     ·  Updated the link to the web page: Motherboards rumored to run
        Linux SMP

     ·  Added Jakob explanation how to time SMP systems with 2.0 kernels


     v0.43, 9 september 1998

     ·  Updated first question in section 3.1

     ·  Updated mt-Mesa link: multi-threaded is now included as
        experimental in the Mesa distribution


     v0.42, 2 september 1998

     ·  Minor cosmetic update in sect 3.3

     ·  Two links (multithreaded Mesa and SMP performance) marked
        outdated

     ·  Updated the item about threads and exceptions in C++ (sect 3.3)


     v0.41, 1 september 1998

     ·  Added a major section: "3.3 SMP Programming" written by Jakob
        Østergaard

     ·  moved some item of section "3.2 User side" in sect 3.3


     v0.40, 27 august 1998

     ·  Updated section 3.1, item 7: processor affinity


     v0.39, 27 august 1998

     ·  Updated needed Award BIOS version for Tyan motherboards (hASCII)

     ·  Added an item on IRQ in the crash section (me and hASCII)

     ·  Added good support of Asus P2B-DS (Ulf Rompe)

     ·  Added another smp-list archive in pointer section (Hank
        Leininger)


     v0.38, 8 august 1998

     ·  Added a pointer to the Linux Threads FAQ


     v0.37, 30 July 1998

     ·  Emil Briggs is working on parallel plugins for Gimp (see "Is
        there any threaded programs or library?", sect. "User side")


     v0.36, 26 July 1998

     ·  Thanks to Jakob Østergaard, two changes in "Possible causes of
        Crash"

     ·  Changed 2.0.33 to 2.0.35 (latest stable)

     ·  Added a "BIOS related causes of failure"


     v0.35, 14 July 1998

     ·  Added N440BX Server Board in Motherboards with NO problems

     ·  Added a succes story for GigaByte motherboard with BIOS upgrade

     ·  Added a "How to obtain maximum performance ?" section (waiting
        for your contributions ;)


     v0.34, 10 june 1998

     ·  Added a "Parallelizing/Optimizing Compilers for 586/686
        machines" section in section "Useful Pointers", thanks to Sumit
        Roy


     ·  Corrected a mispelling, "Asus P/I-UP5" is in fact "Asus P/I-
        P65UP5"


     v0.33, 3 june 1998

     ·  Yet another success story for a GigaByte DLX Motherboard.

     ·  A tip for Tyan motherboards, disable the "DRAM Fast Leadoff"
        BIOS option


     v0.32, 27 may 1998

     ·  Asus P/I-UP5 added in the motherboard-with-NO-problem section


     v0.31, 18 may 1998

     ·  Elitegroup P6LX2-A works with 2.1.100 and 101

     ·  Bugs should be reported to linux-smp@vger.rutgers.edu


     v0.30, 12 may 1998

     ·  SuperMicro is now in the motherboard-with-NO-problem section


     v0.29, 11 may 1998

     ·  A success story for a GigaByte 686 motherboard with 2.1.101

     ·  Added a new item in the "User Side" section: "Is there any
        threaded programs or library?"

     ·  OpenGL Mesa library is beeing multithreaded. Cool! See the new
        section for details.


     v0.28, 09 may 1998

     ·  A US mirror of this FAQ is now available (see Introduction)

     ·  Merge of the two confusing Gigabyte 686 entries


     v0.27, 05 may 1998

     ·  New info for the Adaptec and TekRam drivers

     ·  Micronics W6-LI motherboard works under SMP




  3.  Questions related to any architectures


  3.1.  Kernel side


  1. Does Linux support multi-threading?  If I start two or more
     processes, will they be distributed among the available CPUs?

     Yes. Processes and kernel-threads are distributed among processors.
  2. What kind of architectures are supported in SMP?


     From Alan Cox:
        SMP is supported in 2.0 on the hypersparc (SS20, etc.) systems
        and Intel 486, Pentium or higher machines which are Intel
        MP1.1/1.4 compliant. Richard Jelinek adds: right now, systems
        have been tested up to 4 CPUs and the MP standard (and so Linux)
        theoretically allows up to 16 CPUs.

        SMP support for UltraSparc, SparcServer, Alpha and PowerPC
        machines is in progress in 2.1.x.


     From Ralf Bächle:
        MIPS, m68k and ARM does not support SMP; the latter two probly
        won't ever.

        That is, I'm going to hack on MIPS-SMP as soon as I get a SMP
        box ...


  3. How do I make a Linux SMP kernel?

     Uncomment the SMP=1 line in the main Makefile
     (/usr/src/linux/Makefile).

     AND

     enable "RTC support" (from Robert G. Brown). Note that inserting
     RTC support actually doesn't afaik prevent drift, but according to
     a discussion [Robert G. Brown] remember from a year ago or so it
     can prevent lockup when the clock is read at boot time. A note from
     Richard Jelinek says also that activating the Enhanced RTC is
     necessary to get the second CPU working (identified) on some
     original Intel Mainboards.

     AND

     do NOT enable APM!  APM and SMP are not compatible, and your system
     will almost certainly (or at least probably ;)) crash under boot if
     APM is enabled (Jakob Oestergaard). Alan Cox confirms this : 2.1.x
     turns APM off for SMP boxes.  Basically APM is undefined in the
     presence of SMP systems, and anything could occur.



     You must rebuild all your kernel and kernel modules when changing
     to and from SMP mode. Remember to make modules and make
     modules_install (from Alan Cox).


  4. How do I make a Linux non-SMP kernel?

     Comment the SMP=1 line in the Makefile (and not set SMP to 0).



     You must rebuild all your kernel and kernel modules when changing
     to and from SMP mode. Remember to make modules and make
     modules_install.



  5. How can I tell if it worked?

      cat /proc/cpuinfo




  Typical output (dual PentiumII):

  ______________________________________________________________________
  processor       : 0
  cpu             : 686
  model           : 3
  vendor_id       : GenuineIntel
  stepping        : 3
  fdiv_bug        : no
  hlt_bug         : no
  fpu             : yes
  fpu_exception   : yes
  cpuid           : yes
  wp              : yes
  flags           : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmo
v mmx
  bogomips        : 267.06

  processor       : 1
  cpu             : 686
  model           : 3
  vendor_id       : GenuineIntel
  stepping        : 3
  fdiv_bug        : no
  hlt_bug         : no
  fpu             : yes
  fpu_exception   : yes
  cpuid           : yes
  wp              : yes
  flags           : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmo
v mmx
  bogomips        : 267.06
  ______________________________________________________________________




  6. What is the status of converting the kernel toward finer grained
     locking and multithreading?

     2.1.x has signal handling, interrupts and some I/O stuff fine grain
     locked.  The rest is gradually migrating. All the scheduling is SMP
     safe


  7. Does Linux SMP support processor affinity?



     Standard kernel
        No and Yes.  There is no way to force a process onto specific
        CPU's but the linux scheduler has a processor bias for each
        process, which tends to keep processes tied to a specific CPU.


     Patch
        Yes. Look at PSET - Processor Sets for the Linux kernel
        <http://isunix.it.ilstu.edu/~thockin/pset/>:

          The goal of this project is to make a source compatible
          and functionally equivalent version of pset (as defined
          by SGI - partially removed from their IRIX 6.4 kernel)
          for Linux.  This enables users to determine which
     processor or set of processors a process may run on.
     Possible uses include forcing threads to seperate proces
     sors, timings, security (a `root' only CPU?) and probably
     more.




  8. Where should one report SMP bugs to?

     Please report bugs to linux-smp@vger.rutgers.edu.




  3.2.  User side


  1. Do I really need SMP?

     If you have to ask, you probably don't. :)


  2. How does one display mutiple cpu performance?

     Thanks to Samuel S. Chessman, here is some useful utilities:

     Character based:
        http://www.cs.inf.ethz.ch/~rauch/procps.html

        Basically, it's   procps v1.12.2   (top, ps, et. al.)  and some
        patches to support SMP.


     Graphic:
        xosview-1.5.1 supports SMP. And kernels above 2.1.85 (included)
        the cpuX entry in /proc/stat file.

        The official homepage for xosview is:
        http://lore.ece.utexas.edu/~bgrayson/xosview.html

        The various forissier's kernel patches are at: http://www-
        isia.cma.fr/~forissie/smp_kernel_patch/


  3. How can I enable more than 1 process for my kernel compile?

     use:

     ___________________________________________________________________
             # make [modules|zImage|bzImages] MAKE="make -jX"
             where X=max number of processes.
             WARNING: This won't work for "make dep".
     ___________________________________________________________________



  With a 2.1.x like kernel, see also the file
  /usr/src/linux/Documentation/smp for specific instruction.

  BTW, since running multiple compilers allows a machine with sufficient
  memory to use use the otherwise wasted CPU time during I/O caused
  delays make MAKE="make -j 2" -j 2 actually even helps on uniprocessor
  boxes (from Ralf Bächle).


  4. Why the time given by the time command is false ?  (from Joel
     Marchand)

     In the 2.0 series, the result given by the time command is false.
     The sum user+system is right *but* the spreading between user and
     system time is false.

     More precisely: "The explanation is, that all time spent in
     processors other than the boot cpu is accounted as system time.  If
     you time a program, add the user time and the system time, then you
     timing will be almost right, except for also including the system
     time that is correctly accounted for" (Jakob Østergaard).

     This bug in corrected in 2.1 series.


  5. How will my application perform under SMP?

     Look at SMP Performance of Linux [OUTDATED]
     <http://www.interlog.com/~mackin/linux-smp.html> which gives useful
     hints how to bench a specific machine (from a post made by Cameron
     MacKinnon).






  3.3.  SMP Programming

  Section made by Jakob Østergaard.

  This section is intended to outline what works, and what doesn't, when
  it comes to programming multi-threaded software for SMP Linux.


  3.3.1.  Parallelization methods


  1. POSIX Threads

  2. PVM / MPI Message Passing Libraries

  3. fork() -- Multiple processes

  Since both fork() and PVM/MPI processes usually does not share memory,
  but either communicate by means of IPC or a messaging API, they will
  not be described further in this section. They are not very specific
  to SMP, since they are used just as much - or more - on uniprocessor
  computers, and clusters thereof.

  Only POSIX Threads provide us with multiple threads sharing ressources
  like - especially - memory. The exact same thing that makes a SMP
  machine special, by allowing many processors to share their memory. To
  use both (or more ;) processors of an SMP, use a kernel-thread
  library. A good library is the LinuxThreads, a pthread library made by
  Xavier Leroy <http://pauillac.inria.fr/~xleroy/linuxthreads/> which is
  now integrated with glibc2 (aka libc6).

  Multi-threading has never been really popular in the UN*X world
  though. For some reason, applications requiring multiple processes or
  threads, has mostly been written using fork(). Therefore, when using
  the thread approach, one runs into problems of incompatible (not
  thread-ready) libraries, compilers, and debuggers.  GNU/Linux is no
  exception to this. Hopefully the next few sections will sched a little
  light over what is currently possible, and what is not.
  3.3.2.  The C Library

  Older C libraries are not thread-safe. It is very important that you
  use GNU LibC (glibc), also known as libc6. Earlier versions are
  ofcourse possible to use, but it will cause you much more trouble than
  upgrading your system will, well probably   :)

  If you want to use GDB to debug your programs, you should get the
  patches or SRPMS available for glibc at
  http://odin.appliedtheory.com/debug_thread_rpms/RedHat_5.1/ which will
  allow GDB to work with threads.

  It is not nessecary to patch glibc in any way just to make it work
  with threads. If you do not need to debug the software (this could be
  true for all machines that are not development workstations), there is
  no need to patch glibc.


  3.3.3.  Languages, Compilers and debuggers

  There is a wealth of programming languages available for GNU/Linux,
  and many of them can be made to use threads one way or the other (some
  languages like Ada and Java even have threads as primitives in the
  language).

  This section will, however, currently only describe C and C++. If you
  have experience in SMP Programming with other languages, please
  enlighten us.

  GNU C and C++, as well as the EGCS C and C++ compilers work with the
  thread support from the standard C library (glibc). There are however
  a few issues:


  1. When compiling C or C++, use the -D_REENTRANT define in the
     compiler command line. This is necessary to make certain error-
     handling functions work.

  2. When using C++, If two threads throw exceptions concurrently, the
     program will segfault.  The compiler does not generate thread-safe
     exception code.

     The workaround is to put a
     pthread_mutex_lock(&global_exception_lock) in the constructor(s) of
     every class you throw(), and to put the corresponding
     pthread_mutex_unlock(...) in the destructor.  It's ugly, but it
     works.  This solution was given by Markus Ferch.

  The GNU Debugger GDB as of version 4.17, is incapable of handling
  threads, at least in any sensible way. There is however a patched
  version available at
  http://odin.appliedtheory.com/debug_thread_rpms/RedHat_5.1/, for
  RedHat Linux at least. It's probably quite easy to use the standalone
  patches on any other GNU/Linux distribution. Note that you must also
  patch your glibc.

  Note that core-dumps are of no use, when using multiple threads.
  Somehow the core dump is attached to one of the currently running
  threads, and not to the program as a whole. Therefore, whenever you
  are debugging anything, run it from the debugger.

  Hint: If you have a thread running haywire, like eating 100% CPU time,
  and you cannot seem to figure out why, here is a nice way to find out
  what's going on: Run the program straight from the shell, no GDB. Make
  the thread go haywire. Use top to get the PID of the process.  Run GDB
  like gdb program pid. This will make GDB attach itself to the process
  with the PID you specified, and stop the thead. Now you have a GDB
  session with the offending thread, and can use bt and the likes to see
  what is happening.


  3.3.4.  Other libraries

  ElectricFence: This library is not thread safe. It should however be
  possible to make it work in SMP environments, by inserting mutex locks
  in the ElectricFence code.



  3.3.5.  Other points about SMP Programming


  1. Where can I found more information about parallel programming?

     Look at the Linux Parallel Processing HOWTO
     <http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html>

     Lots of useful information can be found at Parallel Processing
     using Linux <http://yara.ecn.purdue.edu/~pplinux/>

     Look also at the Linux Threads FAQ <http://linas.org/linux/threads-
     faq.html>


  2. Is there any threaded programs or library?

     Yes. For programs, you should look at: Multithreaded programs on
     linux <http://www.e.kth.se/~e94_bek/mthread.html> (I love
     hyperlinks, did you know that ? ;))

     As far as library are concerned, there is:


     OpenGL Mesa library
        Thanks to David Buccarelli, Andreas Schiffler and Emil Briggs,
        it exists in a multithreaded version (right now [1998-05-11],
        there is a working version that provides speedups of 5-30% on
        some OpenGL benchmarks). The multithreaded stuff is now included
        in the regular Mesa distribution as an experimental option.  For
        more information, look at the Mesa library
        <http://www.ssec.wisc.edu/~brianp/Mesa.html>


     BLAS
        Pentium Pro Optimized BLAS and FFTs for Intel Linux
        <http://www.cs.utk.edu/~ghenry/distrib/>

        Multithreaded BLAS routines are not available right now, but a
        dual proc library is planned for 1998-05-27, see Blas News
        <http://www.cs.utk.edu/~ghenry/distrib/blasnews> for details.


     The GIMP
        Emil Briggs, the same guy who is involved in multithreaded Mesa,
        is also working on multithreaded The GIMP plugins. Look at
        http://nemo.physics.ncsu.edu/~briggs/gimp/index.html for more
        info.





  4.  Intel architecture specific questions


  4.1.  Why it doesn't work on my machine?


  1. Can I use my Cyrix/AMD/non-Intel CPU in SMP?

     Short answer: no.

     Long answer: Intel claims ownership to the APIC SMP scheme, and
     unless a company licenses it from Intel they may not use it. There
     are currently no companies that have done so.  (This of course can
     change in the future) FYI - Both Cyrix and AMD support the non-
     proprietary OpenPIC SMP standard but currently there are no
     motherboards that use it.


  2. Why doesn't my old Compaq work?

     Put it into MP1.1/1.4 compliant mode.


  3. Why doesnt my ALR work?

     From Robert Hyatt : ALR Revolution quad-6 seems quite safe, while
     some older revolution quad machines without P6 processors seem
     "iffy"...


  4. Why does SMP go so slowly? or Why does one CPU show a very low
     bogomips value while the first one is normal?

     From Alan Cox: If one of your CPU's is reporting a very low
     bogomips value the cache is not enabled on it. Your vendor probably
     provides a buggy BIOS. Get the patch to work around this or better
     yet send it back and buy a board from a competent supplier.

     Another work-around is to use the 2.0.36 kernel. It contains the
     MTRR patch which should solve this problem (select option "Handle
     buggy SMP BIOSes with bad MTRR setup" in the "General setup" menu).


  5. I've heard IBM machines have problems


     Some IBM machines have the MP1.4 bios block in the EBDA, allowed
     but not supported by <2.1.80. Please update to the right kernel.

     There is an old 486SLC based IBM SMP box. Linux/SMP requires
     hardware FPU support.


  6. Is there any advantage of Intel MP 1.4 over 1.1 specification?

     Nope (according to Alan :) ), 1.4 is just a stricker specs of 1.1.


  7. Why does the clock drift so rapidly when I run linux SMP?


     This is known problem with IRQ handling and long kernel locks in
     the 2.0 series kernels.  Consider upgrading to a later 2.1 kernel
     (not garenteed to work).

     From Jakob Oestergaard: Or, consider running xntpd. That should
     keep your clock right on time.  (I think that I've heard that
     enabling RTC in the kernel also fixes the clock drift. It works for
     me! but I'm not sure whether that's general or I'm just being
     lucky)



  8. Why are my CPU's numbered 0 and 2 instead of 0 and 1 (or some other
     odd numbering)?

     The CPU number is assigned by the MB manufacturer and doesn't mean
     anything.  Ignore it.



  9. My SMP system is locking up all the time.  Black screen, nothing in
     the logs.  Help!

     If you're running a 2.0 kernel, consider upgrading to later 2.0.32+
     kernels or apply Leonard Zubkoff's deadlock patch.  If you still
     have deadlocks, apply Ingo Molnar's deadlock detection patch and
     post the results (against your System.map) to linux-smp or linux-
     kernel.  You might also consider running a 2.1 kernel.





  4.2.  Possible causes of crash

  You'll find in this section some possible reasons for a crash of an
  SMP machine (credits are due to Jakob Østergaard for this part). As
  far as I (david) know, theses problems are Intel specific.



  ·  Cooling problems

     From Ralf Bächle: [Related to case size and fans] It's important
     that the air is flowing.  It of course can't where cables etc. are
     preventing this like in too small cases.  On the other side I've
     seen oversized cases causing big problems.  There are some tower
     cases on the market that actually are worse for cooling than
     desktops.  In short, the right thing is thinking about aerodynamics
     in the case.  Extra cases for hot peripherals are usefull as well.



  ·  Bad memory

     Don't buy too cheap RAM and don't use mixed RAM modules on a
     motherboard that is picky about it.

     Especially Tyan motherboards are known to be picky about RAM speed
     (see the Tyan paragraph below for a possible solution).



  ·  Bad combination of different stepping CPUs

     Check /proc/cpuinfo to see that your CPUs are same stepping.


  ·  You are running 2.0.35 aren't you ?

     If you run 2.0.31 or 2.1.xx you can't be sure that SMP is stable.
     2.0.35 is the right kernel for a production system. 2.1.xx kernels
     perform better, but they are development releases and should NOT be
     considered stable!


  ·  If your system is unstable, then DON'T overclock it!

     ...and even if it is stable, DON'T overclock.

     From Ralf Bächle: Overclocking causes very subtile problems.  I
     have a nice example, one of my overclocked old machines
     misscomputes a couple of pixels of a 640 x 400 fractal.  The
     problem is only visible when comparing them using tools. So better
     say never, nuncas, jamais, niemals overclock.



  ·  2.0.x kernel and fast ethernet (from Robert G. Brown)

     2.0.X kernels on high performance fast ethernet systems have
     significant (and known) problems with a race/deadlock condition in
     the networking interrupt handler.

     The solution is to get the latest 100BT development drivers from
     CESDIS Linux Ethernet device drivers site
     <http://cesdis.gsfc.nasa.gov/linux/drivers/> (ones that define
     SMPCHECK).


  ·  A bug in the 440FX chipset (from Emil Briggs)

     If you had a system using the 440FX chipset then your problem with
     the lockups was possibly due to a documented errata in the chipset.
     Here is a reference

     References: Intel 440FX PCIset 82441FX (PMC) and 82442FX (DBX)
     Specification Update.  pg. 13

     http://www.intel.com/design/pcisets/specupdt/297654.htm

     The problem can be fixed with a BIOS workaround (Or a kernel patch)
     and in fact David Wragg wrote a patch that's included with Richard
     Gooch's MTTR patch. For more information and a fix look here:

     http://nemo.physics.ncsu.edu/~briggs/vfix.html


  ·  DONT run emm386.exe before booting linux SMP

     From Mark Duguid, dumb rule #1 with W6LI motherboards. ;)


  ·  If the machine reboots/freezes after a while, there can be two good
     BIOS + memory related reasons (from Jakob Østergaard)

  ·  If the BIOS has settings like "memory hole at 16M" and/or "OS/2
     memory > 64MB", try disabling them both. Linux does not always
     react well with theese options.

  ·  If you have more than 64 MB of memory in the machine, and you
     specified the exact number manually in the LILO configuration, you
     should specify one MB less than you actually have in the machine.
     If you have 128 MB, you lilo.conf line looks like:
     append="mem=127M"


  ·  Be aware of IRQ related problems

     Sometime, some cards are not recognized or can trigger IRQ
     conflicts. Try shuffle cards on slots in different ways.

     Contributed by hASCII : removing an " append="hisax=9,2,3"" line in
     lilo.conf allowed using a kernel from the 2.1.xx series with
     activated ISDN + Hisax support. Kernels from the 2.0.xx series
     doesn't make problems like this.



  Some hardware is also known to cause problems. This includes:

  ·  Adaptec SCSI controllers

     Latest news (05 may 1998):

     The latest version of this driver (5.0.13 at this writting) should
     be SMP safe (both in 2.0 and 2.1 kernels). Doug had verified it
     through code review and intensive tests (on a dual PII system).

     This said, some machines still have problems. Doug has kindly given
     some explanations:

  ·  Doug: The simple fact of the matter is that this version of the
     aic7xxx driver places a larger demand on DMA transfers than the old
     driver did, especially for for smaller DMA transactions. So, the
     point should be made that this version of the driver *is* more
     demanding on your system than either the 4.1 or 4.1.1 driver, and
     that certain marginal systems may get hit by this problem (such as
     my own P133 system is).

  ·  In 2.1 kernels, they are still problems (with IRQ) with the kernel
     (2.1.99) and they are not related to this specific driver. People
     are working on this.


  ·  3Com 3c905 cards

     Some work, some don't. Try disabling busmastering if your system is
     unstable.




  4.3.  Motherboard specific information

  Please note: Some more specific information can be found with the list
  of Motherboards rumored to run Linux SMP <http://www.nlug.org/smp/>


  4.3.1.  Motherboards with known problems


  ·  GA686DLX (Andrew Crane)

     Same BIOS related BogoMIPS problem as other motherboards.

     Solution from Alan Cox: Congratulations, send the bill for your
     hair damage to the supplier. You have yet another SMP box with
     faulty bios. There is a patch for 2.0.x on www.uk.linux.org and
     there are people working on generic MTRR handling for 2.1.x

     From Claus-Justus Heine: I'm able to boot above MB with 2.1.90 +
     Ingo's apic + Richard's mtrr patches. From David Mentré: the kernel
     > 2.1.99 have the MTRR patch included and Florian Hinzmann told me
     that with 2.1.101, both CPUs are detected fine.

     David Maslen tells: Fortunately I suspect these problems have been
     fixed, and would like to report my success so that others won't be
     put off what seems to be a nice motherboard. [...] My system has
     been running for about 3 days without a lockup (72hrs) Kernel
     2.1.103 compiled easily. /proc/cpuinfo says the cpus are both
     there. This was also true of 2.0.33.

     Other possible solution: BIOS upgrade (see http://www.giga-
     byte.com).

     (14 July 1998)Volker Reichelt confirms that with a BIOS upgrade
     from v1.14 to v1.20 everything seems to work fine.


  ·  EPoX KP6-LS (Christopher Allen Wing, 16 march 1998)

     It appears to have the same BIOS related BogoMIPS problem as other
     motherboards. (one CPU only gives about 3 BogoMIPS, the other gives
     the full amount) All 2.0.x kernels lock up soon after booting, late
     2.1.x kernels run slowly but don't seem to lock up.  There is no
     BIOS upgrade available (yet). I wrote the manufacturer but have not
     received a reply.

     (update, 10 november 1998) First of all, there are 2 revisions on
     the KP6-LS: (1) Revision 2 (production release, has a power-leak on
     the second processor connector - To fix this problem, contact Epox
     support and they will guide you for further assistence.  - For some
     boards it is needed to replace the entire board. Cause the leak is
     corrected in the middle of this revision).  (2) Revision 3
     (production release). For both the revisions is a patch avaliable
     from their site (http://www.epox.com/support/bios.html). Please
     make sure that Epox has been contacted about the bios upgrade
     before applying.  After applying the patch, kernel 2.0.35 and all
     other expirimental kernels from 2.1.110 up to 2.1.125 report a
     correct BogoMIPS index. (Niels Ammerlaan)


  ·  Tyan

     Tyan motherboards are known to be picky about RAM speed (Jakob
     Oestergaard).

     From Doug Ledford about the onboard aic-7895 SCSI controller (for
     which he wrote the driver): "BTW, make sure you have at least BIOS
     version 1.16 on that Tyan motherboard.  The 1.15 and below BIOS
     versions have a bug relating to IRQ allocation for the 7895 SCSI
     controller" (submitted by Szakacsits Szabolcs).

     (27 august 1998) From hASCII: If you have a tyan with award bios,
     the newest version you can get is 1.02.

     But, as motherboard problems are more close to grayshade than black
     and white ;), Richard Jelinek (from S.u.S.E.) tells that they
     sell/sold several Tyan Titan Pro (Dual-PPro) Motherboards. They
     work perfectly with SMP.

     (3 june 1998) maybe an explanation found by Leonard Zubkoff and
     told tp me by Sean Reifschneider: The Tyan Titan Pro motherboards
     with 3.03 BIOS version (dated 10/31/96) has an option labled "DRAM
     Fast Leadoff", which MUST be disabled.



  ·  MS-6114

     More details for this motherboard at
     http://www.msi.com.tw/product/6114/6114.htm
     <http://www.msi.com.tw/product/6114/6114.htm>

     Solution: BIOS upgrade

     Somebody experienced solid hangs (nothing in the log files) under
     constant load of about 5 running processes within less than 12
     hours with AMI BIOS v1.1.  v1.4b3 runs without problems.





  4.3.2.  Motherboards with NO known problems


  ·  AIR P6NDP and P6NDI (Leonard N. Zubkoff)

     My primary production machine is based on an AIR P6NDP and one of
     my test machines uses a P6NDI.  Both seem to be fine motherboards
     in my experience.  The P6NDI BIOS is a little conservative in its
     programming of the Natoma chipset for 50ns EDO, but a minor tweek
     to one register in rc.local took care of that.


  ·  AIR 54CDP (Chris Mauritz)

     You can also list the following motherboard as working with no
     problems:

     AIR 54CDP motherboard / EISA/PCI / onboard aic7870 / dual P120 /
     Redhat 5.0 (2.0.32 and 2.0.33 kernels)

     However, Jon Lewis adds the following comments: Has Chris tried
     putting multiple PCI cards in these motherboards...say a SCSI card
     and a network card?  Due to BIOS brain damage, the 54CDP insists on
     having all PCI devices share a single IRQ.  AIR lead me on for 2
     weeks and then said "there have been / will be no updates for your
     board".  This means I can't have Tulip network cards and PCI SCSI
     without hacking the drives (which seems to work, though I was
     cautioned not to do it).


  ·  HP XU 6/200 (Jean-Francois Micouleau)

     Works with 2.0 and 2.1 kernels. Some problems under high network
     load with 2.0.x kernel. Works under 2.1.78 with Ingo Molnar IO-APIC
     patch.


  ·  Elitegroup P6FX2-A (Benedikt Heinen)

     Had this mainboard running with ONE PPro on it for several months,
     and since about a year, it's running without problems with TWO PPro
     200MHz. The only crashes this machine ever experienced were before
     Leonard Zubkoff's deadlock-patches for Linux 2.0.30... ;)

     Elitegroup P6FX2-A / ISA/PCI / Dual PPro200 / Debian "hamm"


  ·  Elitegroup P6LX2-A (Ulf Nielsen)

     This motherboard works with a post 2.1.99 kernel (100 and 101
     tested).  All kernels pre 2.1.100 does not work on P6LX2-A.


  ·  QDI P6I440LX/DP "Legend IV" (Tony Kocurko)

     The QDI P6I440LX/DP "Legend IV" dual Pentium II motherboard now
     boots properly with Linux-SMP. As of 0326 hours on 15 April, there
     is a necessary BIOS upgrade to version 1.6 at the Canadian site
     (http://www.qdi.ca). Previously, the Legend IV would bring up the
     first processor normally, but the /proc/cpuinfo file would show the
     second processor barely alive. The BIOS update seems to have
     corrected the problem.


  ·  Asus P2L97-DS

     ASUS P2L97-DS works great under Linux-SMP (from Richard Jelinek).


  ·  Asus P/I-P65UP5 (Jukka Tainio)

     I have run it with both 2x233 MMX Pentium and 2x200 PPRO. Both
     boards are in heavy www-proxy use and work without problems. Kernel
     version is 2.0.33.


  ·  Asus P2B-DS (Ulf Rompe)

     Asus P2B-DS is working without problems. It runs with two PII-266,
     the onboard Adaptec 7890 (Patch from
     ftp://ftp.dialnet.net/pub/linux/aic7xxx/ required) in U2W mode, an
     Elsa Victory Erazor AGP an some obscure other hardware.

     The only thing missing is support for the new SMP-aware APM, but
     that is a general Linux problem.



  ·  Micronics W6-LI

     From Mark Garlanger: 2.0.33 is very stable on the board, and the
     2.1.xx kernels are improving.  People have also been able to run
     other OSes including Windows NT(yuck...), FreeBSD-SMP, Rhapsody.


  ·  SuperMicro

     Thomas Schenk is running about 80 SMP machines with SuperMicro
     motherboards without any problem. Kernel used: 2.0.33. No "bad
     bogomips" problem for him (was I [david] wrong about this ?).


  ·  N440BX Server Board

     Moni Hollmann using a N440BX Server Board unter Linux (SMP) and no
     Problems occured so far. RAM: 256 MB, 2x Pentium/II- 350
     (Deschutes).



  4.4.  How to obtain maximum performance ? (not strictly SMP related)





  4.4.1.  From disks


  Some useful information pointed out by Robert G. Brown, one of the
  bigest contributor to this FAQ.


  Date: Wed, 8 Jul 1998 09:12:41 +1000 (EST)
  From: "Andrew Mc.Ghee" <mcghee@mech.uq.edu.au>

  I believe there was work done on this at - (I've got the report sitting in
  front of me, but there is no http reference, must have used UNIX netscape
  to print it, but it was done by Eric Hendriks at cesdis.gsfc.nasa.gov)

  Distilling it down, the paper presents

  Interface   Disk Arrangement   Cpu Usage    Throughput
  =============================================================
  IDE-PIO  4 Disks / 2 Channels   98%         read   7.53 MB/s
                                  58%         write  4.84 MB/s
  IDE-DMA  4 Disks / 2 Channels   34%         read  11.78 MB/s
                                  57%         write  9.21 MB/s
  IDE-DMA  2 Disks / 2 Channels   31%         -----  9.87 MB/s
  SCSI     4 Disks / 1 Channel    53%         read  13.41 MB/s
                                  92%         write 11.93 MB/s
  SCSI     4 Disks / 2 Channels   77%         read  21.99 MB/s
                                  97%         write 13.44 MB/s

  The test where done using RAID0 via the md system, but of course the test
  involved using a large, single contiguous data transfer.

  Andrew Mc.Ghee
  Mechanical Engineering
  University of Queensland




  5.  Useful pointers


  5.1.  Various


  ·  Parallel Processing using Linux
     <http://yara.ecn.purdue.edu/~pplinux/>

  ·  Linux Parallel Processing HOWTO
     <http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html>

  ·  (outdated) Linux SMP home page
     <http://www.uk.linux.org/SMP/title.html>

  ·  linux-smp mailing list

     To subscribe, send subscribe linux-smp in the message body at
     majordomo@vger.rutgers.edu

     To unsubscribe, send unsubscribe linux-smp in the message body at
     majordomo@vger.rutgers.edu

     Linux SMP archives <http://www.linuxhq.com/lnxlists/linux-smp/>

     Linux SMP archives at progressive-comp.com <http://www.progressive-
     comp.com/Lists/?l=linux-smp&r=1&w=2#linux-smp>

  ·  pthread library made by Xavier Leroy
     <http://pauillac.inria.fr/~xleroy/linuxthreads/>

  ·  Motherboards rumored to run Linux SMP <http://www.nlug.org/smp/>

  ·  procps <http://www.cs.inf.ethz.ch/~rauch/procps.html>

  ·  xosview <http://lore.ece.utexas.edu/~bgrayson/xosview.html>

  ·  SMP Performance of Linux [OUTDATED]
     <http://www.interlog.com/~mackin/linux-smp.html>

  ·  CESDIS Linux Ethernet device drivers site
     <http://cesdis.gsfc.nasa.gov/linux/drivers/>



  5.2.  Multithreaded programs and library


  ·  Linux Threads FAQ <http://linas.org/linux/threads-faq.html>

  ·  Multithreaded programs on linux
     <http://www.e.kth.se/~e94_bek/mthread.html>

  ·  Pentium Pro Optimized BLAS and FFTs for Intel Linux
     <http://www.cs.utk.edu/~ghenry/distrib/> (not available right now,
     but a dual proc library is planned for 5/27/98, see Blas News
     <http://www.cs.utk.edu/~ghenry/distrib/blasnews> for details)

  ·  Mesa library <http://www.ssec.wisc.edu/~brianp/Mesa.html> (with
     experimental multi-threading)

  ·  Parallel plugins for The GIMP
     <http://nemo.physics.ncsu.edu/~briggs/gimp/index.html>




  5.3.  SMP specific patches


  ·  Forissier kernel patches <http://www-
     isia.cma.fr/~forissie/smp_kernel_patch/>

  ·  Patch for a bug in the 440FX chipset
     <http://nemo.physics.ncsu.edu/~briggs/vfix.html>

  ·  MTRR patch (latest version: 1.9)
     <http://www.atnf.csiro.au/~rgooch/kernel-patches.html>

  ·  PSET - Processor Sets for the Linux kernel
     <http://isunix.it.ilstu.edu/~thockin/pset/>




  5.4.  ( Sumit Roy ) Parallelizing/Optimizing Compilers for 586/686
  machines


  ·  Pentium Compiler Group <http://www.goof.com/pcg/> creators of pgcc

  ·  Absoft <http://www.absoft.com/> , Fortran 90 and Fortran 77
     compilers

  ·  The Portland Group, Inc. <http://www.pgroup.com/>, supports the
     OpenMP <http://www.openmp.org> standard for Fortran parallelization
     on Linux

  ·  Pacific-Sierra Research Corporation <http://www.psrv.com/>, has a
     free F90 compiler for Linux, as well as parallelizing compilers for
     SMP Linux

  ·  Applied Parallel Research <http://s006.infomall.org/index.html>,
     currently have parallelizing compilers for WinNT



  6.  Glossary


  ·  SMP Symetric Multi-Processors

  ·  APIC Advanced Programmable Interrupt Controler

  ·  thread A thread is a processor activity in a process. The same
     process can have multiple threads. Those threads share the process
     address space and can therefore share data.

  ·  pthread Posix thread, threads defined by the Posix standard.



  7.  List of contributors

  Many thanks to those who help me to maintain this FAQ :


  ·  Tigran A. Aivazian

  ·  Niels Ammerlaan

  ·  Emil Briggs

  ·  Robert G. Brown

  ·  Samuel S. Chessman

  ·  Alan Cox

  ·  Andrew Crane

  ·  Mark Duguid

  ·  Jocelyne Erhel

  ·  Byron Faber

  ·  Mark Garlanger

  ·  hASCII

  ·  Claus-Justus Heine

  ·  Benedikt Heinen

  ·  Florian Hinzmann

  ·  Moni Hollmann


  ·  Robert Hyatt

  ·  Richard Jelinek

  ·  Tony Kocurko

  ·  Doug Ledford

  ·  Hank Leininger

  ·  Cameron MacKinnon

  ·  Joel Marchand

  ·  David Maslen

  ·  Chris Mauritz

  ·  Jean-Francois Micouleau

  ·  Ulf Nielsen

  ·  Jakob Oestergaard

  ·  Ulf Rompe

  ·  Jean-Michel Rouet

  ·  Ralf Bächle

  ·  Volker Reichelt

  ·  Sean Reifschneider

  ·  Sumit Roy

  ·  Thomas Schenk

  ·  Szakacsits Szabolcs

  ·  Jukka Tainio

  ·  Simen Timian Thoresen

  ·  El Warren

  ·  Christopher Allen Wing

  ·  Leonard N. Zubkoff


















