SCons.compat._scons_subprocess

Package SCons :: Package compat :: Module _scons_subprocess |
[hide private]
[frames] | no frames]

# Source Code for Module SCons.compat._scons_subprocess

       1  # subprocess - Subprocesses with accessible I/O streams
       2  #
       3  # For more information about this module, see PEP 324.
       4  #
       5  # This module should remain compatible with Python 2.2, see PEP 291.
       6  #
       7  # Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
       8  #
       9  # Licensed to PSF under a Contributor Agreement.
      10  # See http://www.python.org/2.4/license for licensing details.
      11
      12  r"""subprocess - Subprocesses with accessible I/O streams
      13
      14  This module allows you to spawn processes, connect to their
      15  input/output/error pipes, and obtain their return codes.  This module
      16  intends to replace several other, older modules and functions, like:
      17
      18  os.system
      19  os.spawn*
      20  os.popen*
      21  popen2.*
      22  commands.*
      23
      24  Information about how the subprocess module can be used to replace these
      25  modules and functions can be found below.
      26
      27
      28
      29  Using the subprocess module
      30  ===========================
      31  This module defines one class called Popen:
      32
      33  class Popen(args, bufsize=0, executable=None,
      34              stdin=None, stdout=None, stderr=None,
      35              preexec_fn=None, close_fds=False, shell=False,
      36              cwd=None, env=None, universal_newlines=False,
      37              startupinfo=None, creationflags=0):
      38
      39
      40  Arguments are:
      41
      42  args should be a string, or a sequence of program arguments.  The
      43  program to execute is normally the first item in the args sequence or
      44  string, but can be explicitly set by using the executable argument.
      45
      46  On UNIX, with shell=False (default): In this case, the Popen class
      47  uses os.execvp() to execute the child program.  args should normally
      48  be a sequence.  A string will be treated as a sequence with the string
      49  as the only item (the program to execute).
      50
      51  On UNIX, with shell=True: If args is a string, it specifies the
      52  command string to execute through the shell.  If args is a sequence,
      53  the first item specifies the command string, and any additional items
      54  will be treated as additional shell arguments.
      55
      56  On Windows: the Popen class uses CreateProcess() to execute the child
      57  program, which operates on strings.  If args is a sequence, it will be
      58  converted to a string using the list2cmdline method.  Please note that
      59  not all MS Windows applications interpret the command line the same
      60  way: The list2cmdline is designed for applications using the same
      61  rules as the MS C runtime.
      62
      63  bufsize, if given, has the same meaning as the corresponding argument
      64  to the built-in open() function: 0 means unbuffered, 1 means line
      65  buffered, any other positive value means use a buffer of
      66  (approximately) that size.  A negative bufsize means to use the system
      67  default, which usually means fully buffered.  The default value for
      68  bufsize is 0 (unbuffered).
      69
      70  stdin, stdout and stderr specify the executed programs' standard
      71  input, standard output and standard error file handles, respectively.
      72  Valid values are PIPE, an existing file descriptor (a positive
      73  integer), an existing file object, and None.  PIPE indicates that a
      74  new pipe to the child should be created.  With None, no redirection
      75  will occur; the child's file handles will be inherited from the
      76  parent.  Additionally, stderr can be STDOUT, which indicates that the
      77  stderr data from the applications should be captured into the same
      78  file handle as for stdout.
      79
      80  If preexec_fn is set to a callable object, this object will be called
      81  in the child process just before the child is executed.
      82
      83  If close_fds is true, all file descriptors except 0, 1 and 2 will be
      84  closed before the child process is executed.
      85
      86  if shell is true, the specified command will be executed through the
      87  shell.
      88
      89  If cwd is not None, the current directory will be changed to cwd
      90  before the child is executed.
      91
      92  If env is not None, it defines the environment variables for the new
      93  process.
      94
      95  If universal_newlines is true, the file objects stdout and stderr are
      96  opened as a text files, but lines may be terminated by any of '\n',
      97  the Unix end-of-line convention, '\r', the Macintosh convention or
      98  '\r\n', the Windows convention.  All of these external representations
      99  are seen as '\n' by the Python program.  Note: This feature is only
     100  available if Python is built with universal newline support (the
     101  default).  Also, the newlines attribute of the file objects stdout,
     102  stdin and stderr are not updated by the communicate() method.
     103
     104  The startupinfo and creationflags, if given, will be passed to the
     105  underlying CreateProcess() function.  They can specify things such as
     106  appearance of the main window and priority for the new process.
     107  (Windows only)
     108
     109
     110  This module also defines two shortcut functions:
     111
     112  call(*popenargs, **kwargs):
     113      Run command with arguments.  Wait for command to complete, then
     114      return the returncode attribute.
     115
     116      The arguments are the same as for the Popen constructor.  Example:
     117
     118      retcode = call(["ls", "-l"])
     119
     120  check_call(*popenargs, **kwargs):
     121      Run command with arguments.  Wait for command to complete.  If the
     122      exit code was zero then return, otherwise raise
     123      CalledProcessError.  The CalledProcessError object will have the
     124      return code in the returncode attribute.
     125
     126      The arguments are the same as for the Popen constructor.  Example:
     127
     128      check_call(["ls", "-l"])
     129
     130  Exceptions
     131  ----------
     132  Exceptions raised in the child process, before the new program has
     133  started to execute, will be re-raised in the parent.  Additionally,
     134  the exception object will have one extra attribute called
     135  'child_traceback', which is a string containing traceback information
     136  from the childs point of view.
     137
     138  The most common exception raised is OSError.  This occurs, for
     139  example, when trying to execute a non-existent file.  Applications
     140  should prepare for OSErrors.
     141
     142  A ValueError will be raised if Popen is called with invalid arguments.
     143
     144  check_call() will raise CalledProcessError, if the called process
     145  returns a non-zero return code.
     146
     147
     148  Security
     149  --------
     150  Unlike some other popen functions, this implementation will never call
     151  /bin/sh implicitly.  This means that all characters, including shell
     152  metacharacters, can safely be passed to child processes.
     153
     154
     155  Popen objects
     156  =============
     157  Instances of the Popen class have the following methods:
     158
     159  poll()
     160      Check if child process has terminated.  Returns returncode
     161      attribute.
     162
     163  wait()
     164      Wait for child process to terminate.  Returns returncode attribute.
     165
     166  communicate(input=None)
     167      Interact with process: Send data to stdin.  Read data from stdout
     168      and stderr, until end-of-file is reached.  Wait for process to
     169      terminate.  The optional stdin argument should be a string to be
     170      sent to the child process, or None, if no data should be sent to
     171      the child.
     172
     173      communicate() returns a tuple (stdout, stderr).
     174
     175      Note: The data read is buffered in memory, so do not use this
     176      method if the data size is large or unlimited.
     177
     178  The following attributes are also available:
     179
     180  stdin
     181      If the stdin argument is PIPE, this attribute is a file object
     182      that provides input to the child process.  Otherwise, it is None.
     183
     184  stdout
     185      If the stdout argument is PIPE, this attribute is a file object
     186      that provides output from the child process.  Otherwise, it is
     187      None.
     188
     189  stderr
     190      If the stderr argument is PIPE, this attribute is file object that
     191      provides error output from the child process.  Otherwise, it is
     192      None.
     193
     194  pid
     195      The process ID of the child process.
     196
     197  returncode
     198      The child return code.  A None value indicates that the process
     199      hasn't terminated yet.  A negative value -N indicates that the
     200      child was terminated by signal N (UNIX only).
     201
     202
     203  Replacing older functions with the subprocess module
     204  ====================================================
     205  In this section, "a ==> b" means that b can be used as a replacement
     206  for a.
     207
     208  Note: All functions in this section fail (more or less) silently if
     209  the executed program cannot be found; this module raises an OSError
     210  exception.
     211
     212  In the following examples, we assume that the subprocess module is
     213  imported with "from subprocess import *".
     214
     215
     216  Replacing /bin/sh shell backquote
     217  ---------------------------------
     218  output=`mycmd myarg`
     219  ==>
     220  output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
     221
     222
     223  Replacing shell pipe line
     224  -------------------------
     225  output=`dmesg | grep hda`
     226  ==>
     227  p1 = Popen(["dmesg"], stdout=PIPE)
     228  p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
     229  output = p2.communicate()[0]
     230
     231
     232  Replacing os.system()
     233  ---------------------
     234  sts = os.system("mycmd" + " myarg")
     235  ==>
     236  p = Popen("mycmd" + " myarg", shell=True)
     237  pid, sts = os.waitpid(p.pid, 0)
     238
     239  Note:
     240
     241  * Calling the program through the shell is usually not required.
     242
     243  * It's easier to look at the returncode attribute than the
     244    exitstatus.
     245
     246  A more real-world example would look like this:
     247
     248  try:
     249      retcode = call("mycmd" + " myarg", shell=True)
     250      if retcode < 0:
     251          print >>sys.stderr, "Child was terminated by signal", -retcode
     252      else:
     253          print >>sys.stderr, "Child returned", retcode
     254  except OSError, e:
     255      print >>sys.stderr, "Execution failed:", e
     256
     257
     258  Replacing os.spawn*
     259  -------------------
     260  P_NOWAIT example:
     261
     262  pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
     263  ==>
     264  pid = Popen(["/bin/mycmd", "myarg"]).pid
     265
     266
     267  P_WAIT example:
     268
     269  retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
     270  ==>
     271  retcode = call(["/bin/mycmd", "myarg"])
     272
     273
     274  Vector example:
     275
     276  os.spawnvp(os.P_NOWAIT, path, args)
     277  ==>
     278  Popen([path] + args[1:])
     279
     280
     281  Environment example:
     282
     283  os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
     284  ==>
     285  Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})
     286
     287
     288  Replacing os.popen*
     289  -------------------
     290  pipe = os.popen(cmd, mode='r', bufsize)
     291  ==>
     292  pipe = Popen(cmd, shell=True, bufsize=bufsize, stdout=PIPE).stdout
     293
     294  pipe = os.popen(cmd, mode='w', bufsize)
     295  ==>
     296  pipe = Popen(cmd, shell=True, bufsize=bufsize, stdin=PIPE).stdin
     297
     298
     299  (child_stdin, child_stdout) = os.popen2(cmd, mode, bufsize)
     300  ==>
     301  p = Popen(cmd, shell=True, bufsize=bufsize,
     302            stdin=PIPE, stdout=PIPE, close_fds=True)
     303  (child_stdin, child_stdout) = (p.stdin, p.stdout)
     304
     305
     306  (child_stdin,
     307   child_stdout,
     308   child_stderr) = os.popen3(cmd, mode, bufsize)
     309  ==>
     310  p = Popen(cmd, shell=True, bufsize=bufsize,
     311            stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)
     312  (child_stdin,
     313   child_stdout,
     314   child_stderr) = (p.stdin, p.stdout, p.stderr)
     315
     316
     317  (child_stdin, child_stdout_and_stderr) = os.popen4(cmd, mode, bufsize)
     318  ==>
     319  p = Popen(cmd, shell=True, bufsize=bufsize,
     320            stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
     321  (child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)
     322
     323
     324  Replacing popen2.*
     325  ------------------
     326  Note: If the cmd argument to popen2 functions is a string, the command
     327  is executed through /bin/sh.  If it is a list, the command is directly
     328  executed.
     329
     330  (child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
     331  ==>
     332  p = Popen(["somestring"], shell=True, bufsize=bufsize
     333            stdin=PIPE, stdout=PIPE, close_fds=True)
     334  (child_stdout, child_stdin) = (p.stdout, p.stdin)
     335
     336
     337  (child_stdout, child_stdin) = popen2.popen2(["mycmd", "myarg"], bufsize, mode)
     338  ==>
     339  p = Popen(["mycmd", "myarg"], bufsize=bufsize,
     340            stdin=PIPE, stdout=PIPE, close_fds=True)
     341  (child_stdout, child_stdin) = (p.stdout, p.stdin)
     342
     343  The popen2.Popen3 and popen3.Popen4 basically works as subprocess.Popen,
     344  except that:
     345
     346  * subprocess.Popen raises an exception if the execution fails
     347  * the capturestderr argument is replaced with the stderr argument.
     348  * stdin=PIPE and stdout=PIPE must be specified.
     349  * popen2 closes all filedescriptors by default, but you have to specify
     350    close_fds=True with subprocess.Popen.
     351
     352
     353  """
     354
     355  import sys
     356  mswindows = (sys.platform == "win32")
     357
     358  import os
     359  import string
     360  import types
     361  import traceback
     362
     363  # Exception classes used by this module.
     364 -class CalledProcessError(Exception):
     365      """This exception is raised when a process run by check_call() returns
     366      a non-zero exit status.  The exit status will be stored in the
     367      returncode attribute."""
     368 -    def __init__(self, returncode, cmd):
     369          self.returncode = returncode
     370          self.cmd = cmd
     371 -    def __str__(self):
     372          return "Command '%s' returned non-zero exit status %d" % (self.cmd, self.returncode)
     373
     374
     375  if mswindows:
     376      try:
     377          import threading
     378      except ImportError:
     379          # SCons:  the threading module is only used by the communicate()
     380          # method, which we don't actually use, so don't worry if we
     381          # can't import it.
     382          pass
     383      import msvcrt
     384      if 0: # <-- change this to use pywin32 instead of the _subprocess driver
     385          import pywintypes
     386          from win32api import GetStdHandle, STD_INPUT_HANDLE, \
     387                               STD_OUTPUT_HANDLE, STD_ERROR_HANDLE
     388          from win32api import GetCurrentProcess, DuplicateHandle, \
     389                               GetModuleFileName, GetVersion
     390          from win32con import DUPLICATE_SAME_ACCESS, SW_HIDE
     391          from win32pipe import CreatePipe
     392          from win32process import CreateProcess, STARTUPINFO, \
     393                                   GetExitCodeProcess, STARTF_USESTDHANDLES, \
     394                                   STARTF_USESHOWWINDOW, CREATE_NEW_CONSOLE
     395          from win32event import WaitForSingleObject, INFINITE, WAIT_OBJECT_0
     396      else:
     397          from _subprocess import *
     398 -        class STARTUPINFO:
     399              dwFlags = 0
     400              hStdInput = None
     401              hStdOutput = None
     402              hStdError = None
     403              wShowWindow = 0
     404 -        class pywintypes:
     405              error = IOError
     406  else:
     407      import select
     408      import errno
     409      import fcntl
     410      import pickle
     411
     412      try:
     413          fcntl.F_GETFD
     414      except AttributeError:
     415          fcntl.F_GETFD = 1
     416
     417      try:
     418          fcntl.F_SETFD
     419      except AttributeError:
     420          fcntl.F_SETFD = 2
     421
     422  __all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call", "CalledProcessError"]
     423
     424  try:
     425      MAXFD = os.sysconf("SC_OPEN_MAX")
     426  except KeyboardInterrupt:
     427      raise       # SCons:  don't swallow keyboard interrupts
     428  except:
     429      MAXFD = 256
     430
     431  # True/False does not exist on 2.2.0
     432  try:
     433      False
     434  except NameError:
     435      False = 0
     436      True = 1
     437
     438  try:
     439      isinstance(1, int)
     440  except TypeError:
     441 -    def is_int(obj):
     442          return type(obj) == type(1)
     443 -    def is_int_or_long(obj):
     444          return type(obj) in (type(1), type(1L))
     445  else:
     446 -    def is_int(obj):
     447          return isinstance(obj, int)
     448 -    def is_int_or_long(obj):
     449          return isinstance(obj, (int, long))
     450
     451  try:
     452      types.StringTypes
     453  except AttributeError:
     454      try:
     455          types.StringTypes = (types.StringType, types.UnicodeType)
     456      except AttributeError:
     457          types.StringTypes = (types.StringType,)
     458 -    def is_string(obj):
     459          return type(obj) in types.StringTypes
     460  else:
     461 -    def is_string(obj):
     462          return isinstance(obj, types.StringTypes)
     463
     464  _active = []
     465
     466 -def _cleanup():
     467      for inst in _active[:]:
     468          if inst.poll(_deadstate=sys.maxint) >= 0:
     469              try:
     470                  _active.remove(inst)
     471              except ValueError:
     472                  # This can happen if two threads create a new Popen instance.
     473                  # It's harmless that it was already removed, so ignore.
     474                  pass
     475
     476  PIPE = -1
     477  STDOUT = -2
     478
     479
     480 -def call(*popenargs, **kwargs):
     481      """Run command with arguments.  Wait for command to complete, then
     482      return the returncode attribute.
     483
     484      The arguments are the same as for the Popen constructor.  Example:
     485
     486      retcode = call(["ls", "-l"])
     487      """
     488      return apply(Popen, popenargs, kwargs).wait()
     489
     490
     491 -def check_call(*popenargs, **kwargs):
     492      """Run command with arguments.  Wait for command to complete.  If
     493      the exit code was zero then return, otherwise raise
     494      CalledProcessError.  The CalledProcessError object will have the
     495      return code in the returncode attribute.
     496
     497      The arguments are the same as for the Popen constructor.  Example:
     498
     499      check_call(["ls", "-l"])
     500      """
     501      retcode = apply(call, popenargs, kwargs)
     502      cmd = kwargs.get("args")
     503      if cmd is None:
     504          cmd = popenargs[0]
     505      if retcode:
     506          raise CalledProcessError(retcode, cmd)
     507      return retcode
     508
     509
     510 -def list2cmdline(seq):
     511      """
     512      Translate a sequence of arguments into a command line
     513      string, using the same rules as the MS C runtime:
     514
     515      1) Arguments are delimited by white space, which is either a
     516         space or a tab.
     517
     518      2) A string surrounded by double quotation marks is
     519         interpreted as a single argument, regardless of white space
     520         contained within.  A quoted string can be embedded in an
     521         argument.
     522
     523      3) A double quotation mark preceded by a backslash is
     524         interpreted as a literal double quotation mark.
     525
     526      4) Backslashes are interpreted literally, unless they
     527         immediately precede a double quotation mark.
     528
     529      5) If backslashes immediately precede a double quotation mark,
     530         every pair of backslashes is interpreted as a literal
     531         backslash.  If the number of backslashes is odd, the last
     532         backslash escapes the next double quotation mark as
     533         described in rule 3.
     534      """
     535
     536      # See
     537      # http://msdn.microsoft.com/library/en-us/vccelng/htm/progs_12.asp
     538      result = []
     539      needquote = False
     540      for arg in seq:
     541          bs_buf = []
     542
     543          # Add a space to separate this argument from the others
     544          if result:
     545              result.append(' ')
     546
     547          needquote = (" " in arg) or ("\t" in arg)
     548          if needquote:
     549              result.append('"')
     550
     551          for c in arg:
     552              if c == '\\':
     553                  # Don't know if we need to double yet.
     554                  bs_buf.append(c)
     555              elif c == '"':
     556                  # Double backspaces.
     557                  result.append('\\' * len(bs_buf)*2)
     558                  bs_buf = []
     559                  result.append('\\"')
     560              else:
     561                  # Normal char
     562                  if bs_buf:
     563                      result.extend(bs_buf)
     564                      bs_buf = []
     565                  result.append(c)
     566
     567          # Add remaining backspaces, if any.
     568          if bs_buf:
     569              result.extend(bs_buf)
     570
     571          if needquote:
     572              result.extend(bs_buf)
     573              result.append('"')
     574
     575      return string.join(result, '')
     576
     577
     578  try:
     579      object
     580  except NameError:
     581 -    class object:
     582          pass
     583
     584 -class Popen(object):
     585 -    def __init__(self, args, bufsize=0, executable=None,
     586                   stdin=None, stdout=None, stderr=None,
     587                   preexec_fn=None, close_fds=False, shell=False,
     588                   cwd=None, env=None, universal_newlines=False,
     589                   startupinfo=None, creationflags=0):
     590          """Create new Popen instance."""
     591          _cleanup()
     592
     593          self._child_created = False
     594          if not is_int_or_long(bufsize):
     595              raise TypeError("bufsize must be an integer")
     596
     597          if mswindows:
     598              if preexec_fn is not None:
     599                  raise ValueError("preexec_fn is not supported on Windows "
     600                                   "platforms")
     601              if close_fds:
     602                  raise ValueError("close_fds is not supported on Windows "
     603                                   "platforms")
     604          else:
     605              # POSIX
     606              if startupinfo is not None:
     607                  raise ValueError("startupinfo is only supported on Windows "
     608                                   "platforms")
     609              if creationflags != 0:
     610                  raise ValueError("creationflags is only supported on Windows "
     611                                   "platforms")
     612
     613          self.stdin = None
     614          self.stdout = None
     615          self.stderr = None
     616          self.pid = None
     617          self.returncode = None
     618          self.universal_newlines = universal_newlines
     619
     620          # Input and output objects. The general principle is like
     621          # this:
     622          #
     623          # Parent                   Child
     624          # ------                   -----
     625          # p2cwrite   ---stdin--->  p2cread
     626          # c2pread    <--stdout---  c2pwrite
     627          # errread    <--stderr---  errwrite
     628          #
     629          # On POSIX, the child objects are file descriptors.  On
     630          # Windows, these are Windows file handles.  The parent objects
     631          # are file descriptors on both platforms.  The parent objects
     632          # are None when not using PIPEs. The child objects are None
     633          # when not redirecting.
     634
     635          (p2cread, p2cwrite,
     636           c2pread, c2pwrite,
     637           errread, errwrite) = self._get_handles(stdin, stdout, stderr)
     638
     639          self._execute_child(args, executable, preexec_fn, close_fds,
     640                              cwd, env, universal_newlines,
     641                              startupinfo, creationflags, shell,
     642                              p2cread, p2cwrite,
     643                              c2pread, c2pwrite,
     644                              errread, errwrite)
     645
     646          if p2cwrite:
     647              self.stdin = os.fdopen(p2cwrite, 'wb', bufsize)
     648          if c2pread:
     649              if universal_newlines:
     650                  self.stdout = os.fdopen(c2pread, 'rU', bufsize)
     651              else:
     652                  self.stdout = os.fdopen(c2pread, 'rb', bufsize)
     653          if errread:
     654              if universal_newlines:
     655                  self.stderr = os.fdopen(errread, 'rU', bufsize)
     656              else:
     657                  self.stderr = os.fdopen(errread, 'rb', bufsize)
     658
     659
     660 -    def _translate_newlines(self, data):
     661          data = data.replace("\r\n", "\n")
     662          data = data.replace("\r", "\n")
     663          return data
     664
     665
     666 -    def __del__(self):
     667          if not self._child_created:
     668              # We didn't get to successfully create a child process.
     669              return
     670          # In case the child hasn't been waited on, check if it's done.
     671          self.poll(_deadstate=sys.maxint)
     672          if self.returncode is None and _active is not None:
     673              # Child is still running, keep us alive until we can wait on it.
     674              _active.append(self)
     675
     676
     677 -    def communicate(self, input=None):
     678          """Interact with process: Send data to stdin.  Read data from
     679          stdout and stderr, until end-of-file is reached.  Wait for
     680          process to terminate.  The optional input argument should be a
     681          string to be sent to the child process, or None, if no data
     682          should be sent to the child.
     683
     684          communicate() returns a tuple (stdout, stderr)."""
     685
     686          # Optimization: If we are only using one pipe, or no pipe at
     687          # all, using select() or threads is unnecessary.
     688          if [self.stdin, self.stdout, self.stderr].count(None) >= 2:
     689              stdout = None
     690              stderr = None
     691              if self.stdin:
     692                  if input:
     693                      self.stdin.write(input)
     694                  self.stdin.close()
     695              elif self.stdout:
     696                  stdout = self.stdout.read()
     697              elif self.stderr:
     698                  stderr = self.stderr.read()
     699              self.wait()
     700              return (stdout, stderr)
     701
     702          return self._communicate(input)
     703
     704
     705      if mswindows:
     706          #
     707          # Windows methods
     708          #
     709 -        def _get_handles(self, stdin, stdout, stderr):
     710              """Construct and return tupel with IO objects:
     711              p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
     712              """
     713              if stdin is None and stdout is None and stderr is None:
     714                  return (None, None, None, None, None, None)
     715
     716              p2cread, p2cwrite = None, None
     717              c2pread, c2pwrite = None, None
     718              errread, errwrite = None, None
     719
     720              if stdin is None:
     721                  p2cread = GetStdHandle(STD_INPUT_HANDLE)
     722              elif stdin == PIPE:
     723                  p2cread, p2cwrite = CreatePipe(None, 0)
     724                  # Detach and turn into fd
     725                  p2cwrite = p2cwrite.Detach()
     726                  p2cwrite = msvcrt.open_osfhandle(p2cwrite, 0)
     727              elif is_int(stdin):
     728                  p2cread = msvcrt.get_osfhandle(stdin)
     729              else:
     730                  # Assuming file-like object
     731                  p2cread = msvcrt.get_osfhandle(stdin.fileno())
     732              p2cread = self._make_inheritable(p2cread)
     733
     734              if stdout is None:
     735                  c2pwrite = GetStdHandle(STD_OUTPUT_HANDLE)
     736              elif stdout == PIPE:
     737                  c2pread, c2pwrite = CreatePipe(None, 0)
     738                  # Detach and turn into fd
     739                  c2pread = c2pread.Detach()
     740                  c2pread = msvcrt.open_osfhandle(c2pread, 0)
     741              elif is_int(stdout):
     742                  c2pwrite = msvcrt.get_osfhandle(stdout)
     743              else:
     744                  # Assuming file-like object
     745                  c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
     746              c2pwrite = self._make_inheritable(c2pwrite)
     747
     748              if stderr is None:
     749                  errwrite = GetStdHandle(STD_ERROR_HANDLE)
     750              elif stderr == PIPE:
     751                  errread, errwrite = CreatePipe(None, 0)
     752                  # Detach and turn into fd
     753                  errread = errread.Detach()
     754                  errread = msvcrt.open_osfhandle(errread, 0)
     755              elif stderr == STDOUT:
     756                  errwrite = c2pwrite
     757              elif is_int(stderr):
     758                  errwrite = msvcrt.get_osfhandle(stderr)
     759              else:
     760                  # Assuming file-like object
     761                  errwrite = msvcrt.get_osfhandle(stderr.fileno())
     762              errwrite = self._make_inheritable(errwrite)
     763
     764              return (p2cread, p2cwrite,
     765                      c2pread, c2pwrite,
     766                      errread, errwrite)
     767
     768
     769 -        def _make_inheritable(self, handle):
     770              """Return a duplicate of handle, which is inheritable"""
     771              return DuplicateHandle(GetCurrentProcess(), handle,
     772                                     GetCurrentProcess(), 0, 1,
     773                                     DUPLICATE_SAME_ACCESS)
     774
     775
     776 -        def _find_w9xpopen(self):
     777              """Find and return absolut path to w9xpopen.exe"""
     778              w9xpopen = os.path.join(os.path.dirname(GetModuleFileName(0)),
     779                                      "w9xpopen.exe")
     780              if not os.path.exists(w9xpopen):
     781                  # Eeek - file-not-found - possibly an embedding
     782                  # situation - see if we can locate it in sys.exec_prefix
     783                  w9xpopen = os.path.join(os.path.dirname(sys.exec_prefix),
     784                                          "w9xpopen.exe")
     785                  if not os.path.exists(w9xpopen):
     786                      raise RuntimeError("Cannot locate w9xpopen.exe, which is "
     787                                         "needed for Popen to work with your "
     788                                         "shell or platform.")
     789              return w9xpopen
     790
     791
     792 -        def _execute_child(self, args, executable, preexec_fn, close_fds,
     793                             cwd, env, universal_newlines,
     794                             startupinfo, creationflags, shell,
     795                             p2cread, p2cwrite,
     796                             c2pread, c2pwrite,
     797                             errread, errwrite):
     798              """Execute program (MS Windows version)"""
     799
     800              if not isinstance(args, types.StringTypes):
     801                  args = list2cmdline(args)
     802
     803              # Process startup details
     804              if startupinfo is None:
     805                  startupinfo = STARTUPINFO()
     806              if None not in (p2cread, c2pwrite, errwrite):
     807                  startupinfo.dwFlags = startupinfo.dwFlags | STARTF_USESTDHANDLES
     808                  startupinfo.hStdInput = p2cread
     809                  startupinfo.hStdOutput = c2pwrite
     810                  startupinfo.hStdError = errwrite
     811
     812              if shell:
     813                  startupinfo.dwFlags = startupinfo.dwFlags | STARTF_USESHOWWINDOW
     814                  startupinfo.wShowWindow = SW_HIDE
     815                  comspec = os.environ.get("COMSPEC", "cmd.exe")
     816                  args = comspec + " /c " + args
     817                  if (GetVersion() >= 0x80000000L or
     818                          os.path.basename(comspec).lower() == "command.com"):
     819                      # Win9x, or using command.com on NT. We need to
     820                      # use the w9xpopen intermediate program. For more
     821                      # information, see KB Q150956
     822                      # (http://web.archive.org/web/20011105084002/http://support.microsoft.com/support/kb/articles/Q150/9/56.asp)
     823                      w9xpopen = self._find_w9xpopen()
     824                      args = '"%s" %s' % (w9xpopen, args)
     825                      # Not passing CREATE_NEW_CONSOLE has been known to
     826                      # cause random failures on win9x.  Specifically a
     827                      # dialog: "Your program accessed mem currently in
     828                      # use at xxx" and a hopeful warning about the
     829                      # stability of your system.  Cost is Ctrl+C wont
     830                      # kill children.
     831                      creationflags = creationflags | CREATE_NEW_CONSOLE
     832
     833              # Start the process
     834              try:
     835                  hp, ht, pid, tid = CreateProcess(executable, args,
     836                                           # no special security
     837                                           None, None,
     838                                           # must inherit handles to pass std
     839                                           # handles
     840                                           1,
     841                                           creationflags,
     842                                           env,
     843                                           cwd,
     844                                           startupinfo)
     845              except pywintypes.error, e:
     846                  # Translate pywintypes.error to WindowsError, which is
     847                  # a subclass of OSError.  FIXME: We should really
     848                  # translate errno using _sys_errlist (or simliar), but
     849                  # how can this be done from Python?
     850                  raise apply(WindowsError, e.args)
     851
     852              # Retain the process handle, but close the thread handle
     853              self._child_created = True
     854              self._handle = hp
     855              self.pid = pid
     856              ht.Close()
     857
     858              # Child is launched. Close the parent's copy of those pipe
     859              # handles that only the child should have open.  You need
     860              # to make sure that no handles to the write end of the
     861              # output pipe are maintained in this process or else the
     862              # pipe will not close when the child process exits and the
     863              # ReadFile will hang.
     864              if p2cread is not None:
     865                  p2cread.Close()
     866              if c2pwrite is not None:
     867                  c2pwrite.Close()
     868              if errwrite is not None:
     869                  errwrite.Close()
     870
     871
     872 -        def poll(self, _deadstate=None):
     873              """Check if child process has terminated.  Returns returncode
     874              attribute."""
     875              if self.returncode is None:
     876                  if WaitForSingleObject(self._handle, 0) == WAIT_OBJECT_0:
     877                      self.returncode = GetExitCodeProcess(self._handle)
     878              return self.returncode
     879
     880
     881 -        def wait(self):
     882              """Wait for child process to terminate.  Returns returncode
     883              attribute."""
     884              if self.returncode is None:
     885                  obj = WaitForSingleObject(self._handle, INFINITE)
     886                  self.returncode = GetExitCodeProcess(self._handle)
     887              return self.returncode
     888
     889
     890 -        def _readerthread(self, fh, buffer):
     891              buffer.append(fh.read())
     892
     893
     894 -        def _communicate(self, input):
     895              stdout = None # Return
     896              stderr = None # Return
     897
     898              if self.stdout:
     899                  stdout = []
     900                  stdout_thread = threading.Thread(target=self._readerthread,
     901                                                   args=(self.stdout, stdout))
     902                  stdout_thread.setDaemon(True)
     903                  stdout_thread.start()
     904              if self.stderr:
     905                  stderr = []
     906                  stderr_thread = threading.Thread(target=self._readerthread,
     907                                                   args=(self.stderr, stderr))
     908                  stderr_thread.setDaemon(True)
     909                  stderr_thread.start()
     910
     911              if self.stdin:
     912                  if input is not None:
     913                      self.stdin.write(input)
     914                  self.stdin.close()
     915
     916              if self.stdout:
     917                  stdout_thread.join()
     918              if self.stderr:
     919                  stderr_thread.join()
     920
     921              # All data exchanged.  Translate lists into strings.
     922              if stdout is not None:
     923                  stdout = stdout[0]
     924              if stderr is not None:
     925                  stderr = stderr[0]
     926
     927              # Translate newlines, if requested.  We cannot let the file
     928              # object do the translation: It is based on stdio, which is
     929              # impossible to combine with select (unless forcing no
     930              # buffering).
     931              if self.universal_newlines and hasattr(file, 'newlines'):
     932                  if stdout:
     933                      stdout = self._translate_newlines(stdout)
     934                  if stderr:
     935                      stderr = self._translate_newlines(stderr)
     936
     937              self.wait()
     938              return (stdout, stderr)
     939
     940      else:
     941          #
     942          # POSIX methods
     943          #
     944 -        def _get_handles(self, stdin, stdout, stderr):
     945              """Construct and return tupel with IO objects:
     946              p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
     947              """
     948              p2cread, p2cwrite = None, None
     949              c2pread, c2pwrite = None, None
     950              errread, errwrite = None, None
     951
     952              if stdin is None:
     953                  pass
     954              elif stdin == PIPE:
     955                  p2cread, p2cwrite = os.pipe()
     956              elif is_int(stdin):
     957                  p2cread = stdin
     958              else:
     959                  # Assuming file-like object
     960                  p2cread = stdin.fileno()
     961
     962              if stdout is None:
     963                  pass
     964              elif stdout == PIPE:
     965                  c2pread, c2pwrite = os.pipe()
     966              elif is_int(stdout):
     967                  c2pwrite = stdout
     968              else:
     969                  # Assuming file-like object
     970                  c2pwrite = stdout.fileno()
     971
     972              if stderr is None:
     973                  pass
     974              elif stderr == PIPE:
     975                  errread, errwrite = os.pipe()
     976              elif stderr == STDOUT:
     977                  errwrite = c2pwrite
     978              elif is_int(stderr):
     979                  errwrite = stderr
     980              else:
     981                  # Assuming file-like object
     982                  errwrite = stderr.fileno()
     983
     984              return (p2cread, p2cwrite,
     985                      c2pread, c2pwrite,
     986                      errread, errwrite)
     987
     988
     989 -        def _set_cloexec_flag(self, fd):
     990              try:
     991                  cloexec_flag = fcntl.FD_CLOEXEC
     992              except AttributeError:
     993                  cloexec_flag = 1
     994
     995              old = fcntl.fcntl(fd, fcntl.F_GETFD)
     996              fcntl.fcntl(fd, fcntl.F_SETFD, old | cloexec_flag)
     997
     998
     999 -        def _close_fds(self, but):
    1000              for i in xrange(3, MAXFD):
    1001                  if i == but:
    1002                      continue
    1003                  try:
    1004                      os.close(i)
    1005                  except KeyboardInterrupt:
    1006                      raise       # SCons:  don't swallow keyboard interrupts
    1007                  except:
    1008                      pass
    1009
    1010
    1011 -        def _execute_child(self, args, executable, preexec_fn, close_fds,
    1012                             cwd, env, universal_newlines,
    1013                             startupinfo, creationflags, shell,
    1014                             p2cread, p2cwrite,
    1015                             c2pread, c2pwrite,
    1016                             errread, errwrite):
    1017              """Execute program (POSIX version)"""
    1018
    1019              if is_string(args):
    1020                  args = [args]
    1021
    1022              if shell:
    1023                  args = ["/bin/sh", "-c"] + args
    1024
    1025              if executable is None:
    1026                  executable = args[0]
    1027
    1028              # For transferring possible exec failure from child to parent
    1029              # The first char specifies the exception type: 0 means
    1030              # OSError, 1 means some other error.
    1031              errpipe_read, errpipe_write = os.pipe()
    1032              self._set_cloexec_flag(errpipe_write)
    1033
    1034              self.pid = os.fork()
    1035              self._child_created = True
    1036              if self.pid == 0:
    1037                  # Child
    1038                  try:
    1039                      # Close parent's pipe ends
    1040                      if p2cwrite:
    1041                          os.close(p2cwrite)
    1042                      if c2pread:
    1043                          os.close(c2pread)
    1044                      if errread:
    1045                          os.close(errread)
    1046                      os.close(errpipe_read)
    1047
    1048                      # Dup fds for child
    1049                      if p2cread:
    1050                          os.dup2(p2cread, 0)
    1051                      if c2pwrite:
    1052                          os.dup2(c2pwrite, 1)
    1053                      if errwrite:
    1054                          os.dup2(errwrite, 2)
    1055
    1056                      # Close pipe fds.  Make sure we don't close the same
    1057                      # fd more than once, or standard fds.
    1058                      try:
    1059                          set
    1060                      except NameError:
    1061                          # Fall-back for earlier Python versions, so epydoc
    1062                          # can use this module directly to execute things.
    1063                          if p2cread:
    1064                              os.close(p2cread)
    1065                          if c2pwrite and c2pwrite not in (p2cread,):
    1066                              os.close(c2pwrite)
    1067                          if errwrite and errwrite not in (p2cread, c2pwrite):
    1068                              os.close(errwrite)
    1069                      else:
    1070                          for fd in set((p2cread, c2pwrite, errwrite))-set((0,1,2)):
    1071                              if fd: os.close(fd)
    1072
    1073                      # Close all other fds, if asked for
    1074                      if close_fds:
    1075                          self._close_fds(but=errpipe_write)
    1076
    1077                      if cwd is not None:
    1078                          os.chdir(cwd)
    1079
    1080                      if preexec_fn:
    1081                          apply(preexec_fn)
    1082
    1083                      if env is None:
    1084                          os.execvp(executable, args)
    1085                      else:
    1086                          os.execvpe(executable, args, env)
    1087
    1088                  except KeyboardInterrupt:
    1089                      raise       # SCons:  don't swallow keyboard interrupts
    1090
    1091                  except:
    1092                      exc_type, exc_value, tb = sys.exc_info()
    1093                      # Save the traceback and attach it to the exception object
    1094                      exc_lines = traceback.format_exception(exc_type,
    1095                                                             exc_value,
    1096                                                             tb)
    1097                      exc_value.child_traceback = string.join(exc_lines, '')
    1098                      os.write(errpipe_write, pickle.dumps(exc_value))
    1099
    1100                  # This exitcode won't be reported to applications, so it
    1101                  # really doesn't matter what we return.
    1102                  os._exit(255)
    1103
    1104              # Parent
    1105              os.close(errpipe_write)
    1106              if p2cread and p2cwrite:
    1107                  os.close(p2cread)
    1108              if c2pwrite and c2pread:
    1109                  os.close(c2pwrite)
    1110              if errwrite and errread:
    1111                  os.close(errwrite)
    1112
    1113              # Wait for exec to fail or succeed; possibly raising exception
    1114              data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
    1115              os.close(errpipe_read)
    1116              if data != "":
    1117                  os.waitpid(self.pid, 0)
    1118                  child_exception = pickle.loads(data)
    1119                  raise child_exception
    1120
    1121
    1122 -        def _handle_exitstatus(self, sts):
    1123              if os.WIFSIGNALED(sts):
    1124                  self.returncode = -os.WTERMSIG(sts)
    1125              elif os.WIFEXITED(sts):
    1126                  self.returncode = os.WEXITSTATUS(sts)
    1127              else:
    1128                  # Should never happen
    1129                  raise RuntimeError("Unknown child exit status!")
    1130
    1131
    1132 -        def poll(self, _deadstate=None):
    1133              """Check if child process has terminated.  Returns returncode
    1134              attribute."""
    1135              if self.returncode is None:
    1136                  try:
    1137                      pid, sts = os.waitpid(self.pid, os.WNOHANG)
    1138                      if pid == self.pid:
    1139                          self._handle_exitstatus(sts)
    1140                  except os.error:
    1141                      if _deadstate is not None:
    1142                          self.returncode = _deadstate
    1143              return self.returncode
    1144
    1145
    1146 -        def wait(self):
    1147              """Wait for child process to terminate.  Returns returncode
    1148              attribute."""
    1149              if self.returncode is None:
    1150                  pid, sts = os.waitpid(self.pid, 0)
    1151                  self._handle_exitstatus(sts)
    1152              return self.returncode
    1153
    1154
    1155 -        def _communicate(self, input):
    1156              read_set = []
    1157              write_set = []
    1158              stdout = None # Return
    1159              stderr = None # Return
    1160
    1161              if self.stdin:
    1162                  # Flush stdio buffer.  This might block, if the user has
    1163                  # been writing to .stdin in an uncontrolled fashion.
    1164                  self.stdin.flush()
    1165                  if input:
    1166                      write_set.append(self.stdin)
    1167                  else:
    1168                      self.stdin.close()
    1169              if self.stdout:
    1170                  read_set.append(self.stdout)
    1171                  stdout = []
    1172              if self.stderr:
    1173                  read_set.append(self.stderr)
    1174                  stderr = []
    1175
    1176              input_offset = 0
    1177              while read_set or write_set:
    1178                  rlist, wlist, xlist = select.select(read_set, write_set, [])
    1179
    1180                  if self.stdin in wlist:
    1181                      # When select has indicated that the file is writable,
    1182                      # we can write up to PIPE_BUF bytes without risk
    1183                      # blocking.  POSIX defines PIPE_BUF >= 512
    1184                      bytes_written = os.write(self.stdin.fileno(), buffer(input, input_offset, 512))
    1185                      input_offset = input_offset + bytes_written
    1186                      if input_offset >= len(input):
    1187                          self.stdin.close()
    1188                          write_set.remove(self.stdin)
    1189
    1190                  if self.stdout in rlist:
    1191                      data = os.read(self.stdout.fileno(), 1024)
    1192                      if data == "":
    1193                          self.stdout.close()
    1194                          read_set.remove(self.stdout)
    1195                      stdout.append(data)
    1196
    1197                  if self.stderr in rlist:
    1198                      data = os.read(self.stderr.fileno(), 1024)
    1199                      if data == "":
    1200                          self.stderr.close()
    1201                          read_set.remove(self.stderr)
    1202                      stderr.append(data)
    1203
    1204              # All data exchanged.  Translate lists into strings.
    1205              if stdout is not None:
    1206                  stdout = string.join(stdout, '')
    1207              if stderr is not None:
    1208                  stderr = string.join(stderr, '')
    1209
    1210              # Translate newlines, if requested.  We cannot let the file
    1211              # object do the translation: It is based on stdio, which is
    1212              # impossible to combine with select (unless forcing no
    1213              # buffering).
    1214              if self.universal_newlines and hasattr(file, 'newlines'):
    1215                  if stdout:
    1216                      stdout = self._translate_newlines(stdout)
    1217                  if stderr:
    1218                      stderr = self._translate_newlines(stderr)
    1219
    1220              self.wait()
    1221              return (stdout, stderr)
    1222
    1223
    1224 -def _demo_posix():
    1225      #
    1226      # Example 1: Simple redirection: Get process list
    1227      #
    1228      plist = Popen(["ps"], stdout=PIPE).communicate()[0]
    1229      print "Process list:"
    1230      print plist
    1231
    1232      #
    1233      # Example 2: Change uid before executing child
    1234      #
    1235      if os.getuid() == 0:
    1236          p = Popen(["id"], preexec_fn=lambda: os.setuid(100))
    1237          p.wait()
    1238
    1239      #
    1240      # Example 3: Connecting several subprocesses
    1241      #
    1242      print "Looking for 'hda'..."
    1243      p1 = Popen(["dmesg"], stdout=PIPE)
    1244      p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
    1245      print repr(p2.communicate()[0])
    1246
    1247      #
    1248      # Example 4: Catch execution error
    1249      #
    1250      print
    1251      print "Trying a weird file..."
    1252      try:
    1253          print Popen(["/this/path/does/not/exist"]).communicate()
    1254      except OSError, e:
    1255          if e.errno == errno.ENOENT:
    1256              print "The file didn't exist.  I thought so..."
    1257              print "Child traceback:"
    1258              print e.child_traceback
    1259          else:
    1260              print "Error", e.errno
    1261      else:
    1262          sys.stderr.write( "Gosh.  No error.\n" )
    1263
    1264
    1265 -def _demo_windows():
    1266      #
    1267      # Example 1: Connecting several subprocesses
    1268      #
    1269      print "Looking for 'PROMPT' in set output..."
    1270      p1 = Popen("set", stdout=PIPE, shell=True)
    1271      p2 = Popen('find "PROMPT"', stdin=p1.stdout, stdout=PIPE)
    1272      print repr(p2.communicate()[0])
    1273
    1274      #
    1275      # Example 2: Simple execution of program
    1276      #
    1277      print "Executing calc..."
    1278      p = Popen("calc")
    1279      p.wait()
    1280
    1281
    1282  if __name__ == "__main__":
    1283      if mswindows:
    1284          _demo_windows()
    1285      else:
    1286          _demo_posix()
    1287