PEP: 3145 Title: Asynchronous I/O For subprocess.Popen Version:
$Revision$ Last-Modified: $Date$ Author: (James) Eric Pruitt, Charles R.
McCreary, Josiah Carlson Status: Withdrawn Type: Standards Track
Content-Type: text/x-rst Created: 04-Aug-2009 Python-Version: 3.2
Post-History:

Abstract
========

In its present form, the `subprocess.Popen` implementation is prone to
dead-locking and blocking of the parent Python script while waiting on
data from the child process. This PEP proposes to make
`subprocess.Popen` more asynchronous to help alleviate these problems.

PEP Deferral
============

Further exploration of the concepts covered in this PEP has been
deferred at least until after PEP 3156 has been resolved.

PEP Withdrawal
==============

This can be dealt with in the bug tracker. A specific proposal is
attached to \[11\]\_.

Motivation
==========

A search for "python asynchronous subprocess" will turn up numerous
accounts of people wanting to execute a child process and communicate
with it from time to time reading only the data that is available
instead of blocking to wait for the program to produce data \[1\]\_
\[2\]\_ \[3\]*. The current behavior of the `subprocess` module is that
when a user sends or receives data via the stdin, stderr and stdout file
objects, dead locks are common and documented \[4\]* \[5\]\_. While
communicate can be used to alleviate some of the buffering issues, it
will still cause the parent process to block while attempting to read
data when none is available to be read from the child process.

Rationale
=========

There is a documented need for asynchronous, non-blocking functionality
in `subprocess.Popen` \[6\]\_ \[7\]\_ \[2\]\_ \[3\]\_. Inclusion of the
code would improve the utility of the Python standard library that can
be used on Unix based and Windows builds of Python. Practically every
I/O object in Python has a file-like wrapper of some sort. Sockets
already act as such and for strings there is `StringIO`. Popen can be
made to act like a file by simply using the methods attached to the
`subprocess.Popen.stderr`, stdout and stdin file-like objects. But when
using the read and write methods of those options, you do not have the
benefit of asynchronous I/O. In the proposed solution the wrapper wraps
the asynchronous methods to mimic a file object.

Reference Implementation
========================

I have been maintaining a Google Code repository that contains all of my
changes including tests and documentation \[9\]\_ as well as blog
detailing the problems I have come across in the development process
\[10\]\_.

I have been working on implementing non-blocking asynchronous I/O in the
`subprocess` module as well as a wrapper class for `subprocess.Popen`
that makes it so that an executed process can take the place of a file
by duplicating all of the methods and attributes that file objects have.

There are two base functions that have been added to the
`subprocess.Popen` class: `Popen.send` and `Popen._recv`, each with two
separate implementations, one for Windows and one for Unix-based
systems. The Windows implementation uses ctypes to access the functions
needed to control pipes in the kernel 32 DLL in an asynchronous manner.
On Unix based systems, the Python interface for file control serves the
same purpose. The different implementations of `Popen.send` and
`Popen._recv` have identical arguments to make code that uses these
functions work across multiple platforms.

When calling the `Popen._recv` function, it requires the pipe name be
passed as an argument so there exists the `Popen.recv` function that
passes selects stdout as the pipe for `Popen._recv` by default.
`Popen.recv_err` selects stderr as the pipe by default. `Popen.recv` and
`Popen.recv_err` are much easier to read and understand than
`Popen._recv('stdout' ...)` and `Popen._recv('stderr' ...)`
respectively.

Since the `Popen._recv` function does not wait on data to be produced
before returning a value, it may return empty bytes. `Popen.asyncread`
handles this issue by returning all data read over a given time
interval.

The `ProcessIOWrapper` class uses the `asyncread` and `asyncwrite`
functions to allow a process to act like a file so that there are no
blocking issues that can arise from using the stdout and stdin file
objects produced from a `subprocess.Popen` call.

References
==========

.. \[1\] \[ python-Feature Requests-1191964 \] asynchronous Subprocess
https://mail.python.org/pipermail/python-bugs-list/2006-December/
036524.html

.. \[2\] Daily Life in an Ivory Basement :
/feb-07/problems-with-subprocess
http://ivory.idyll.org/blog/feb-07/problems-with-subprocess

.. \[3\] How can I run an external command asynchronously from Python? -
Stack Overflow
http://stackoverflow.com/questions/636561/how-can-i-run-an-external-
command-asynchronously-from-python

.. \[4\] 18.1. subprocess - Subprocess management - Python v2.6.2
documentation
http://docs.python.org/library/subprocess.html\#subprocess.Popen.wait

.. \[5\] 18.1. subprocess - Subprocess management - Python v2.6.2
documentation
http://docs.python.org/library/subprocess.html\#subprocess.Popen.kill

.. \[6\] Issue 1191964: asynchronous Subprocess - Python tracker
http://bugs.python.org/issue1191964

.. \[7\] Module to allow Asynchronous subprocess use on Windows and
Posix platforms - ActiveState Code
http://code.activestate.com/recipes/440554/

.. \[8\] subprocess.rst - subprocdev - Project Hosting on Google Code
http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5\#437

.. \[9\] subprocdev - Project Hosting on Google Code
http://code.google.com/p/subprocdev

.. \[10\] Python Subprocess Dev http://subdev.blogspot.com/

.. \[11\] https://bugs.python.org/issue18823 -- Idle: use pipes instead
of sockets to talk with user subprocess

Copyright
=========

This P.E.P. is licensed under the Open Publication License;
http://www.opencontent.org/openpub/.
