PEP: 319 Title: Python Synchronize/Asynchronize Block Version:
$Revision$ Last-Modified: $Date$ Author: Michel Pelletier
<michel@users.sourceforge.net> Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 24-Feb-2003 Python-Version: 2.4?
Post-History:

Abstract
========

This PEP proposes adding two new keywords to Python, 'synchronize' and
'asynchronize'.

Pronouncement
=============

This PEP is rejected in favor of PEP 343.

The 'synchronize' Keyword The concept of code synchronization in Python
is too low-level. To synchronize code a programmer must be aware of the
details of the following pseudo-code pattern::

        initialize_lock()

        ...

        acquire_lock()
        try:
            change_shared_data()
        finally:
            release_lock()

    This synchronized block pattern is not the only pattern (more
    discussed below) but it is very common.  This PEP proposes
    replacing the above code with the following equivalent::

        synchronize:
            change_shared_data()

    The advantages of this scheme are simpler syntax and less room for
    user error.  Currently users are required to write code about
    acquiring and releasing thread locks in 'try/finally' blocks;
    errors in this code can cause notoriously difficult concurrent
    thread locking issues.

The 'asynchronize' Keyword While executing a 'synchronize' block of code
a programmer may want to "drop back" to running asynchronously
momentarily to run blocking input/output routines or something else that
might take a indeterminate amount of time and does not require
synchronization. This code usually follows the pattern::

        initialize_lock()

        ...

        acquire_lock()
        try:
            change_shared_data()
            release_lock()             # become async
            do_blocking_io()
            acquire_lock()             # sync again
            change_shared_data2()

        finally:
            release_lock()

    The asynchronous section of the code is not very obvious visually,
    so it is marked up with comments.  Using the proposed
    'asynchronize' keyword this code becomes much cleaner, easier to
    understand, and less prone to error::

        synchronize:
            change_shared_data()

            asynchronize:
               do_blocking_io()

            change_shared_data2()

    Encountering an 'asynchronize' keyword inside a non-synchronized
    block can raise either an error or issue a warning (as all code
    blocks are implicitly asynchronous anyway).  It is important to
    note that the above example is **not** the same as::

        synchronize:
            change_shared_data()

        do_blocking_io()

        synchronize:
            change_shared_data2()

    Because both synchronized blocks of code may be running inside the
    same iteration of a loop, Consider::

        while in_main_loop():
            synchronize:
                change_shared_data()

                asynchronize:
                   do_blocking_io()

                change_shared_data2()

    Many threads may be looping through this code.  Without the
    'asynchronize' keyword one thread cannot stay in the loop and
    release the lock at the same time while blocking IO is going on.
    This pattern of releasing locks inside a main loop to do blocking
    IO is used extensively inside the CPython interpreter itself.

Synchronization Targets
=======================

As proposed the 'synchronize' and 'asynchronize' keywords synchronize a
block of code. However programmers may want to specify a target object
that threads synchronize on. Any object can be a synchronization target.

Consider a two-way queue object: two different objects are used by the
same 'synchronize' code block to synchronize both queues separately in
the 'get' method::

    class TwoWayQueue:
        def __init__(self):
            self.front = []
            self.rear = []

        def putFront(self, item):
            self.put(item, self.front)

        def getFront(self):
            item = self.get(self.front)
            return item

        def putRear(self, item):
            self.put(item, self.rear)

        def getRear(self):
            item = self.get(self.rear)
            return item

        def put(self, item, queue):
            synchronize queue:
                queue.append(item)

        def get(self, queue):
            synchronize queue:
                item = queue[0]
                del queue[0]
                return item

Here is the equivalent code in Python as it is now without a
'synchronize' keyword::

    import thread

    class LockableQueue:

        def __init__(self):
            self.queue = []
            self.lock = thread.allocate_lock()

    class TwoWayQueue:
        def __init__(self):
            self.front = LockableQueue()
            self.rear = LockableQueue()

        def putFront(self, item):
            self.put(item, self.front)

        def getFront(self):
            item = self.get(self.front)
            return item

        def putRear(self, item):
            self.put(item, self.rear)

        def getRear(self):
            item = self.get(self.rear)
            return item

        def put(self, item, queue):
            queue.lock.acquire()
            try:
                queue.append(item)
            finally:
                queue.lock.release()

        def get(self, queue):
            queue.lock.acquire()
            try:
                item = queue[0]
                del queue[0]
                return item
            finally:
                queue.lock.release()

The last example had to define an extra class to associate a lock with
the queue where the first example the 'synchronize' keyword does this
association internally and transparently.

Other Patterns that Synchronize
===============================

There are some situations where the 'synchronize' and 'asynchronize'
keywords cannot entirely replace the use of lock methods like `acquire`
and `release`. Some examples are if the programmer wants to provide
arguments for `acquire` or if a lock is acquired in one code block but
released in another, as shown below.

Here is a class from Zope modified to use both the 'synchronize' and
'asynchronize' keywords and also uses a pool of explicit locks that are
acquired and released in different code blocks and thus don't use
'synchronize'::

    import thread
    from ZServerPublisher import ZServerPublisher

    class ZRendevous:

        def __init__(self, n=1):
            pool=[]
            self._lists=pool, [], []

            synchronize:
                while n > 0:
                    l=thread.allocate_lock()
                    l.acquire()
                    pool.append(l)
                    thread.start_new_thread(ZServerPublisher,
                                            (self.accept,))
                    n=n-1

        def accept(self):
            synchronize:
                pool, requests, ready = self._lists
                while not requests:
                    l=pool[-1]
                    del pool[-1]
                    ready.append(l)

                    asynchronize:
                        l.acquire()

                    pool.append(l)

                r=requests[0]
                del requests[0]
                return r

        def handle(self, name, request, response):
            synchronize:
                pool, requests, ready = self._lists
                requests.append((name, request, response))
                if ready:
                    l=ready[-1]
                    del ready[-1]
                    l.release()

Here is the original class as found in the
'Zope/ZServer/PubCore/ZRendevous.py' module. The "convenience" of the
'\_a' and '\_r' shortcut names obscure the code::

    import thread
    from ZServerPublisher import ZServerPublisher

    class ZRendevous:

        def __init__(self, n=1):
            sync=thread.allocate_lock()
            self._a=sync.acquire
            self._r=sync.release
            pool=[]
            self._lists=pool, [], []
            self._a()
            try:
                while n > 0:
                    l=thread.allocate_lock()
                    l.acquire()
                    pool.append(l)
                    thread.start_new_thread(ZServerPublisher,
                                            (self.accept,))
                    n=n-1
            finally: self._r()

        def accept(self):
            self._a()
            try:
                pool, requests, ready = self._lists
                while not requests:
                    l=pool[-1]
                    del pool[-1]
                    ready.append(l)
                    self._r()
                    l.acquire()
                    self._a()
                    pool.append(l)

                r=requests[0]
                del requests[0]
                return r
            finally: self._r()

        def handle(self, name, request, response):
            self._a()
            try:
                pool, requests, ready = self._lists
                requests.append((name, request, response))
                if ready:
                    l=ready[-1]
                    del ready[-1]
                    l.release()
            finally: self._r()

In particular the asynchronize section of the `accept` method is not
very obvious. To beginner programmers, 'synchronize' and 'asynchronize'
remove many of the problems encountered when juggling multiple `acquire`
and `release` methods on different locks in different `try/finally`
blocks.

Formal Syntax
=============

Python syntax is defined in a modified BNF grammar notation described in
the Python Language Reference \[1\]\_. This section describes the
proposed synchronization syntax using this grammar::

    synchronize_stmt: 'synchronize' [test] ':' suite
    asynchronize_stmt: 'asynchronize' [test] ':' suite
    compound_stmt: ... | synchronized_stmt | asynchronize_stmt

(The '...' indicates other compound statements elided).

Proposed Implementation
=======================

The author of this PEP has not explored an implementation yet. There are
several implementation issues that must be resolved. The main
implementation issue is what exactly gets locked and unlocked during a
synchronized block.

During an unqualified synchronized block (the use of the 'synchronize'
keyword without a target argument) a lock could be created and
associated with the synchronized code block object. Any threads that are
to execute the block must first acquire the code block lock.

When an 'asynchronize' keyword is encountered in a 'synchronize' block
the code block lock is unlocked before the inner block is executed and
re-locked when the inner block terminates.

When a synchronized block target is specified the object is associated
with a lock. How this is implemented cleanly is probably the highest
risk of this proposal. Java Virtual Machines typically associate a
special hidden lock object with target object and use it to synchronized
the block around the target only.

Backward Compatibility
======================

Backward compatibility is solved with the new `from __future__` Python
syntax \[2\]*, and the new warning framework \[3\]* to evolve the Python
language into phasing out any conflicting names that use the new
keywords 'synchronize' and 'asynchronize'. To use the syntax now, a
developer could use the statement::

    from __future__ import threadsync  # or whatever

In addition, any code that uses the keyword 'synchronize' or
'asynchronize' as an identifier will be issued a warning from Python.
After the appropriate period of time, the syntax would become standard,
the above import statement would do nothing, and any identifiers named
'synchronize' or 'asynchronize' would raise an exception.

PEP 310 Reliable Acquisition/Release Pairs
==========================================

PEP 310 \[4\]\_ proposes the 'with' keyword that can serve the same
function as 'synchronize' (but no facility for 'asynchronize'). The
pattern::

    initialize_lock()

    with the_lock:
        change_shared_data()

is equivalent to the proposed::

    synchronize the_lock:
        change_shared_data()

PEP 310 must synchronize on an existing lock, while this PEP proposes
that unqualified 'synchronize' statements synchronize on a global,
internal, transparent lock in addition to qualified 'synchronize'
statements. The 'with' statement also requires lock initialization,
while the 'synchronize' statement can synchronize on any target object
**including** locks.

While limited in this fashion, the 'with' statement is more abstract and
serves more purposes than synchronization. For example, transactions
could be used with the 'with' keyword::

    initialize_transaction()

    with my_transaction:
        do_in_transaction()

    # when the block terminates, the transaction is committed.

The 'synchronize' and 'asynchronize' keywords cannot serve this or any
other general acquire/release pattern other than thread synchronization.

How Java Does It
================

Java defines a 'synchronized' keyword (note the grammatical tense
different between the Java keyword and this PEP's 'synchronize') which
must be qualified on any object. The syntax is::

    synchronized (Expression) Block

Expression must yield a valid object (null raises an error and
exceptions during 'Expression' terminate the 'synchronized' block for
the same reason) upon which 'Block' is synchronized.

How Jython Does It
==================

Jython uses a 'synchronize' class with the static method
'make\_synchronized' that accepts one callable argument and returns a
newly created, synchronized, callable "wrapper" around the argument.

Summary of Proposed Changes to Python
=====================================

Adding new 'synchronize' and 'asynchronize' keywords to the language.

Risks
=====

This PEP proposes adding two keywords to the Python language. This may
break code.

There is no implementation to test.

It's not the most important problem facing Python programmers today
(although it is a fairly notorious one).

The equivalent Java keyword is the past participle 'synchronized'. This
PEP proposes the present tense, 'synchronize' as being more in spirit
with Python (there being less distinction between compile-time and
run-time in Python than Java).

Dissenting Opinion
==================

This PEP has not been discussed on python-dev.

References
==========

.. \[1\] The Python Language Reference http://docs.python.org/reference/

.. \[2\] PEP 236, Back to the **future**, Peters
http://www.python.org/dev/peps/pep-0236/

.. \[3\] PEP 230, Warning Framework, van Rossum
http://www.python.org/dev/peps/pep-0230/

.. \[4\] PEP 310, Reliable Acquisition/Release Pairs, Hudson, Moore
http://www.python.org/dev/peps/pep-0310/

Copyright
=========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 End:
