PEP: 3104 Title: Access to Names in Outer Scopes Version: $Revision$
Last-Modified: $Date$ Author: Ka-Ping Yee <ping@zesty.ca> Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 12-Oct-2006
Python-Version: 3.0 Post-History:

Abstract
========

In most languages that support nested scopes, code can refer to or
rebind (assign to) any name in the nearest enclosing scope. Currently,
Python code can refer to a name in any enclosing scope, but it can only
rebind names in two scopes: the local scope (by simple assignment) or
the module-global scope (using a `global` declaration).

This limitation has been raised many times on the Python-Dev mailing
list and elsewhere, and has led to extended discussion and many
proposals for ways to remove this limitation. This PEP summarizes the
various alternatives that have been suggested, together with advantages
and disadvantages that have been mentioned for each.

Rationale
=========

Before version 2.1, Python's treatment of scopes resembled that of
standard C: within a file there were only two levels of scope, global
and local. In C, this is a natural consequence of the fact that function
definitions cannot be nested. But in Python, though functions are
usually defined at the top level, a function definition can be executed
anywhere. This gave Python the syntactic appearance of nested scoping
without the semantics, and yielded inconsistencies that were surprising
to some programmers -- for example, a recursive function that worked at
the top level would cease to work when moved inside another function,
because the recursive function's own name would no longer be visible in
its body's scope. This violates the intuition that a function should
behave consistently when placed in different contexts. Here's an
example::

    def enclosing_function():
        def factorial(n):
            if n < 2:
                return 1
            return n * factorial(n - 1)  # fails with NameError
        print factorial(5)

Python 2.1 moved closer to static nested scoping by making visible the
names bound in all enclosing scopes (see PEP 227). This change makes the
above code example work as expected. However, because any assignment to
a name implicitly declares that name to be local, it is impossible to
rebind a name in an outer scope (except when a `global` declaration
forces the name to be global). Thus, the following code, intended to
display a number that can be incremented and decremented by clicking
buttons, doesn't work as someone familiar with lexical scoping might
expect::

    def make_scoreboard(frame, score=0):
        label = Label(frame)
        label.pack()
        for i in [-10, -1, 1, 10]:
            def increment(step=i):
                score = score + step  # fails with UnboundLocalError
                label['text'] = score
            button = Button(frame, text='%+d' % i, command=increment)
            button.pack()
        return label

Python syntax doesn't provide a way to indicate that the name `score`
mentioned in `increment` refers to the variable `score` bound in
`make_scoreboard`, not a local variable in `increment`. Users and
developers of Python have expressed an interest in removing this
limitation so that Python can have the full flexibility of the
Algol-style scoping model that is now standard in many programming
languages, including JavaScript, Perl, Ruby, Scheme, Smalltalk, C with
GNU extensions, and C\# 2.0.

It has been argued that such a feature isn't necessary, because a
rebindable outer variable can be simulated by wrapping it in a mutable
object::

    class Namespace:
        pass

    def make_scoreboard(frame, score=0):
        ns = Namespace()
        ns.score = 0
        label = Label(frame)
        label.pack()
        for i in [-10, -1, 1, 10]:
            def increment(step=i):
                ns.score = ns.score + step
                label['text'] = ns.score
            button = Button(frame, text='%+d' % i, command=increment)
            button.pack()
        return label

However, this workaround only highlights the shortcomings of existing
scopes: the purpose of a function is to encapsulate code in its own
namespace, so it seems unfortunate that the programmer should have to
create additional namespaces to make up for missing functionality in the
existing local scopes, and then have to decide whether each name should
reside in the real scope or the simulated scope.

Another common objection is that the desired functionality can be
written as a class instead, albeit somewhat more verbosely. One rebuttal
to this objection is that the existence of a different implementation
style is not a reason to leave a supported programming construct (nested
scopes) functionally incomplete. Python is sometimes called a
"multi-paradigm language" because it derives so much strength, practical
flexibility, and pedagogical power from its support and graceful
integration of multiple programming paradigms.

A proposal for scoping syntax appeared on Python-Dev as far back as 1994
\[1\]\_, long before PEP 227's support for nested scopes was adopted. At
the time, Guido's response was:

    This is dangerously close to introducing CSNS [classic static
    nested scopes]. *If* you were to do so, your proposed semantics
    of scoped seem allright. I still think there is not enough need
    for CSNS to warrant this kind of construct ...

After PEP 227, the "outer name rebinding discussion" has reappeared on
Python-Dev enough times that it has become a familiar event, having
recurred in its present form since at least 2003 \[2\]*. Although none
of the language changes proposed in these discussions have yet been
adopted, Guido has acknowledged that a language change is worth
considering \[12\]*.

Other Languages
===============

To provide some background, this section describes how some other
languages handle nested scopes and rebinding.

JavaScript, Perl, Scheme, Smalltalk, GNU C, C\# 2.0
---------------------------------------------------

These languages use variable declarations to indicate scope. In
JavaScript, a lexically scoped variable is declared with the `var`
keyword; undeclared variable names are assumed to be global. In Perl, a
lexically scoped variable is declared with the `my` keyword; undeclared
variable names are assumed to be global. In Scheme, all variables must
be declared (with `define` or `let`, or as formal parameters). In
Smalltalk, any block can begin by declaring a list of local variable
names between vertical bars. C and C\# require type declarations for all
variables. For all these cases, the variable belongs to the scope
containing the declaration.

Ruby (as of 1.8)
----------------

Ruby is an instructive example because it appears to be the only other
currently popular language that, like Python, tries to support
statically nested scopes without requiring variable declarations, and
thus has to come up with an unusual solution. Functions in Ruby can
contain other function definitions, and they can also contain code
blocks enclosed in curly braces. Blocks have access to outer variables,
but nested functions do not. Within a block, an assignment to a name
implies a declaration of a local variable only if it would not shadow a
name already bound in an outer scope; otherwise assignment is
interpreted as rebinding of the outer name. Ruby's scoping syntax and
rules have also been debated at great length, and changes seem likely in
Ruby 2.0 \[28\]\_.

Overview of Proposals
=====================

There have been many different proposals on Python-Dev for ways to
rebind names in outer scopes. They all fall into two categories: new
syntax in the scope where the name is bound, or new syntax in the scope
where the name is used.

New Syntax in the Binding (Outer) Scope
---------------------------------------

Scope Override Declaration ''''''''''''''''''''''''''

The proposals in this category all suggest a new kind of declaration
statement similar to JavaScript's `var`. A few possible keywords have
been proposed for this purpose:

-   `scope x` \[4\]\_
-   `var x` \[4\]\_ \[9\]\_
-   `my x` \[13\]\_

In all these proposals, a declaration such as `var x` in a particular
scope S would cause all references to `x` in scopes nested within S to
refer to the `x` bound in S.

The primary objection to this category of proposals is that the meaning
of a function definition would become context-sensitive. Moving a
function definition inside some other block could cause any of the local
name references in the function to become nonlocal, due to declarations
in the enclosing block. For blocks in Ruby 1.8, this is actually the
case; in the following example, the two setters have different effects
even though they look identical::

    setter1 = proc { | x | y = x }      # y is local here
    y = 13
    setter2 = proc { | x | y = x }      # y is nonlocal here
    setter1.call(99)
    puts y                              # prints 13
    setter2.call(77)
    puts y                              # prints 77

Note that although this proposal resembles declarations in JavaScript
and Perl, the effect on the language is different because in those
languages undeclared variables are global by default, whereas in Python
undeclared variables are local by default. Thus, moving a function
inside some other block in JavaScript or Perl can only reduce the scope
of a previously global name reference, whereas in Python with this
proposal, it could expand the scope of a previously local name
reference.

Required Variable Declaration '''''''''''''''''''''''''''''

A more radical proposal \[21\]\_ suggests removing Python's
scope-guessing convention altogether and requiring that all names be
declared in the scope where they are to be bound, much like Scheme. With
this proposal, `var x = 3` would both declare `x` to belong to the local
scope and bind it, where as `x = 3` would rebind the existing visible
`x`. In a context without an enclosing scope containing a `var x`
declaration, the statement `x = 3` would be statically determined to be
illegal.

This proposal yields a simple and consistent model, but it would be
incompatible with all existing Python code.

New Syntax in the Referring (Inner) Scope
-----------------------------------------

There are three kinds of proposals in this category.

Outer Reference Expression ''''''''''''''''''''''''''

This type of proposal suggests a new way of referring to a variable in
an outer scope when using the variable in an expression. One syntax that
has been suggested for this is `.x` \[7\]*, which would refer to `x`
without creating a local binding for it. A concern with this proposal is
that in many contexts `x` and `.x` could be used interchangeably, which
would confuse the reader. A closely related idea is to use multiple dots
to specify the number of scope levels to ascend \[8\]*, but most
consider this too error-prone \[17\]\_.

Rebinding Operator ''''''''''''''''''

This proposal suggests a new assignment-like operator that rebinds a
name without declaring the name to be local \[2\]\_. Whereas the
statement `x = 3` both declares `x` a local variable and binds it to 3,
the statement `x := 3` would change the existing binding of `x` without
declaring it local.

This is a simple solution, but according to PEP 3099 it has been
rejected (perhaps because it would be too easy to miss or to confuse
with `=`).

Scope Override Declaration ''''''''''''''''''''''''''

The proposals in this category suggest a new kind of declaration
statement in the inner scope that prevents a name from becoming local.
This statement would be similar in nature to the `global` statement, but
instead of making the name refer to a binding in the top module-level
scope, it would make the name refer to the binding in the nearest
enclosing scope.

This approach is attractive due to its parallel with a familiar Python
construct, and because it retains context-independence for function
definitions.

This approach also has advantages from a security and debugging
perspective. The resulting Python would not only match the functionality
of other nested-scope languages but would do so with a syntax that is
arguably even better for defensive programming. In most other languages,
a declaration contracts the scope of an existing name, so inadvertently
omitting the declaration could yield farther-reaching (i.e. more
dangerous) effects than expected. In Python with this proposal, the
extra effort of adding the declaration is aligned with the increased
risk of non-local effects (i.e. the path of least resistance is the
safer path).

Many spellings have been suggested for such a declaration:

-   `scoped x` \[1\]\_
-   `global x in f` \[3\]\_ (explicitly specify which scope)
-   `free x` \[5\]\_
-   `outer x` \[6\]\_
-   `use x` \[9\]\_
-   `global x` \[10\]\_ (change the meaning of `global`)
-   `nonlocal x` \[11\]\_
-   `global x outer` \[18\]\_
-   `global in x` \[18\]\_
-   `not global x` \[18\]\_
-   `extern x` \[20\]\_
-   `ref x` \[22\]\_
-   `refer x` \[22\]\_
-   `share x` \[22\]\_
-   `sharing x` \[22\]\_
-   `common x` \[22\]\_
-   `using x` \[22\]\_
-   `borrow x` \[22\]\_
-   `reuse x` \[23\]\_
-   `scope f x` \[25\]\_ (explicitly specify which scope)

The most commonly discussed choices appear to be `outer`, `global`, and
`nonlocal`. `outer` is already used as both a variable name and an
attribute name in the standard library. The word `global` has a
conflicting meaning, because "global variable" is generally understood
to mean a variable with top-level scope \[27\]\_. In C, the keyword
`extern` means that a name refers to a variable in a different
compilation unit. While `nonlocal` is a bit long and less
pleasant-sounding than some of the other options, it does have precisely
the correct meaning: it declares a name not local.

Proposed Solution
=================

The solution proposed by this PEP is to add a scope override declaration
in the referring (inner) scope. Guido has expressed a preference for
this category of solution on Python-Dev \[14\]\_ and has shown approval
for `nonlocal` as the keyword \[19\]\_.

The proposed declaration::

    nonlocal x

prevents `x` from becoming a local name in the current scope. All
occurrences of `x` in the current scope will refer to the `x` bound in
an outer enclosing scope. As with `global`, multiple names are
permitted::

    nonlocal x, y, z

If there is no pre-existing binding in an enclosing scope, the compiler
raises a SyntaxError. (It may be a bit of a stretch to call this a
syntax error, but so far SyntaxError is used for all compile-time
errors, including, for example, **future** import with an unknown
feature name.) Guido has said that this kind of declaration in the
absence of an outer binding should be considered an error \[16\]\_.

If a `nonlocal` declaration collides with the name of a formal parameter
in the local scope, the compiler raises a SyntaxError.

A shorthand form is also permitted, in which `nonlocal` is prepended to
an assignment or augmented assignment::

    nonlocal x = 3

The above has exactly the same meaning as `nonlocal x; x = 3`. (Guido
supports a similar form of the `global` statement \[24\]\_.)

On the left side of the shorthand form, only identifiers are allowed,
not target expressions like `x[0]`. Otherwise, all forms of assignment
are allowed. The proposed grammar of the `nonlocal` statement is::

    nonlocal_stmt ::=
        "nonlocal" identifier ("," identifier)*
                   ["=" (target_list "=")+ expression_list]
      | "nonlocal" identifier augop expression_list

The rationale for allowing all these forms of assignment is that it
simplifies understanding of the `nonlocal` statement. Separating the
shorthand form into a declaration and an assignment is sufficient to
understand what it means and whether it is valid.

.. note::

The shorthand syntax was not added in the original implementation of the
PEP. Later discussions \[29\]\_ \[30\]\_ concluded this syntax should
not be implemented.

Backward Compatibility
======================

This PEP targets Python 3000, as suggested by Guido \[19\]*. However,
others have noted that some options considered in this PEP may be small
enough changes to be feasible in Python 2.x \[26\]*, in which case this
PEP could possibly be moved to be a 2.x series PEP.

As a (very rough) measure of the impact of introducing a new keyword,
here is the number of times that some of the proposed keywords appear as
identifiers in the standard library, according to a scan of the Python
SVN repository on November 5, 2006::

    nonlocal    0
    use         2
    using       3
    reuse       4
    free        8
    outer     147

`global` appears 214 times as an existing keyword. As a measure of the
impact of using `global` as the outer-scope keyword, there are 18 files
in the standard library that would break as a result of such a change
(because a function declares a variable `global` before that variable
has been introduced in the global scope)::

    cgi.py
    dummy_thread.py
    mhlib.py
    mimetypes.py
    idlelib/PyShell.py
    idlelib/run.py
    msilib/__init__.py
    test/inspect_fodder.py
    test/test_compiler.py
    test/test_decimal.py
    test/test_descr.py
    test/test_dummy_threading.py
    test/test_fileinput.py
    test/test_global.py (not counted: this tests the keyword itself)
    test/test_grammar.py (not counted: this tests the keyword itself)
    test/test_itertools.py
    test/test_multifile.py
    test/test_scope.py (not counted: this tests the keyword itself)
    test/test_threaded_import.py
    test/test_threadsignals.py
    test/test_warnings.py

References
==========

.. \[1\] Scoping (was Re: Lambda binding solved?) (Rafael Bracho)
http://www.python.org/search/hypermail/python-1994q1/0301.html

.. \[2\] Extended Function syntax (Just van Rossum)
https://mail.python.org/pipermail/python-dev/2003-February/032764.html

.. \[3\] Closure semantics (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2003-October/039214.html

.. \[4\] Better Control of Nested Lexical Scopes (Almann T. Goo)
https://mail.python.org/pipermail/python-dev/2006-February/061568.html

.. \[5\] PEP for Better Control of Nested Lexical Scopes (Jeremy Hylton)
https://mail.python.org/pipermail/python-dev/2006-February/061602.html

.. \[6\] PEP for Better Control of Nested Lexical Scopes (Almann T. Goo)
https://mail.python.org/pipermail/python-dev/2006-February/061603.html

.. \[7\] Using and binding relative names (Phillip J. Eby)
https://mail.python.org/pipermail/python-dev/2006-February/061636.html

.. \[8\] Using and binding relative names (Steven Bethard)
https://mail.python.org/pipermail/python-dev/2006-February/061749.html

.. \[9\] Lexical scoping in Python 3k (Ka-Ping Yee)
https://mail.python.org/pipermail/python-dev/2006-July/066862.html

.. \[10\] Lexical scoping in Python 3k (Greg Ewing)
https://mail.python.org/pipermail/python-dev/2006-July/066889.html

.. \[11\] Lexical scoping in Python 3k (Ka-Ping Yee)
https://mail.python.org/pipermail/python-dev/2006-July/066942.html

.. \[12\] Lexical scoping in Python 3k (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066950.html

.. \[13\] Explicit Lexical Scoping (pre-PEP?) (Talin)
https://mail.python.org/pipermail/python-dev/2006-July/066978.html

.. \[14\] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066991.html

.. \[15\] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066995.html

.. \[16\] Lexical scoping in Python 3k (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066968.html

.. \[17\] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/067004.html

.. \[18\] Explicit Lexical Scoping (pre-PEP?) (Andrew Clover)
https://mail.python.org/pipermail/python-dev/2006-July/067007.html

.. \[19\] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/067067.html

.. \[20\] Explicit Lexical Scoping (pre-PEP?) (Matthew Barnes)
https://mail.python.org/pipermail/python-dev/2006-July/067221.html

.. \[21\] Sky pie: a "var" keyword (a thread started by Neil Toronto)
https://mail.python.org/pipermail/python-3000/2006-October/003968.html

.. \[22\] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

.. \[23\] Alternatives to 'outer' (Jim Jewett)
https://mail.python.org/pipermail/python-3000/2006-November/004153.html

.. \[24\] Draft PEP for outer scopes (Guido van Rossum)
https://mail.python.org/pipermail/python-3000/2006-November/004166.html

.. \[25\] Draft PEP for outer scopes (Talin)
https://mail.python.org/pipermail/python-3000/2006-November/004190.html

.. \[26\] Draft PEP for outer scopes (Nick Coghlan)
https://mail.python.org/pipermail/python-3000/2006-November/004237.html

.. \[27\] Global variable (version 2006-11-01T01:23:16)
http://en.wikipedia.org/wiki/Global\_variable

.. \[28\] Ruby 2.0 block local variable
http://redhanded.hobix.com/inspect/ruby20BlockLocalVariable.html

.. \[29\] Issue 4199: combining assignment with global & nonlocal (Guido
van Rossum)
https://mail.python.org/pipermail/python-dev/2013-June/127142.html

.. \[30\] Whatever happened to 'nonlocal x = y'? (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2018-January/151627.html

Acknowledgements
================

The ideas and proposals mentioned in this PEP are gleaned from countless
Python-Dev postings. Thanks to Jim Jewett, Mike Orr, Jason Orendorff,
and Christian Tanzer for suggesting specific edits to this PEP.

Copyright
=========

This document has been placed in the public domain.

 .. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
