PEP: 448 Title: Additional Unpacking Generalizations Version: $Revision$
Last-Modified: $Date$ Author: Joshua Landau <joshua@landau.ws>
Discussions-To: python-ideas\@python.org Status: Final Type: Standards
Track Content-Type: text/x-rst Created: 29-Jun-2013 Python-Version: 3.5
Post-History:

Abstract
========

This PEP proposes extended usages of the `*` iterable unpacking operator
and `**` dictionary unpacking operators to allow unpacking in more
positions, an arbitrary number of times, and in additional
circumstances. Specifically, in function calls, in comprehensions and
generator expressions, and in displays.

Function calls are proposed to support an arbitrary number of unpackings
rather than just one::

    >>> print(*[1], *[2], 3)
    1 2 3
    >>> dict(**{'x': 1}, y=2, **{'z': 3})
    {'x': 1, 'y': 2, 'z': 3}

Unpacking is proposed to be allowed inside tuple, list, set, and
dictionary displays::

    >>> *range(4), 4
    (0, 1, 2, 3, 4)
    >>> [*range(4), 4]
    [0, 1, 2, 3, 4]
    >>> {*range(4), 4}
    {0, 1, 2, 3, 4}
    >>> {'x': 1, **{'y': 2}}
    {'x': 1, 'y': 2}

In dictionaries, later values will always override earlier ones::

    >>> {'x': 1, **{'x': 2}}
    {'x': 2}

    >>> {**{'x': 2}, 'x': 1}
    {'x': 1}

This PEP does not include unpacking operators inside list, set and
dictionary comprehensions although this has not been ruled out for
future proposals.

Rationale
=========

Current usage of the `*` iterable unpacking operator features
unnecessary restrictions that can harm readability.

Unpacking multiple times has an obvious rationale. When you want to
unpack several iterables into a function definition or follow an unpack
with more positional arguments, the most natural way would be to write::

    function(**kw_arguments, **more_arguments)

    function(*arguments, argument)

Simple examples where this is useful are `print` and `str.format`.
Instead, you could be forced to write::

    kwargs = dict(kw_arguments)
    kwargs.update(more_arguments)
    function(**kwargs)

    args = list(arguments)
    args.append(arg)
    function(*args)

or, if you know to do so::

    from collections import ChainMap
    function(**ChainMap(more_arguments, arguments))

    from itertools import chain
    function(*chain(args, [arg]))

which add unnecessary line-noise and, with the first methods, causes
duplication of work.

There are two primary rationales for unpacking inside of containers.
Firstly there is a symmetry of assignment, where
`fst, *other, lst = elems` and `elems = fst, *other, lst` are
approximate inverses, ignoring the specifics of types. This, in effect,
simplifies the language by removing special cases.

Secondly, it vastly simplifies types of "addition" such as combining
dictionaries, and does so in an unambiguous and well-defined way::

    combination = {**first_dictionary, "x": 1, "y": 2}

instead of::

    combination = first_dictionary.copy()
    combination.update({"x": 1, "y": 2})

which is especially important in contexts where expressions are
preferred. This is also useful as a more readable way of summing
iterables into a list, such as
`my_list + list(my_tuple) + list(my_range)` which is now equivalent to
just `[*my_list, *my_tuple, *my_range]`.

Specification
=============

Function calls may accept an unbounded number of `*` and `**`
unpackings. There will be no restriction of the order of positional
arguments with relation to `*` unpackings nor any restriction of the
order of keyword arguments with relation to `**` unpackings.

Function calls continue to have the restriction that keyword arguments
must follow positional arguments and `**` unpackings must additionally
follow `*` unpackings.

Currently, if an argument is given multiple times --- such as a
positional argument given both positionally and by keyword --- a
`TypeError` is raised. This remains true for duplicate arguments
provided through multiple `**` unpackings,
e.g. `f(**{'x': 2}, **{'x': 3})`, except that the error will be detected
at runtime.

A function looks like this::

    function(
        argument or *args, argument or *args, ...,
        kwargument or *args, kwargument or *args, ...,
        kwargument or **kwargs, kwargument or **kwargs, ...
    )

Tuples, lists, sets and dictionaries will allow unpacking. This will act
as if the elements from unpacked items were inserted in order at the
site of unpacking, much as happens in unpacking in a function-call.
Dictionaries require `**` unpacking; all the others require `*`
unpacking.

The keys in a dictionary remain in a right-to-left priority order, so
`{**{'a': 1}, 'a': 2, **{'a': 3}}` evaluates to `{'a': 3}`. There is no
restriction on the number or position of unpackings.

Disadvantages
=============

The allowable orders for arguments in a function call are more
complicated than before. The simplest explanation for the rules may be
"positional arguments precede keyword arguments and `**` unpacking; `*`
unpacking precedes `**` unpacking".

Whilst `*elements, = iterable` causes `elements` to be a list,
`elements = *iterable,` causes `elements` to be a tuple. The reason for
this may confuse people unfamiliar with the construct.

Concerns have been raised about the unexpected difference between
duplicate keys in dictionaries being allowed but duplicate keys in
function call syntax raising an error. Although this is already the case
with current syntax, this proposal might exacerbate the issue. It
remains to be seen how much of an issue this is in practice.

Variations
==========

The PEP originally considered whether the ordering of argument types in
a function call (positional, keyword, `*` or `**`) could become less
strict. This met little support so the idea was shelved.

Earlier iterations of this PEP allowed unpacking operators inside list,
set, and dictionary comprehensions as a flattening operator over
iterables of containers::

    >>> ranges = [range(i) for i in range(5)]
    >>> [*item for item in ranges]
    [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]

    >>> {*item for item in ranges}
    {0, 1, 2, 3}

This was met with a mix of strong concerns about readability and mild
support. In order not to disadvantage the less controversial aspects of
the PEP, this was not accepted with the rest of the proposal.

Unbracketed comprehensions in function calls, such as
`f(x for x in it)`, are already valid. These could be extended to::

    f(*x for x in it) == f((*x for x in it))
    f(**x for x in it) == f({**x for x in it})

However, it wasn't clear if this was the best behaviour or if it should
unpack into the arguments of the call to `f`. Since this is likely to be
confusing and is of only very marginal utility, it is not included in
this PEP. Instead, these will throw a `SyntaxError` and comprehensions
with explicit brackets should be used instead.

Approval
========

This PEP was accepted by Guido on February 25, 2015 \[1\]\_.

Implementation
==============

An implementation for Python 3.5 is found at Issue 2292 on bug tracker
\[2\]\_. This currently includes support for unpacking inside
comprehensions, which should be removed.

References
==========

.. \[1\] PEP accepted, "PEP 448 review", Guido van Rossum
(https://mail.python.org/pipermail/python-dev/2015-February/138564.html)

.. \[2\] Issue 2292, "Missing `*`-unpacking generalizations", Thomas
Wouters (http://bugs.python.org/issue2292)

.. \[3\] Discussion on Python-ideas list, "list / array comprehensions
extension", Alexander Heger
(https://mail.python.org/pipermail/python-ideas/2011-December/013097.html)

Copyright
=========

This document has been placed in the public domain.

 .. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
