PEP: 306 Title: How to Change Python's Grammar Version: $Revision$
Last-Modified: $Date$ Author: Michael Hudson <mwh@python.net>, Jack
Diederich <jackdied@gmail.com>, Nick Coghlan <ncoghlan@gmail.com>,
Benjamin Peterson <benjamin@python.org> Status: Withdrawn Type:
Informational Content-Type: text/x-rst Created: 29-Jan-2003
Post-History: 30-Jan-2003

Note
====

This PEP has been moved to the Python dev guide \[1\]\_.

Abstract
========

There's more to changing Python's grammar than editing `Grammar/Grammar`
and `Python/compile.c`. This PEP aims to be a checklist of places that
must also be fixed.

It is probably incomplete. If you see omissions, just add them if you
can -- you are not going to offend the author's sense of ownership.
Otherwise submit a bug or patch and assign it to mwh.

This PEP is not intended to be an instruction manual on Python grammar
hacking, for several reasons.

Rationale
=========

People are getting this wrong all the time; it took well over a year
before someone noticed \[2\]\_ that adding the floor division operator
(`//`) broke the `parser` module.

Checklist
=========

-   `Grammar/Grammar`: OK, you'd probably worked this one out :)

-   `Parser/Python.asdl` may need changes to match the `Grammar`. Run
    `make` to regenerate `Include/Python-ast.h` and
    `Python/Python-ast.c`.

-   `Python/ast.c` will need changes to create the AST objects involved
    with the `Grammar` change. `Lib/compiler/ast.py` will need matching
    changes to the pure-python AST objects.

-   `Parser/pgen` needs to be rerun to regenerate `Include/graminit.h`
    and `Python/graminit.c`. (make should handle this for you.)

-   `Python/symbtable.c`: This handles the symbol collection pass that
    happens immediately before the compilation pass.

-   `Python/compile.c`: You will need to create or modify the
    `compiler_*` functions to generate opcodes for your productions.

-   You may need to regenerate `Lib/symbol.py` and/or `Lib/token.py`
    and/or `Lib/keyword.py`.

-   The `parser` module. Add some of your new syntax to `test_parser`,
    bang on `Modules/parsermodule.c` until it passes.

-   Add some usage of your new syntax to `test_grammar.py`.

-   The `compiler` package. A good test is to compile the standard
    library and test suite with the `compiler` package and then check it
    runs. Note that this only needs to be done in Python 2.x.

-   If you've gone so far as to change the token structure of Python,
    then the `Lib/tokenizer.py` library module will need to be changed.

-   Certain changes may require tweaks to the library module `pyclbr`.

-   Documentation must be written!

-   After everything's been checked in, you're likely to see a new
    change to `Python/Python-ast.c`. This is because this (generated)
    file contains the SVN version of the source from which it was
    generated. There's no way to avoid this; you just have to submit
    this file separately.

References
==========

.. \[1\] CPython Developer's Guide: Changing CPython's Grammar
https://devguide.python.org/grammar/

.. \[2\] SF Bug \#676521, parser module validation failure
https://bugs.python.org/issue676521

Copyright
=========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 End:
