Contributing

Contributing to RestrictedPython

Preperations for Contributing

If you want to contribute to this package, please prepare a development environment that includes git, tox, and several Python versions available through a Python manager such as pyenv. Please read the section Understanding How RestrictedPython works internally first.

For all commits, use tox to run tests and lint, and build the docs, before pushing your commit to the remote repository.

Preperations for a new Python version

RestrictedPython should be updated for each new version of Python. To do so:

  • Read the changelog (What’s new in Python).

  • Copy and adjust the new AST Grammar (found under: Python 3 AST) to /docs/contributing/ast/python<version>.ast.

  • Add a new file changes_from<old_version>to<new_version>.rst in the directory /docs/contributing/. If the changes are significant, especially if related to security, then add a description of the changes in that file.

  • Add those files to the toctree directive in index.rst.

  • For each new AST Node or functionality:

    • Add tests to /tests/.

    • Add a visit_<AST Node> to /src/RestrictedPython/transformer.py.

      If the new AST Node should be enabled by default, with or without any modification, please add a visit_<AST Node> method such as the following:

      def visit_<AST Node>(self, node):
    """Allow `<AST Node>` expressions."""
    ...  # modifications
    return self.node_contents_visit(node)

      All AST Nodes without an explicit visit_<AST Node> method, are denied by default. So the usage of this expression and functionality is not allowed.

  • Add a corresponding changelog entry.

  • Additionally modify .meta.toml and run the meta/config script (for details see: https://github.com/mgedmin/check-python-versions) to update the following files:

    • /setup.py - Check that the new Python version classifier has been added "Programming Language :: Python :: <version>",, and that the python_requires section has been updated correctly.
    • /tox.ini - Check that a testenv entry is added to the general envlist statement.
    • /.github/workflows/tests.yml - Check that a corresponding Python version entry has been added to the matrix definition.
    • /docs/conf.py - Add the Python version to the intersphinx_mapping list.

  • On your local environment, use tox to run tests and lint, and build the docs, before pushing your commit to the remote repository.

  • Create a pull request.

Enable a Python Feature in RestrictedPython

To enable a certain functionality in RestrictedPython, do the following:

  • Create a new issue on GitHub, requesting the new feature, for discussion.
  • In /src/RestrictedPython/transformer.py, change the corresponding visit_<AST Node> method.
  • In /tests/, add or change the corresponding tests for this functionality.
  • Add a changelog entry.
  • On your local environment, use tox to run tests and lint, and build the docs, before pushing your commit to the remote repository.
  • Create a pull request and request a review by a core maintainer, e.g.:
    • icemac
    • loechel

Differences between different Python versions

A (modified style) Copy of all Abstract Grammar Definitions for the Python versions does live in this Documentation (ast Subfolder) to help finding difference quicker by comparing files.

Understanding How RestrictedPython works internally

RestrictedPython is a classic approach of compiler construction to create a limited subset of an existing programming language.

Defining a programming language requires a regular grammar (Chomsky 3 / EBNF) definition. This grammar will be implemented in an abstract syntax tree (AST), which will be passed on to a code generator to produce a machine-readable version.

Code generation

As Python is a platform independent programming language, this machine readable version is a byte code which will be translated on the fly by an interpreter into machine code. This machine code then gets executed on the specific CPU architecture, with the standard operating system restrictions.

The byte code produced must be compatible with the execution environment that the Python interpreter is running in, so we do not generate the byte code directly from compile_restricted and the other compile_restricted_* methods manually, it may not match what the interpreter expects.

Thankfully, the Python compile() function introduced the capability to compile ast.AST code into byte code in Python 2.6, so we can return the platform-independent AST and keep byte code generation delegated to the interpreter.

ast module (Abstract Syntax Trees)

The ast module consists of four areas:

  • AST (Basis of all Nodes) + all node class implementations
  • NodeVisitor and NodeTransformer (tool to consume and modify the AST)
  • Helper methods
    • parse
    • walk
    • dump
  • Constants
    • PyCF_ONLY_AST

NodeVisitor & NodeTransformer

A NodeVisitor is a class of a node / AST consumer, it reads the data by stepping through the tree without modifying it. In contrast, a NodeTransformer (which inherits from a NodeVisitor) is allowed to modify the tree and nodes.

Technical decissions on how to implement / maintain RestrictedPython (Design, Structure, Tools, …)

RestrictedPython is a core Package of the Zope & Plone Stack. Until Version 3.6 RestrictedPython was Python 2 only, and a critical blocker for Zope & Plone. With RestrictedPython 4.0 an API compatible rewrite has happend, which supports modern Python Versions.

  • Use modern python tool stack for maintainance and tests

    • tox
    • pytest
    • black
    • linting tools: flake8

  • Use clear package Structure

    • /src - Source Code
    • /tests - separated tests
    • /docs - Documentation

    Tests and documentation are distributed within released packages.

Todo

Resolve discussion about how RestrictedPython should be treat new expressions / ast.Nodes. This belongs to Preperations for a new Python version.

Option 1 - reduce maintainance burden (prefered by icemac)

All AST Nodes without an explicit visit_<AST Node> method, are denied by default. So the usage of this expression and functionality is not allowed.

This is currently the promoted version.

Option 2 - be as explicite as possible (prefered by loechel)

If the new AST Node should be disabled by default, add a visit_<AST Node> method such as the following:

def visit_<AST Node>(self, node):
    """`<AST Node>` expression currently not allowed."""
    self.not_allowed(node)

Please note, that for all AST Nodes without an explicit visit_<AST Node> method, a default applies which denies the usage of this expression and functionality. As we try to be as explicit as possible, all language features should have a corresponding visit_<AST Node>.

That follows the Zen of Python:

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Todos

Todo

Resolve discussion about how RestrictedPython should be treat new expressions / ast.Nodes. This belongs to Preperations for a new Python version.

Option 1 - reduce maintainance burden (prefered by icemac)

All AST Nodes without an explicit visit_<AST Node> method, are denied by default. So the usage of this expression and functionality is not allowed.

This is currently the promoted version.

Option 2 - be as explicite as possible (prefered by loechel)

If the new AST Node should be disabled by default, add a visit_<AST Node> method such as the following:

def visit_<AST Node>(self, node):
    """`<AST Node>` expression currently not allowed."""
    self.not_allowed(node)

Please note, that for all AST Nodes without an explicit visit_<AST Node> method, a default applies which denies the usage of this expression and functionality. As we try to be as explicit as possible, all language features should have a corresponding visit_<AST Node>.

That follows the Zen of Python:

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/restrictedpython/checkouts/latest/docs/contributing/index.rst, line 176.)

Todo

Complete documentation of all public API elements with docstyle comments https://www.python.org/dev/peps/pep-0257/ http://www.sphinx-doc.org/en/stable/ext/autodoc.html http://thomas-cokelaer.info/tutorials/sphinx/docstring_python.html

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/restrictedpython/checkouts/latest/docs/roadmap/index.rst, line 4.)

Todo

Resolve Discussion in https://github.com/zopefoundation/RestrictedPython/pull/39#issuecomment-283074699

compile_restricted optional params flags and dont_inherit will not work as expected with the current implementation.

stephan-hof did propose a solution, should be discussed and if approved implemented.

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/restrictedpython/checkouts/latest/docs/roadmap/index.rst, line 21.)

Todo

Describe Guards and predefined guard methods in details

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/restrictedpython/checkouts/latest/docs/usage/policy.rst, line 28.)

Todo

Describe full_write_guard more in detail and how it works.

(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/restrictedpython/checkouts/latest/docs/usage/policy.rst, line 42.)