Developer Documentation¶
Setting up an Environment for pyMOR Development¶
Getting the Source¶
Clone the pyMOR git repository using:
git clone git@github.com:pymor/pymor.git
cd pymor
and, optionally, switch to the branch you are interested in, e.g.:
git checkout 2022.2.x
Environment with venv¶
Create and activate a new venv (alternatively, you can use
virtualenv
and
virtualenvwrapper
):
python3 -m venv venv
source venv/bin/activate
Then, it may be necessary to upgrade pip
:
pip install -U pip
Finally, make an editable installation of pyMOR with minimal dependencies:
pip install -e .
or, to install all optional dependencies and development tools:
pip install -e '.[full-compiled,dev]'
Note that the full-compiled
extra will install mpi4py
and slycot
, which will require C and
Fortran compilers as well as MPI and OpenBLAS headers.
Alternatively, use the full
extra to avoid building these additional packages.
Environment with CI images¶
The docker images used in pyMOR’s CI pipeline can be pulled and executed using the
ci_<current|oldest|fenics>_image_<pull|run>
make targets.
E.g., to run the ‘current’ CI image use
make ci_current_image_run
This will automatically mount the pyMOR source tree into /src
. Note that pyMOR itself is not
installed in the image, so you still have to install it using pip install -e .
.
To launch a notebook server and bind it to port 8888, execute
make ci_current_image_run_notebook
(not available for the fenics
image).
Coding Guidelines and Project Management¶
Python Code Style¶
pyMOR follows the coding style of PEP8 apart from a few
exceptions.
Configurations for the ruff code checker are contained in
pyproject.toml
. To check your code using ruff
, execute
ruff .
at the root of pyMOR’s source tree.
Further guidelines:
Functions and classes called or instantiated by users should be sufficiently well documented.
Use keyword arguments for parameters with defaults. This will make your code less likely to break, when the called function is extended.
Generally use verbose identifiers instead of single letter names, also for mathematical objects (use
residual
instead ofr
). Exceptions are well-established variable names (like A,B,C,D,E for LTI systems) or temporary variables.Prefer assertions over exceptions in potentially performance-relevant code. Assertions can be ignored by invoking Python with the
-O
argument. Try to check a single condition in an assertion and add helpful error messages.Use
warnings.warn
for code-related issues. Useself.logger.warning
for issues related to an algorithm or user input.It is generally ok to use builtin names as function parameters (e.g.
type
) when there is no other adequate name. There is no need to add underscores before or after the name.Use the
self.__auto_init(locals())
idiom to initialize instance attributes from__init__
args of the same name.
Note
If you are a first-time contributor, do not worry too much about code style. The main developers will be happy to help you to bring your code into proper shape for inclusion in pyMOR.
Markdown Style¶
The Markdown style is determined by the
markdownlint
rules,
specified in .markdownlint.yml
.
The markdownlint-cli2
tool
(or the pre-commit
hook; see below)
can be used to check for errors.
GitHub Project¶
All new code enters pyMOR by means of a pull request. Pull requests (PR) trigger automatic tests
which are mandatory to pass before the PR can be merged. A PR must also be tagged with either one of the
pr:new-feature
, pr:fix
, pr:removal
, pr:deprecation
or pr:change
labels to clearly identify the type of
changes it introduces. See the labels’ descriptions for more details.
pre-commit
Hooks¶
pyMOR ships a config for the pre-commit hook management system.
Using this setup can be a good way to find code style errors before pushing to GitHub, but using
it is not required. Once you have pre-commit
installed in you environment, run
pre-commit install
Afterwards, the hooks configured in .pre-commit-config.yaml
will run on all changed
files prior to committing changes. Errors will block the commit, some
checks will automatically fix the file.
Updating pyMOR’s Dependencies¶
All required or optional dependencies of pyMOR are specified in pyproject.toml
.
We use pip-compile, to generate requirements-ci-*.txt
files from these specifications, which contain pinned versions of all packaged installed into the
respective GitLab CI images.
The extras included into the images are specified in Makefile
.
For the oldest
CI image, requirements-ci-oldest-pins.in
is used in addition, which pins some of
pyMOR’s core dependencies to the oldest version supported by pyMOR.
Similarly to the pip-compile
workflow, we use conda-lock
to create conda-forge environment lock files that are used for the
GitHub actions CI builds.
If you update pyMOR’s dependencies, make sure to execute
make ci_requirements
and commit the changes made to the lock files to ensure that the updated dependencies are picked up by CI.
Note that make ci_requirements
requires docker or a compatible
container runtime such as podman.
The pyMOR main developers will be happy to take care of this step for you.
Testing / Continuous Integration Setup¶
pyMOR’s Test Suite¶
pyMOR uses pytest for unit testing.
All tests are contained within the src/pymortests
directory and can be run
by invoking the pytest
executable.
Please refer to the pytest documentation
for detailed examples.
To run the entire test suite, it is also possible to execute
make test
which will use the Xvfb virtual framebuffer X server, to prevent GUI windows from popping up during the test run.
pyMOR uses the hypothesis for property-based testing.
To select the amount of test samples to create, you can specify the hypothesis profile to use by
setting the PYMOR_HYPOTHESIS_PROFILE.
Available profiles are dev
(default, shortest execution time), ci
and ci_large
(longest
execution time).
The profiles are defined in conftest.py
.
To disable tests for some of pyMOR’s optional dependencies, set PYMOR_CONFIG_DISABLE
to a
whitespace-separated list of config
items which should be prevented
from being loaded during the test session.
Conversely, to disable all tests that only use pyMOR’s builtin discretization toolkit, set
PYMOR_FIXTURES_DISABLE_BUILTIN
to 1
and pass -m 'not builtin'
to the pytest
command line.
GitLab CI¶
We use GitLab deployed at https://zivgitlab.uni-muenster.de/pymor/pymor as our main CI infrastructure. All CI stages are run in docker containers that are built and pushed to the GitLab container registry using
make ci_preflight_image # used for preflight/docker images stages
make ci_preflight_image_push
make ci_images # used for actual tests
make ci_images_push
The corresponding Dockerfiles
are all contained in the docker
directory.
The images are tagged with a sha256sum
of the corresponding requirements-ci-*.txt
file.
Images corresponding to the current state of main
are tagged with main
.
Images corresponding to tagged commits are tagged with the respective git tag
(see Stage: deploy).
Stage: preflight¶
The main responsible of the preflight
job is to compute the sha256sum
of all
requirements-ci-*.txt
files in order to infer the tags of the CI images used for testing.
It also queries GitLab’s container registry to check if the needed images are available.
The results are saved in a
dotenv
file to make them available as environment variables in later CI stages.
Stage: docker images¶
When need, updated CI images are built using a podman-in-docker setup. The resulting images are then uploaded to the container registry.
Stage: test/build¶
This stage executes the test_*.bash
scripts located in ./.ci/gitlab/
.
For each supported external PDE solver backend, an individual CI job is run.
Documentation is also built in this stage.
Stage: deploy¶
- docs
Commits documentation built in Stage: test/buildto the documentation repository. This repository is the source for https://docs.pymor.org/ served via GitHub Pages. A binder setup for the generated tutorials notebooks is added on a branch with a name matching the currently checked out git branch of pyMOR.
- submit coverage
This job accumulates all the coverage databases generated by previous stages and submits that to codecov.io.
- coverage html
Generates an html coverage report, which can be downloaded as a job artifact.
- tag docker images
If the running pipeline corresponds to a push on
main
or a tagged commit (release), the used CI images are tagged with eithermain
or the corresponding git tag in GitLab’s container registry. This will prevent these images from being removed during registry cleanup. In particular, this guarantees that the images are available for the binder setups linked in the online documentation.
GitHub Actions¶
We use GitHub Actions to run some additional checks. This includes Conda-based MacOS and Windows test suite runs and management of GitHub labels. Further, for PRs from external repositories, a local mirror branch is created and updated. This enables GitLab CI runs those PRs.
pre-commit.ci¶
The main pyMOR repository has the pre-commit.ci
GitHub App installed. This runs the pre-commit hooks
defined in .pre-commit-config.yaml
on every pull request.
If the hooks change files, the changes are pushed back to the PR branch.
Configuration is done
via the .pre-commit-config.yaml
file.