Development and release processes
=================================

Definitions
-----------

* *PyGranta meta-package*: A package which depends on the *idiomatic* packages and *auto-generated packages*. Includes
  documentation describing the PyGranta ecosystem, including this document.
* *Idiomatic package*: A hand-written package. Typically depends on one *auto-generated package* and ``openapi-common``.
  There are currently four *idiomatic packages* in PyGranta:

  * ``ansys-grantami-bomanalytics``
  * ``ansys-grantami-recordlists``
  * ``ansys-grantami-jobqueue``
  * ``ansys-grantami-system``

* *Auto-generated package*: A package generated by the *codegen template*. There are currently two *auto-generated*
  packages in PyGranta:

  * ``ansys-grantami-bomanalytics-openapi``
  * ``ansys-grantami-serverapi-openapi``

* *Codegen template*: The template used to generate the *auto-generated packages*.
* *Production test VM*: A virtual machine used as part of CI. A new *Production test VM* is commissioned for each new
  Granta MI release, and the VM is only upgraded if a service pack is made available for that release. It is used as
  part of *idiomatic package* CI during development and release to ensure compatibility with a specific version of
  Granta MI.
* *Development test VM*: Virtual machine used as part of CI. This machine is upgraded as required during development.
  It is used as part of *idiomatic package* CI during development **only** to ensure compatibility with development
  builds of Granta MI.

Development and release timeline
--------------------------------

The timeline is split into three distinct phases:

1. Phase 1: Development. There is active development upstream on closed-source Granta MI components, the *codegen
   template*, and on the open source *idiomatic packages* and *auto-generated packages*. Effort is made to ensure CI is
   passing on all ``main`` branches, but from time to time CI may fail. The *development test VM* is upgraded as
   required to support CI passing on ``main`` branches.
2. Phase 2: Hardening. Bugs are fixed and minor improvements are made to *idiomatic packages*, *auto-generated
   packages*, and the *codegen template*. During this phase the *codegen template* and *auto-generated packages* are
   finalized.
3. Phase 3: Pre-release. The upstream closed-source development has completed. The *PyGranta meta-package* and
   *idiomatic packages* are manually tested and finalized.

Phase 1: Development
~~~~~~~~~~~~~~~~~~~~

During development, the following processes happen continuously:

1. Incorporate changes to upstream API definitions into *auto-generated package* repositories.
2. Check that code generation is successful by monitoring CI in the *auto-generated package* repositories.
3. If code generation is not successful, resolve by either fixing the *codegen template*, fixing the *auto-generated
   package* configuration, or raise the issue with the internal Ansys development team.

If *idiomatic package* development does not depend on upstream API development, then there is no dependency between
the *idiomatic package* and the *auto-generated package* and work can proceed in parallel.

If significant *idiomatic package* development is being done that depends on the current state of an *auto-generated
package*, you should update the ``main`` branch of the corresponding *idiomatic package* to depend on the
*auto-generated package* ``main`` directly.

During this phase, the *development test VM* may be upgraded if required to support new upstream Granta MI capabilities.

Phase 2: Hardening
~~~~~~~~~~~~~~~~~~

Phase 2a: Initial dev releases
******************************

#. Update the *development test VM* with the latest Granta MI software and databases. See the
   ``ansys-grantami-bomanalytics`` repository for details about preparing the Restricted Substances test database.
#. Publish *auto-generated packages* to the internal PyPI mirror with ``.dev`` version numbers.
#. Update *idiomatic package* ``main`` branches to depend on the *auto-generated package* ``.dev`` releases. If the
   ``main`` branch has not been updated to depend on the *auto-generated package* ``main`` branch during development,
   then there may be integration issues discovered at this point - these should be resolved in the PR before merging.
   The PR typically includes:

   * Updates to Python version support to align with changes to the *auto-generated package*.
   * Updates to the ``README.rst`` to reflect any changes in the minimum required Granta MI version.
   * Changes at the interface with the *auto-generated package* if either the *codegen template* or API definition has
     changed.
   * Changes to expected test results. This is typically relevant for ``ansys-grantami-bomanalytics`` which includes
     acceptance-style tests.
#. Publish *idiomatic packages* to the internal PyPI mirror with ``.dev`` version numbers.

   .. note::
      At this point release branches should **not** be created. ``.dev`` releases should be published to the internal
      PyPI mirror from the ``main`` branch.

   .. note::
      These ``.dev`` releases can be used for internal testing of closed-source packages that have runtime
      dependencies on PyGranta packages.

#. Continue to review any changes to API definitions via *auto-generated packages* throughout the hardening phase. If
   there are any issues with the API definitions, these must be escalated with high priority.

Phase 2b: Documentation review
******************************

Review the documentation for all *idiomatic packages* and the *PyGranta meta-package*:

* Significant new documentation is reviewed by the appropriate reviews for linguistic and technical accuracy.
* References to dependencies are updated, including:

  * Server software versions
  * Required Python versions
  * Compatible third-party library versions

* References to Ansys documentation of other packages/products are checked to see if they need to be updated to the
  previous version of that documentation.

  .. note::
     The Ansys documentation for the current in-development release is published after the PyGranta packages are
     released. The documentation for the previous release should be used to ensure links in published PyGranta
     documentation can always be resolved.

Phase 2c: Release candidates
****************************

Towards the end of the hardening phase, once the API definitions are determined to be stable, the *auto-generated
packages* are released:

#. Create ``release`` branches for *auto-generated packages* and publish as release candidates to public PyPI.
#. Create Pull Requests in *idiomatic package* repositories to update ``main`` to depend on the *auto-generated
   package* release candidate releases. Ensure CI passes for *idiomatic packages*.
#. Publish *auto-generated packages* as stable releases to public PyPI.
#. Create Pull Requests in *idiomatic package* repositories to update ``main`` to depend on the *auto-generated
   package* stable releases. Ensure CI passes for *idiomatic packages*.
#. Create ``release`` branches for *idiomatic packages* and publish as release candidates to public PyPI.
#. Update the *PyGranta meta-package* ``main`` branch to depend on the public *idiomatic packages* release candidates
   and *auto-generated packages* stable releases.

   .. note::
      At this point release branches should be created for *idiomatic packages*. However, during hardening these release
      branches still run CI against the *development test VM*.

   .. note::
      *Idiomatic package* release candidates must depend on stable release versions of *auto-generated packages* only.
      They must not depend on pre-release versions of *auto-generated packages*.

   .. note::
      When performing pre-releases of packages with towncrier, the Ansys action creates a Pull Request to update
      CHANGELOG.md on ``main``, and commits the CHANGELOG.md changes directly to the release branch. The commit to the
      release branch should be reverted, and the Pull Request to merge the changes to ``main`` should be closed and
      not merged.

The stable *auto-generated packages* and release candidate *idiomatic packages* can now be used for the final release of
internal packages. It is recommended to make any closed-source *idiomatic package* release candidate dependencies
flexible enough such that they match subsequent stable releases of those *idiomatic packages*.

At the end of this phase, the *PyGranta meta-package* depends on release candidates for the *idiomatic packages* and
releases of the *auto-generated packages*. This guarantees that there exists a consistent set of dependencies across all
*idiomatic packages* and *auto-generated packages*. Any issues at this stage should be resolved by addressing any
dependency conflicts in the *idiomatic packages* and publishing new release candidates to public PyPI as necessary.

Checklist
*********

* |chkbx| The *development test VM* is running an up-to-date development version of Granta MI.
* |chkbx| CI for all *idiomatic packages* is passing.
* |chkbx| Documentation for all *idiomatic packages* and the *PyGranta meta-package* has been reviewed to verify
  significant new content, changes in dependencies (for example Python version support, minimum Granta MI version), and
  references to other Ansys documentation sources.
* |chkbx| ``release`` branches have been created for *auto-generated packages* and *idiomatic packages*.
* |chkbx| Stable *auto-generated packages* are available on PyPI.
* |chkbx| Internal packages with runtime dependencies on *auto-generated packages* depend on the stable PyPI releases.
* |chkbx| *Idiomatic packages* depend on stable releases of *auto-generated packages* only.
* |chkbx| Release candidate *idiomatic packages* are available on PyPI.
* |chkbx| Internal packages with runtime dependencies on *idiomatic packages* depend on the release candidate PyPI
  releases.
* |chkbx| The *PyGranta meta-package* ``main`` branch depends on *idiomatic packages* release candidates and
  *auto-generated packages* stable releases only.

Phase 3: Pre-release
~~~~~~~~~~~~~~~~~~~~

#. Provision a new *production test VM* to run the Granta MI release validated during the hardening phase.
#. Update the latest *idiomatic package* ``release`` branches created during the hardening phase to run CI against the
   newly provisioned *production test VM*.
#. Update the latest *idiomatic package* ``release`` branches created during the hardening phase to **not** run CI
   against the *development test VM*.
#. Manually test the *PyGranta meta-package* against the Granta MI release candidate by installing the ``main`` branch
   directly using ``pip``.

.. note::
   If there have been significant changes to the *PyGranta meta-package*, a release candidate may optionally be
   published to PyPI. Examples of significant changes include re-organization of the documentation and changes to the
   CI process, both of which are not checked as part of the manual test required in this phase.

Testing should occur within 4 weeks following Phase 2. Any issues raised during testing should be immediately triaged
and fixed or deferred.

Checklist for completing this phase:

* |chkbx| There is a new *production test VM* provisioned and running the release version of Granta MI.
* |chkbx| All ``release`` branches no longer target the *development test VM*.
* |chkbx| CI for all *idiomatic packages* is passing.
* |chkbx| Manual testing has been completed against the *PyGranta meta-package* and all *idiomatic packages*.
* |chkbx| The GitHub PyGranta Development project contains no must-deliver features or bug fixes still outstanding for
  the active release.

Phase 4: Release
~~~~~~~~~~~~~~~~

The PyAnsys meta-package is typically released a week before Granta MI is available to customers. As a result, the
stable versions of the packages should be released at least a week before Granta MI is available to customers:

#. Defer any issues still associated with the active release in the GitHub PyGranta Development to a future release.
#. Publish *idiomatic packages* as stable releases to public PyPI.
#. Update the *PyGranta meta-package* ``main`` branch to depend on the public *idiomatic packages* stable releases.
#. Create (or update if a release candidate was published in Phase 3) a ``release`` branch for the *PyGranta
   meta-package*.
#. Publish the *PyGranta meta-package* as a stable release to public PyPI.
#. Create a Pull Request in the PyAnsys meta-package repository to move the *PyGranta meta-package* dependency to the
   new stable release.
#. If any new *idiomatic package* releases are compatible with the previous release of Granta MI and PyGranta,
   including the minimum supported Python version, create a patch release of the previous *PyGranta meta-package*
   release which includes the new *idiomatic package* releases.

Checklist for completing this phase:

* |chkbx| The active release in the GitHub PyGranta Development project is empty, with all deferred tickets re-assigned
  to a future release.
* |chkbx| Stable releases of all *idiomatic packages* and the *PyGranta meta-package* are available on PyPI.
* |chkbx| The PyAnsys meta-package depends on the *PyGranta meta-package* stable release.
* |chkbx| A patch release has been made for the previous *PyGranta meta-package* if there is at least one backwards-
  compatible *idiomatic package* released during this development cycle.

Addenda
-------

Test VM management
~~~~~~~~~~~~~~~~~~

There are multiple *production test VMs* in use at one time, and each runs a validated Granta MI version. The names and
URLs of the *production test VM* are stored in the ``AZURE_VM_NAME_yyRn`` and ``TEST_SERVER_yyRn_URL`` secrets
respectively, where ``yyRn`` is the Granta MI version, for example ``25R2``.

*Production test VMs* are commissioned for a new Granta MI release once it is declared stable, and is decommissioned
when support is withdrawn for that Granta MI release.

The *development test VM* runs a development build of Granta MI, and is upgraded as required to support CI on
*idiomatic package* ``main`` branches during development. The name and URL of the *development test VM* are stored in
the ``AZURE_VM_NAME_DEV`` and ``TEST_SERVER_DEV_URL`` secrets respectively.

The *development test VM* has a green desktop background to indicate it is safe to upgrade.

This approach has the following advantages:

* CI is generally passing on all *idiomatic package* ``main`` branches at any point in development and release, which
  reflects successful CI run against all supported Granta MI releases and a recent Granta MI development build.
* All *PyGranta meta-package* and *idiomatic package* **released** ``release`` branches can be patched and used for
  releases if required. Changes in behavior from subsequent Granta MI development do not affect existing ``release``
  branches.

.. note:: If a *production test VM* has been decommissioned, it should be removed from ``release`` branch CI if a patch
   release is required.

.. |chkbx| raw:: html

    <input type="checkbox">
