Bob 0.15 Release Notes
======================

Changes made since Bob 0.14.0 include the following.

Prerequisites
-------------

Bob requires Python 3.5 or later starting with 0.15.

New features
------------

Recipes
~~~~~~~

* Added the ``sslVerify`` SCM attribute to git, svn and url SCMs.

  The boolean attribute controls whether to verify the SSL certificate when
  fetching. It defaults to ``true`` but might be set to ``false`` to ignore
  certificate problems. The default for git is defined by the
  :ref:`policies-secureSSL` policy.

* Introduced the ``netAccess`` :ref:`configuration-recipes-provideTools`
  property.

  Starting with the introduction of the :ref:`policies-offlineBuild` policy the
  network access is usually restricted during build and package steps. There
  might be tools that need network access and are used during these steps,
  though. This can be configured with the
  :ref:`configuration-recipes-netAccess` recipe properties but if the recipe
  does not know the exact tool it might not want to set these properties
  unconditionally.

  The main use case are proprietary compilers that need to talk to a license
  server. The recipe should not bother which compiler is used exactly.

* :ref:`configuration-recipes-provideTools` gained the ``environment`` property.

  Certain tools (e.g. the C-compiler) and their associated environment
  variables (``CC``, ``LD``, ...) are related to each other and must be defined
  and used together. Traditionally they are provided more or less independently
  of each other by a tools recipe with ``provideTools`` and ``provideVars``.
  This has several drawbacks:

  * Not all recipes that define the same tool (e.g. the compiler) are
    providing the same set of environment variables.
  * Different tools might provide the same variables.

  By defining the environment variables with the tool their value can be picked
  up where it is actually used. This does not pollute the environment with
  variables that are only used in conjunction with the tool and are not used
  elsewhere.

  To prevent ambiguities each variable must only be defined by one tool. The
  parsing will fail if two tools are used in the same recipe that define the
  same variable. The reason is that Bob assumes that tools are independent of
  each other and no particular order is defined among them.

* Host fingerprint support.

  The fingerprint feature is used to track all dependencies of the recipes to
  the host environment in a generic manner. This is done by defining small
  scripts in the recipes that are invoked when the fingerprint is required. The
  output of these scripts is then taken as fingerprint.

  Fingerprints may range from the host C-compiler up to some libraries that are
  used by certain packages from the host. Because such external dependencies
  limit the exchange of artifacts between hosts the fingerprint is attached to
  binary artifacts to prevent unintended sharing between incompatible systems.

  The fingerprints are also used to detect relevant changes of the host
  environment. A package is always rebuilt if the fingerprint changes.
  The fingerprint is thus like an external build trigger. There are two major
  cases why Bob wants to rebuild when the fingerprint changes:

  * The package was built inside the sandbox and now the user builds outside of
    the sandbox (or vice versa).
  * The package was built without sandbox and the fingerprint changes. Bob has
    to assume that the external dependency has changed its state and thus an
    incremental rebuild is needed.

  If a package is not relocatable the workspace path is also some kind of host
  dependency. Bob treats the installation path of these tools as an additional
  fingerprint. This enables safe exchange of such artifacts that were
  previously only up-/downloaded when building inside a sandbox. This is only
  done if the :ref:`policies-allRelocatable` policy is set to the new
  behaviour, though.

  See :ref:`configuration-recipes-fingerprintScript` and
  :ref:`configuration-recipes-provideTools` for further details.

* Allow local files with the "url" SCM.

  If the URL starts with ``/``, it is treated as a local file and is just
  copied with ``cp`` instead of downloaded using ``curl``.  This makes it
  possible to easily build from tarballs in a sandbox that has no network
  access and no ``curl`` command.

* Add ``BOB_RECIPE_NAME`` and ``BOB_PACKAGE_NAME`` built-in variables

  These variables are set internally by Bob and can be consumed by a recipe or
  class to know which recipe/package is built. For example it might be used to
  name the artifact (think rpm/deb/ipkg pacakge) like the recipe.

User configuration (default.yaml)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* Added ``require`` keyword.

  User configuration files may also require sometimes specific files to be
  included. Therefore the ``require`` keyword is introduced. The ``require``
  keyword throws an error for missing files. Other than that it behaves just
  like the existing ``include`` keyword.

  See :ref:`configuration-config-usr`.

* Added the ``sslVerify`` http archive backend attribute.

  The boolean attribute controls whether to verify the SSL certificate when
  fetching/uploading to/from HTTPS servers. The default depends on the
  :ref:`policies-secureSSL` policy. The new behavior is to default to ``true``
  but it might be set to ``false`` to ignore certificate problems.


Bob build / bob dev
~~~~~~~~~~~~~~~~~~~

* Gained parallel build support.

  If requested by ``-j`` a number of jobs can run simultaneously. Any
  checkout/build/package step that needs to be executed are counted as a job.
  Downloads and uploads of binary artifacts are separate jobs too. If a job
  fails the other currently running jobs are still finished before Bob returns.
  No new jobs are scheduled, though, unless the ``-k`` option is given.

  If the ``-j`` option is given without an argument, Bob will run as many jobs
  as there are processors on the machine.

* Added the ``--no-link-deps`` option.

  Creating links to dependencies confuses indexers like OpenGrok. This switch
  is there to disable the creation of linked dependencies.

Bob clean
~~~~~~~~~

The ``clean`` command has been extend to develop mode and attic directories.
It will now purge unused workspace directories from develop mode builds too.
If requested, it also removes attic directories.

At the same time the default is changed to 'develop' mode. This streamlines the
behaviour with the other commands in Bob that also work in 'develop' mode by
default. Together with the added ``--(no-)sandbox`` options it's configuration
is finally consistent with the other commands.

If the user removes source workspaces (``-s`` or ``--attic`` options) Bob will
now check the SCMs in these directories for unsaved changes. The directory will
only be deleted if no changes are found.

Attic directories are tracked starting with Bob 0.15. Any attic directories
that were created with an older version of Bob in a project are not known and
will not be cleaned. Bob will print a warning in case the project was created
with an older version and attic directories should be cleaned.

Bob Jenkins
~~~~~~~~~~~

* Added the ``jobs.update`` option

  Updating a Jenkins job is costly. On projects with hundreds of jobs the
  overhead can be significant when just the description or audit trail
  information is updated but the job is not actually scheduled. The
  'jobs.update' option adds a knob to control the update behaviour and provide
  lazy update methods.


Bob status
~~~~~~~~~~

* Flag directories that will move to the attic or will be created.

  If the recipe is changed for a SCM checkout it will be moved to the attic
  when the checkout is run the next time. Bob now flags the directory as ``A``
  (as in "attic") so that the user knows about the stale state. Likewise the
  ``N`` flag shows checkouts that do not exist in the workspace yet but will be
  created on the next run.

* Print override status as yaml.

  The ``scmOverrides`` configuration is done in a yaml file. Print the active
  overrides in the same format for consistency.

* The package argument is now optional.

  If no package is given then all known checkouts are scanned for changes. This
  will only give meaningful results if the project was created with Bob 0.15,
  though, because older versions of Bob did not store enough information about
  the project.

* Added an ``--attic`` switch to display information about the attic
  directories.

  The switch is orthogonal to the package argument. If a package is selected
  then all attic directories of this package are displayed. Otherwise all known
  attic directories are scanned.

* For git repositories ``bob status`` will now check all unpushed commits.

  This does not only check local branches for unpushed commits but all refs.
  Therefore stash, detached HEAD, etc. are checked as well.

* Improved output verbosity and streamlined its selection.

  The ``--show-clean`` was added to explicitly show unchanged checkouts. These
  are now hidden by default. The behaviour of the ``-v`` and ``--show-clean``
  options has been made more consistent. See
  :ref:`manpage-bob-status-verbosity` of the ``bob status`` manpage.

* The ``--sandbox`` / ``--no-sandbox`` options were added.

Bob query-meta, query-scm, query-recipe
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* The ``--sandbox`` / ``--no-sandbox`` options were added.

Changed behaviour
-----------------

Backwards compatible policies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* Added :ref:`policies-mergeEnvironment` policy.

  The ``environment`` and ``privateEnvironment`` sections of the recipes and
  classes it inherits from are merged when the packages are calculated.
  Traditionally this was done on a key-by-key basis without variable
  substitution. Keys from the recipe or an inherited class would simply shadow
  keys from later inherited classes. This had the effect that the definitions
  of later inherited classes were lost.

  The new behavior is to make all environment keys eligible to variable
  substitution. The definitions of the recipe has the highest precedence
  (i.e. it is substituted last). Declarations of classes are substituted
  in their inheritance order, that is, the last inherited class has the
  highest precedence.

  See :ref:`policies-mergeEnvironment` for more details.

* Added the :ref:`policies-secureSSL` policy.

  Due to historical reasons Bob did not check for SSL certificate errors
  everywhere. While most parts were already secure the git SCM and HTTPS
  archive backend were still insecure by default. This is rectified by the
  secureSSL policy where the new behavior is to always check the certificate.

Other behavioural changes
~~~~~~~~~~~~~~~~~~~~~~~~~

* ``bob jenkins`` defaults to secure SSL connections.

  Always use secure SSL connections by default. If the user still needs to
  connect to insecure HTTPS servers the ``--no-ssl-verify`` option may be used.

* The default project directory name of the built-in ``bob project`` generators
  have been shortened.

  By default the package path was used for the project directory. If the built
  package was deep in the hierarchy this could lead to excessively long path
  names. Instead Bob now uses the project name (which defaults to the package
  name) as directory name.

  This could theoretically create clashes if different variants of a recipe are
  built simultaneously. But the user can handle that by the ``--destination``
  option rather than creating excessively long paths by default.

* Fixed the :ref:`policies-sandboxInvariant` policy.

  Setting the :ref:`policies-sandboxInvariant` policy to the new behaviour had
  exactly the opposite effect as intended. It caused artifacts of sandbox and
  non-sandbox builds to be always incompatible to each other. This has been
  rectified.

* On git checkouts only tags that are on branches are cloned by default.
  Previously all tags have been cloned. This is done to prevent ``bob status``
  from flagging a repository as having unpushed commits because orphaned tags
  are not referenced from remote heads. Note that git does not fetch such tags
  anyway on a ``git pull`` or ``git fetch``. The user has to fetch orphaned
  tags explicitly either by naming them on the command line or with the
  ``--tags`` option. If the recipe requests such a tag to be checked out then
  it will still be fetched explicitly by Bob.

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* Recipes cannot define variables starting with ``BOB_``

  Variable names starting with ``BOB_`` have historically been used by Bob all
  over the place. This release added even some more (``BOB_RECIPE_NAME``,
  ``BOB_PACKAGE_NAME``). To prevent accidential breakage of recipes by future
  versions of Bob that define additional variables, Bob will reject recipes
  that define variables in this name space. This may break existing recipes but
  it cannot be guaranteed that they will work with future versions either.

* The default mode of the ``bob clean`` command was changed from release mode
  to develop mode. This makes its behaviour consistent with all other Bob
  commands.

* The :ref:`extending-hooks-string` and :class:`bob.input.PluginState` APIs
  have changed.

  Due to the redesigned package calculation it is not possible to pass the
  :class:`bob.input.Tool` or :class:`bob.input.Package` objects to the plugins
  anymore that use  :class:`bob.input.PluginState`. Plugins requiring the old
  API still work but the ``tools`` and ``package`` arguments are filled with
  empty values.

  Similarly the string functions lost the ``tools`` parameter. The ``sandbox``
  parameter was converted to a plain ``bool``.

  This changes the behaviour and projects relying on the removed parameters will
  have to be refactored. But at least the parsing should not crash. A warning
  is displayed for every usage of a deprecated API.

Performance improvements
------------------------

* Refactored package calculation.

  The internal logic of the package calculation was optiomized. The parsing
  time and memory footptrint are improved by up to an order of magnitude.

* ``bob graph`` scales much better on large package graphs.
