Problem with LEX/YACC table modules generated by ply
====================================================

The LEX/YACC table modules are generated by the `ply` package and contain
the version of the `ply` package that generated them. These modules are
packaged into the pywbem distribution archive and are up to date w.r.t.
the `ply` package on the build system.

However, the target Python environment may have a different version of `ply`
installed. Any use of `ply` of these table modules will re-generate them
automatically in that case. There are also more subtle reasons for
re-generation that are not fully understood at this point.

The automatic re-generation of the table modules creates two potential issues:

* The target Python environment may be a system Python, and the user that
  uses the package may not have permission to update the table modules.

* The re-generation produces some log messages on stdout, and depending
  on the circumstances this may be a problem.

Our strategy is therefore to ensure that the table modules are up to date
in the target Python environment, so that they will not need to be
re-generated. That is a solution for many but not for all situations, because
a user with permissions could update the `ply` package and lateron a user
of pywbem without permissions would trigger a re-generation that fails.

The following pywbem installation tools need to be supported:

* `pip install`, with any package source (local dist archive, or PyPI).
* `python setup.py install`, from the repo work directory.

At this point, we did not find a way to build the table modules
automatically in both types of installation. Part of the problem is that
setuptools and pip do the installation quite differently.

Generally, the approach can be to update the table modules in the Python
source files that are used as input to the installation process, or to update
them after the installation in the place where they have been installed to.

Updating them in the input files is the proper approach considering that
there are post-processing steps such as pre-compiling the files into Python
byte code. However, at such an early point in the installation, the
dependencies of the pywbem package (e.g. `ply` itself) have not been
installed (e.g. if the environment is fresh), and they are needed to build
the table modules.

The dependency problem is the reason to choose the approach of updating
the installed modules. That is possible with `python setup.py install` in
a number of ways, but so far we did not find a way to do that for
`pip install`, because every hook that is possible to use is before
pip finally copies the files into the target place where they would
be found in the Python module search path.

Just for the record, here is what was tried:

* Use the `setup_requires` keyword of `setup()` in order to establish
  the prerequisites for building the table modules, and hook up in the
  install process (e.g. custom `install` command, or somewhere else).
  This runs into the problem that the dependencies specified with
  `setup_requires` get installed into a local .eggs directory, and even
  when they are again specified in `install_requires` their 'real'
  installation is skipped because they are already found loaded.
  See
  [setuptools issue 391](https://bitbucket.org/pypa/setuptools/issues/391/).
  Conclusion is that `setup_requires` cannot be used at all, at the moment.

* Have a custom `install` command and invoke this script after calling
  the original function. The `install` command is invoked by both setup.py
  and pip. This runs into the problem that the original function
  (`install.run()` in `setuptools.command.install`) looks at the call stack
  and because of the presence of the custom install hook, takes a different
  path that leads to skipping the installation of dependencies specified in
  `install_requires`. See
  [setuptools issue 456](https://bitbucket.org/pypa/setuptools/issues/456/)
  Conclusion is that a custom `install` command cannot be used at all, at
  the moment.

* Invoke this script in the `setup.py` script after returning from `setup()`.
  This script must be run in a child Python in order to pick up the possibly
  changed module search path resulting from installing eggs.
  For setup.py this works, but has the drawback that it is not clear how
  the setup.py command can be figured out, so it is done for every command.
  For pip, this does not work because even after returning from `setup()`,
  the pywbem module files are not yet in their final destination.

* Some more variations of custom commands, but they were either too early,
  or not used by both setup.py and pip.
