# Funktionsklempner
A library and framework for plumbing C++ code to Python via ctypes.

Why ctypes? It has the advantage of being widely available with Python itself, and interfaces C ABIs that can remain stable for long periods of time. This way, a compiled library can work across many different Python versions in which a dynamically linked extension would have to be recompiled due to binary incompatibility. The maintenance burden due to the recurring need to recompile packages is reduced. This comes at the cost of the additional overhead for having to pipe data through a C/ctypes API. Funktionsklempner generates this wrapping code automatically, which reduces the cost to the computational overhead. You may find Funktionsklempner useful for your open source project with low version iteration rate.

Funktionsklempner's basic workflow is as follows:

   1. Declare the interface in a C++ header (i.e. functions whose equivalents shall be
      passed through to the Python world).
   2. Declare all such headers in a project-wide `funktionsklempner.toml` file.
   3. Generated wrapper source files via the `funktionsklempner` command.
   4. Check the generated wrapper files and check them into your version control
      system.
   5. Keep the auto-generated files updated via a git pre-commit hook.


## Usage
Funktionsklempner assumes that you have created a C++ library with an interface that uses the Funktionsklempner (marker) types. For the sake of brevity, let's use the example in the [example](example) subdirectory; i.e. a project that has an [example.cpp](example/example.cpp) source file and whose interface with the Python world is designed in [example.hpp](example/examples.hpp). Specifically, the important part of the example's interface can be illustrated using the single function
```cpp
int test_function(
    const char* name,
    size_t i,
    double* result
);
```
that takes a string `name` and an integer `i`, does something with it, and returns a double as `result`. Note in this interface the pointer-to-double type of `result`: to transfer values to Python via ctypes, Funktionsklempner uses preallocated ctypes variables and passes pointers to them when calling the compiled library. The C++ side is then expected to write the return value into this provided pointer. The actual return value of the function is unused; Funktionsklempner convention is to use `void`.

Note that the actual function in [example.hpp](example/examples.hpp) looks differently! What I have just described is the mechanism that Funktionsklempner makes use of, which is a bit cumbersome to do by hand. To write more explicit code and to make the parser of Funktionsklempner understand the intention of this function definition, it is annoted using macros and alias types defined in [funktionsklempner/funktionsklempner.hpp](subprojects/funktionsklempner/funktionsklempner/funktionsklempner.hpp) and [funktionsklempner/scalar.hpp](subprojects/funktionsklempner/funktionsklempner/scalar.hpp):
```cpp
FUNKTIONSKLEMPNER_EXPORT_FUNCTION
int test_function(
    funktionsklempner::input<const char*> name,
    funktionsklempner::input<size_t> i,
    funktionsklempner::output<double> result
);
```
The first macro `FUNKTIONSKLEMPNER_EXPORT_FUNCTION` tells the parser that what follows is the declaration of a function that shall be callable from the Python world. The templated type alias `funktionsklempner::input<T>` tells the parser that the variables `name` and `i` are input variables passed by value `T`, and the alias `funktionsklempner::output<T>` tells the parser that `result` is a return value of the function (corresponding to a `T*` input parameter). Note that you shall name your variables when exporting via the `FUNKTIONSKLEMPNER_EXPORT_FUNCTION` macro.

What does Funktionsklempner do with this C++ function declaration? On the Python side, it will generate a function that takes a string `name`, an int `i`, and returns a float, i.e.
```python
def test_function(
      name: str,
      i: int
    ) -> float:
    ...
```
On the C++ side, it will wrap `test_function` defined in the header by an exception handling function (any uncaught exceptions passing through to the Python world would crash the program!) that caches any occurring error messages. Between this C++ wrapper and the Python `test_function`, glue code will initialize ctypes variables and watch out for such error messages, raising errors on the Python side appropriately.

All glue code is generated via the `funktionsklempner generate` command after the header [example.hpp](example/example.hpp) has been properly described in the project-wide [funktionsklempner.toml](example/funktionsklempner.toml). The following two sections describe both the `funktionsklempner` command line tool as well as the `funktionsklempner.toml` configuration file format.


### `funktionsklempner.toml`
The `funktionsklempner.toml` file collects all files in a source code repository that shall be wrapped via the `funktionsklempner` command. It has to be placed in the root of the packages source code directory and it is required for the `funktionsklempner` command line utility to work. As an example of its layout, consider the [funktionsklempner.toml](example/funktionsklempner.toml) file of the [example](example) subdirectory:
```toml
[package]
name = "funkybadger"
root = "funkybadger"

[funkybadger]
[funkybadger.example]
header = "cpp/example.hpp"
sources = [ "cpp/example.cpp" ]
```
First, note that the `[package]` entry---including the `name` and `root` fields---is mandatory. The `package.name` field describes the name of the Python packge, and the `package.root` field describes the location, relative to the source code directory's root, of the package's source code (see [src layout vs flat layout](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/), where `package.root` would correspond to the `awesome_package` folder).

Second, there must be one entry per wrapped shared library. In the example, this is `[funkybadger.example]`. All of these entries must be under a root entry that corresponds to the package name (here `[funkybadger]`). The entry's key must correspond to the module name which shall be generated and from which the wrapped functions shall be imported (here the module `funkybadger.example`). Each shared library entry must contain two fields:

   1. `header`, a string denoting the path to the Funktionsklempner-annotated, API-defining
      header file (here [`example/example.hpp`](example/example.hpp)). The path shall be relative
      to the source directory's root.
   2. `sources`, a list of (strings denoting paths to) source files which encompass the shared
      library.

#### Optional fields
There are further optional fields that can be used to tweak the code that Funktionsklempner injects into the Meson build file. This way, necessary information (such as a static library that must be linked to) can be passed to the autogenerated Meson build code. The following optional fields are available:

   1. `link_with`, a list of strings denoting `static_library` targets of the injection-targeted
      Meson build file which the Python shared library shall be linked with.
   2. `dependencies`, a list of strings denoting `dependency` objects of the injection-targeted
      Meson build files on which the Python shared library depends.
   3. `wrapper_source`, a file name specifying the wrapper C++ source file
      relative to the repository's root.
   4. `include_root`, a directory name specifying the root include directory
      for the library. If not given, will default to the directory in which
      `header` resides.

As an illustration, the code
```toml
[package.library]
header = "..."
sources = [ "..." ]
link_with = [ "linklib" ]
dependencies = [ "a_dependency" ]
wrapper_source = "cpp/src/library_wrapped.cpp"
```
in the `funktionsklempner.toml` configuration file will generate a wrapper source file at `cpp/src/library_wrapped.cpp` (relative to the repository root), and inject the following code into the project's Meson build file:
```meson
package_library = shared_library(
    'library',
    # ...
    'cpp/src/library_wrapped.cpp',
    dependencies: [
        a_dependency,
    ],
    link_with: [
        linklib,
    ],
    # ...
)
```


### Command line
The Funktionsklempner package comes with the `funktionsklempner` command line utility. This program sets up the Meson subprojects and git pre-commit hook, injects the project's main Meson build file, and generates the wrapper code. It has three main commands, two of which are typically used by users themselves.

#### `funktionsklempner init`
This command needs to be called once after the `funktionsklempner.toml` configuration file has been written. The command performs the following tasks:

   1. Setup the `funktionsklempner`, `objektbuch`, `gcem-1.18.0`, and (optional) `boost-1.90.0` Meson subprojects that are required for
      the automatic error propagation to the Python side.
   2. Setup the git pre-commit hook that internally uses `funktionsklempner verify` to check
      that the generated code currently residing in the repository is up to date.
   3. Start injecting the Funktionsklempner code into the project's Meson build file.

Boost will be preferably used from the system side, but particulary during the pip package building, Boost might not be found. In this case, the reduced subset of Boost (shipping code relevant to Boost.Units) will act as a fallback subproject.

#### `funktionsklempner generate`
This command is the main workhorse. It generates or updates wrapper codes for all libraries described in `funktionsklempner.toml`, and updates the injection into the project's Meson build file. Call this command whenever a definition in one of the API-defining headers changes.


#### `funktionsklempner verify`
This command checks whether the potential result of `funktionsklempner generate` is identical to what is currently found in the source tree, raising an error if not. This command is used by the git pre-commit hook to ensure that the autogenerated code stays up to date with the API header definitions.


### Header annotation syntax
The introductory example lays out the basic syntax for annotating C++ headers using `FUNKTIONSKLEMPNER_*` macros. The following macros are available:

| Macro                                       | Purpose                                                                                                | Scope           |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------- |
| `FUNKTIONSKLEMPNER_EXPORT_FUNCTION`         | Export the following function to Python module                                                         | Top level       |
| `FUNKTIONSKLEMPNER_EXPORT_STRUCT`           | Export the following structure to Python module                                                        | Top level       |
| `FUNKTIONSKLEMPNER_SIZE_CONTROL("X", "Y")`  | Declare variable `X` as an (output) array whose size is controlled by the (input) array(s) or integer `Y` | Function export |

####  `FUNKTIONSKLEMPNER_EXPORT_FUNCTION`
This function will generate a wrapper for the following function and export it to the Python module.

####  `FUNKTIONSKLEMPNER_EXPORT_STRUCT`
This function will generate a `ctypes.Structure` wrapping the following `struct` and export it to the Python module. Functions accepting pointers to this struct (including as return values!) will then be able to pass this data type to and from the Python world. This can be helpful when complex data types shall be transferred with a limited amount of ctypes calls.

#### `FUNKTIONSKLEMPNER_SIZE_CONTROL("X", "Y")`
A way to return NumPy arrays with known size. Works with and is required for `funktionsklempner::OutputNDArray` and `funktionsklempner::OutputQuantityArray`-type variables. Declares that the variable `X` is an array of size given by the variable `Y`. On the Python side, the two C++ parameters `X` and `Y` will be translated into a single NumPy array `X` of shape `Y.shape` (if `Y` is an input array) or `(Y,)` (if `Y` is not an input parameter of the C++ function, in which case it will be generated as an integer input on the Python side). This allows handling the frequent task of performing computational tasks on vector data kept in Numpy arrays.

The simplest way to automatically derive the size of an output array is by following the size of an input array:
```cpp
FUNKTIONSKLEMPNER_EXPORT_FUNCTION
FUNKTIONSKLEMPNER_SIZE_CONTROL("out", "in_")
void a_function(
    funktionsklempner::InputNDArray<double> in_,
    funktionsklempner::OutputNDArray<double> out
);
```
Here, `in_` is an automatically converted input array, and the return value `out` of the Python wrapper to this function will be a newly generated Numpy array of size `out.size = in_.size`.

##### Multidimensional size control
Sometimes, following the size of a single array will not be enough. Think of a two-parameter function evaluated on a meshgrid, for instance. For this particular use case, one can use the size multiplication syntax:
```cpp
FUNKTIONSKLEMPNER_EXPORT_FUNCTION
FUNKTIONSKLEMPNER_SIZE_CONTROL("X", "Y,Z")
void a_function(
    funktionsklempner::InputNDArray<double> Y,
    funktionsklempner::InputQuantityArray<boost::units::si::length, double> Z,
    funktionsklempner::OutputNDArray<double> out
);
```
Here, `Y` is an automatically converted input array of shape `Y.shape`, `Z` is an automatically converted input array of shape `Z.shape` and length dimension, and the return value `X` of the Python wrapper to this function will be a newly generated Numpy array of shape `X.shape = (*Y.shape, *Z.shape)`.

The size control works equivalently with QuantityArrays instead of NDArrays.

### `funktionsklempner::NDArray<NumPyDType T>`
This class is a thin random access range wrapper around a scalar pointer (e.g., a NumPy array buffer). It is accessible in the `funktionsklempner` namespace as `InputNDArray<T>` (=`NDArray<const T>`, read-only access to the underlying buffer) and `OutputNDArray<T>` (=`NDArray<T>`, can be written to).

The API of the class is as follows:
```cpp
template<NumPyDType T>
class NDArray
{
public:
    /* Constructors: */
    constexpr NDArray(T* data, size_t N, const char* dtype);
    constexpr NDArray(NDArray&&) noexcept = default;
    constexpr NDArray(const NDArray&) noexcept = default;

    /* Sized range: */
    constexpr size_t size() const noexcept;

    /* Random access iterators, pointers to the underlying buffer: */
    constexpr T* begin() const noexcept;
    constexpr T* end() const noexcept;

    /* Random index-based access: */
    constexpr T& operator[](size_t i) const;
};
```
Note that `NumPyDType` is a concept that captures the basic scalar types of NumPy array buffers (integer and floating point types).


### `funktionsklempner::QuantityArray<Unit U, NumPyDType T>`
A rather thin random access wrapper around a scalar pointer that represents values of a certain physical (SI) unit via Boost.Units. Iterating over the quantity array yields value views that act, basically, as physical quantities of the given unit `U` (`boost::unit::quantity<U, T>`). It is accessible in the `funktionsklempner` namespace as `InputQuantityArray<U, T>` (=`QuantityArray<U, const T>`, read-only access to the underlying buffer) and `OutputQuantityArray<U, T>` (=`QuantityArray<U, T>`, can be written to).

The value type of the iterators of `QuantityArray` is the the `QuantityView<U,T>` class, which internally contains a pointer to the numerical buffer of the `QuantityArray` and a `boost::unit::quantity<U, T>` that represents the physical unit. The `QuantityView` can be casted to `boost::unit::quantity<U, T>`, and addition, subtraction, multiplication, and division operators of `QuantityView<U,T>` with `boost::unit::quantity<U, T>` are provided. In case of the `OutputQuantityArray`, assignment and inplace addition and subtraction operators with `boost::unit::quantity<U, T>` are provided as well. Essentially, `QuantityArray<U,T>` can be used to wrap a floating point buffer `T*` in a way that looks like a buffer of quantities, `boost::unit::quantity<U, T>*`.

The API of the class is as follows:
```cpp
template<Unit U, NumPyDType T>
class QuantityArray
{
public:
    /* Constructor with given unit of same dimension as `U`. */
    constexpr QuantityArray(
        T* data,
        size_t N,
        const char* dtype,
        const char* unit
    );

    /* Constructor with default unit `U` */
    constexpr QuantityArray(
        T* data,
        size_t N,
        const char* dtype
    );

    /* Sized range: */
    constexpr size_t size() const noexcept;

    /* Random access iterators, pointing to the underlying buffer: */
    constexpr QuantityIterator<U, T> begin() const;
    constexpr const std::decay_t<T>* end() const noexcept;

    /* Random index-based access: */
    constexpr QuantityView<U, T> operator[](size_t i) const;

    /* Setting the unit to `u`, a quantity of unit `U`.
     * Note: this method is enabled only if `T` is not a const value
     *       (i.e. for OutputQuantityArray<U,T>).
     */
    constexpr void set_unit(const boost::units::quantity<U>& u);
};
```
Note that `Unit` is a concept that captures `boost::units::unit<Dim, System>` of some dimension `Dim` and system `System` (which essentially has to be the SI system from `boost::units::si` to be able to interact with the remainder of Funktionsklempner). As before, `NumPyDType` is a concept that captures the basic scalar types of NumPy array buffers (integer and floating point types).

## Caveats
Some known caveats to keep in mind when using Funktionsklempner:

   1. This uses a simplified grammar.
   2. There is some effort in this code base to handle comments. However, this
      effort is designed empirically rather than after a standard. You might
      run into complications with more 'unusual' comment or preprocessor layouts.
      If you hit such a bug, you may try to file a bug report--but there is no
      guarantee that all such bugs will be fixed (in which case reduction to the
      simplified grammar may be the only way to run Funktionsklempner).

Note, furthermore, the terms listed in the [LICENSE](LICENSE).


## License
Funktionsklempner itself is licensed under the European Public License, v1.2 (`EUPL-1.2`, see [LICENSE](LICENSE) file in this directory). The template code for the code generation is licensed under the BSD 3-Clause license (`BSD-3-Clause`, noted in corresponding source files). The example under the [example/](example) subfolder is equally licensed under the `BSD-3-Clause` license.

Funktionsklempner vendors a subset of the Boost C++ Libraries, version 1.90.0 in [subprojects/packagefiles/boost-1.90.0_units.tar.gz](subprojects/packagefiles/boost-1.90.0_units.tar.gz). This code is licensed under the Boost Software License as indicated in the archive. You can find the full Boost library at [boost.org](https://www.boost.org/).


## Changelog
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).


### [1.0.0] - 2026-03-09
#### Added
- Start of the changelog.