.. _ds_orm:

ORM basics
==========

As been said before in the *Introduction*, Wefram-based project
uses *SQLAlchemy ORM*.

Every storing in the relational database entity describes as
Python-class. All those classes directly maps to the corresponding
database tables. When the project programmer about to read objects
from the database, about to create or update corresponding
object, delete and deal with relationships - he or she deals not
with database itself with raw SQL queries (it is not prohibitted
at all, but strongly not recommended to), but uses corresponding
method of Python classes.

It is great that Wefram project uses one of the most popular
ORM library, so programmer easely can find answers on most
questions on how to deal with it.

For better understanding *what is ORM* it is best to read corresponding
documentation not here, but on the SQLAlchemy documentation site.


The library
-----------

To use ORM, the programmer about to import [ds] module from the
[system] one:

.. code-block:: python

    from system import ds

The [ds] module has much of needed for ORM operation. Most of the
SQLAlchemy types, classes and functions transparently re-exported
within [ds] module.


Models
------

The every entity about to be defined as python classes, basing
on the :class:`system.ds.orm.model.Model` parent class:

.. code-block:: python

    from system import ds

    class MyModel(ds.Model):
        id = ds.UUIDPrimaryKey()
        name = ds.Column(ds.String(255), nullable=False, default='')
        some_value = ds.Column(ds.Integer(), nullable=True)

    class MyAnotherModel(ds.Model):
        id = ds.BigAutoIncrement()
        name = ds.Column(ds.String(60))
        jsoned_data = ds.Column(ds.JSONB())

As you see here - we have defined two models. Each of them
consists of several attributes. Most of them bases on the
[ds.Column] whose defines the corresponding representation
in the database. So, each attribute which is defined as
[ds.Column(*something*)] - will be created as the corresponding
column in the database with corresponding type and type
parameters.

The very important part of the each model is a *primary key*. As you
can notice - both of example models have first attribute named
[id]. The exact name is not important - you may name this attribute
(as any other) as you with, except of reserved SQL or
PostgreSQL words.

We will not describe the relational database logics here, this is
not the topic of the Wefram. But, we will show you how to make
it simplier.

Defaultly, with SQLAlchemy, a primary key defines as any another
column - using [ ds.Column(...) ]. The Wefram suggests you
to use one of two primary key types:

#.  **UUID** which is, as we think, the best way to identify
    objects in the relational database. It is best to identify
    single objects when their quantity is not very much.

#.  **Integer** or **BigInteger** as an alternative. This type
    is best to identify objects when their quantity is much more
    than several. For example, when we storing history records (logs),
    or kind of orders.

And, to avoid misspelling, errors in declaration of primary keys and so
on, Wefram offers helper functions.


UUID primary keys
^^^^^^^^^^^^^^^^^

The example of models with UUID primary key usage:

.. code-block:: python

    from system import ds

    class MyModel(ds.Model):
        id = ds.UUIDPrimaryKey()
        ...

The PostgreSQL nativelly supports UUIDs and the Wefram
uses the PostgreSQL-based UUID as primary key. Additionally,
Wefram *cares about default* primary key value.


Integer primary keys
^^^^^^^^^^^^^^^^^^^^

Integer primary keys usually called ``auto-increment`` types. The
PostgreSQL does not knows *auto-increment*, instead it uses own
special types: ``serial`` or ``generated by default as identity``.
Let's not flow deep into PostgreSQL logics and etc, but instead
Wefram offers more usual type ``AutoIncrement`` and
``BigAutoIncrement``.

.. code-block:: python

    from system import ds

    class MyIntPkModel(ds.Model):
        """ This is model with Integer auto-increment key """
        id = ds.AutoIncrement()
        ...

    class MyBigIntPkModel(ds.Model):
        """ This is model with BigInteger auto-increment key """
        id = ds.BigAutoIncrement()
        ...


CRUD basics
-----------

CRUD, or **Create**, **Read**, **Update**, **Delete**, like any other
operations on ORM, are handled by SQLAlchemy logics. While the project
programmer is free and, speaking more, advised to use SQLAlchemy v2.0 query
writing for complex requests, Wefram offers helper methods on the [ds.Model]
class to make work with simple cases much easier.


Fetching objects from database
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Reading from the database - is the most often operation we about to do
while speaking about relation databases. There are two approaches to this
operation:

#.  Using the default SQLAlchemy query writing algorithms;

#.  Using simple Wefram helper methods.


The **first** approach is very useful when you have to make complex
query. For example, when you need to create a report query fetching
from many tables with different joins, subqueries and etc. Of cource,
this approach is still 100 percent applicable to execute the simpliest
query in you life, but you may make it much easier with the second
one.

The SQLAlchemy based query writing is much better to read at the
SQLAlchemy docs site, really not here. But, for the basic example
how to begin, let's give an example here:

.. code-block:: python

    from system import ds


    # Declaring the model
    class MyModel(ds.Model):
        id = ds.UUIDPrimaryKey()
        name = ds.Column(ds.String(100), nullable=True, default=None)
        some = ds.Column(ds.Boolean(), nullable=True, default=False)


    # And querying this model with SQLAlchemy based request
    query = ds.select(MyModel).where(ds.or_(
        MyModel.some.is_(False),
        MyModel.name.is_(None)
    )).order_by(MyModel.name)

    results = await(
        ds.execute(query).scalars().fetchall()
    )


The **second** approach is very useful when you using simple queries. We do not
have to use complex query modelling, we want to just call a single, simple
method of the corresponding model. This approach uses Wefram extensions to
the SQLAlchemy ORM model. When you declare your model with parent [ds.Model]
class - you declare with extended logics already.

Lets repeat the example above with the simplier variant.

.. code-block:: python

    from system import ds


    # Declaring the model
    class MyModel(ds.Model):
        id = ds.UUIDPrimaryKey()
        name = ds.Column(ds.String(100), nullable=True, default=None)
        some = ds.Column(ds.Boolean(), nullable=True, default=False)


    # Using a simplier helper method
    results = await MyModel.all(ds.or_(
        MyModel.some.is_(False),
        MyModel.name.is_(None),
        order='name'
    ))


Or even simplier, if we have an *AND* filter in the query:

.. code-block:: python

    ...

    # Even more simplier query
    results: List[MyModel] = await MyModel.all(
        some=False, name=None, order='name'
    )


And the last example, as simple as possible:

.. code-block:: python

    # Whooos maximum simple
    results: List[MyModel] = await MyModel.all()


As we see - instead of writing a SQLAlchemy-based query we called a
single simple method :py:meth:`~system.ds.orm.model.Model.all` of the
corresponding :class:`system.ds.orm.model.Model`. Of cource, helper
methods cannot be used in all cases, but them makes a simple operations
handled simple.


Offset and limit
^^^^^^^^^^^^^^^^

We can limit the quantity of resulting objects by using ``limit``
named argument for :py:meth:`~system.ds.orm.model.Model.all` or
:py:meth:`~system.ds.orm.model.Model.select` methods. This argument
indicates maximal the quantity of objects we allow to be fetched
from the database.

.. code-block:: python

    # Fetch not more than 50 objects
    instances = await MyModel.all(limit=50)


Like ``limit`` limits the quantity of returned in the result objects,
``offset`` set the starting position of that array. In other words,
``offset`` tell the database "okey, fetch objects from X row".

.. code-block:: python

    # Fetch objects begining from 75th one only
    instances = await MyModel.all(offset=75)


Using ``limit`` and ``offset`` in couple we can make pagination. Let's
look at example:

.. code-block:: python

    # Lets assume that every page must not exceed 50 rows

    # Fetching the first page: offset=0, limit=50
    page1 = await MyModel.all(offset=0, limit=50)

    # The second page will have offset=50 and limit still = 50
    page2 = await MyModel.all(offset=50, limit=50)

We think that we made a point on these two great parameters.


Ordering the query results
^^^^^^^^^^^^^^^^^^^^^^^^^^

To order the query results we about to use named argument ``order``.
It assumes one of the next values:

*   The (str) name of the corresponding column
*   The list of (str) names of the corresponding columns
*   The column instance itself
*   The list of column instances

If the string (str) type have been used, then you may revert
ordering by prepending the corresponding column name with minus
sybmol ('-').

For the corresponding column variant, we can use SQLAlchemy's
approach with ``.desc()`` method.

Let's look at examples for better understanding:

.. code-block:: python

    # Sorting by one column 'name'
    instances = await MyModel.all(order='name')

    # The same with column instance
    instances = await MyModel.all(order=MyModel.name)

    # The same but in reverse
    instances = await MyModel.all(order='-name')
    instances = await MyModel.all(order=MyModel.name.desc())

    # Sorting using two columns at a time
    instances = await MyModel.all(order=['last_name', 'name'])
    instances = await MyModel.all(order=[MyModel.last_name, MyModel.name])


Searching by the text term
^^^^^^^^^^^^^^^^^^^^^^^^^^

The Wefram offers one interesting automation - the search facility. This is
not the part of SQLAlchemy itself, as must other facilities, and not a
simplification in general. This is a kind of automation which realises the
one of the most used cases when fetching objects from the database - the
**search**.

To make this happened, we need to use :class:`~system.ds.orm.model.Meta` subclass (which is described
in the :ref:`ds_models`. We will show a very short part of it here,
just to make sense how to make some model searchable. And how to use that.

The search for objects by textual term is done by calling the class method
``like`` or ``ilike`` for case sensetive and case insensetive, respectively,
within corresponding fetching method, like :py:meth:`system.ds.orm.model.Model.all`

.. code-block:: python

    from system import ds


    class Contant(ds.Model):
        id = ds.UUIDPrimaryKey()
        first_name = ds.Column(ds.String(100), nullable=False, default='')
        middle_name = ds.Column(ds.String(100), nullable=False, default='')
        last_name = ds.Column(ds.String(100), nullable=False, default='')
        avatar = ds.Column(ds.Image(), nullable=True)
        phone = ds.Column(ds.Phone(), nullable=False)
        email = ds.Column(ds.Email(), nullable=False)

        class Meta:
            # Here we declares whose columns will be findable for
            # the search facility
            findable = ['first_name', 'last_name', 'email', 'phone']


    # let's find something
    found = await Contact.all(Contact.ilike("mike"))



Selecting for update (locking objects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is normal situation when you need to select objects whose about to
be updated or even deleted in the request process. And you want to be
sure that those objects may be read by other processes if only those
processes does not wants to update them too, and guarantee that those
processes (whose want to update the same array of objects) will wait
until the current process finish.

In the raw SQL queries this usually done by adding special modifiers
to the SELECT query, ``SELECT .. FOR UPDATE`` when we speaking about
PostgreSQL, for example.

To add this kind of modifier using Wefram helper methods such as
:py:meth:`~system.ds.orm.model.Model.all`,
:py:meth:`~system.ds.orm.model.Model.select`,
:py:meth:`~system.ds.orm.model.Model.first`,
:py:meth:`~system.ds.orm.model.Model.get`,
:py:meth:`~system.ds.orm.model.Model.fetch`,
you may pass named argument to the corresponding method call: ``update=True``.


.. code-block:: python

    # Selecting all objects with enabled=True and locking them
    # using update=True
    instances = await MyModel.all(enabled=False, update=True)



Fetching several objects by their primary key
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The one special, but often used case is fetching several model
objects from the database by their key (primary key). In this
case, in general, we can write a query like:

.. code-block:: python

    instances = await MyModel.all(MyModel.id.in_(['1', '2', '3']))
    # resulting list will consist of objects with primary key ('id')
    # equals to '1', '2' or '3'... for example

But Wefram offers a simplier way (a little simplification for
this case):

.. code-block:: python

    instances = await MyModel.fetch(['1', '2', '3'])
    # resulting list will consist of objects with primary key
    # equals to '1', '2' or '3'... for example

As we see - we even not cares about primary key column name(s)
in the query. Just calling the method
:py:meth:`~system.ds.orm.model.Model.fetch`
giving primary key values to it as arguments.

Even if we need to fetch objects with **complex primary key**
declared - we can do this using
:py:meth:`~system.ds.orm.model.Model.fetch`` method like
:py:meth:`~system.ds.orm.model.Model.get`
method (described below). Just give not plain values, but
tuple of primary key values instead, in order they been
declared in the model. For example:

.. code-block:: python

    from system import ds


    class MyModel(ds.Model):
        id1 = ds.Column(ds.Integer(), primary_key=True)
        id2 = ds.Column(ds.Integer(), primary_key=True)


    instances = await MyModel.fetch(
        (1, 101), (2, 190), (3, 404)
    )
    # The resulting list will consist of objects with primary
    # key values:
    # (1)  id1 = 1, id2 = 101
    # (2)  id1 = 2, id2 = 190
    # (3)  id1 = 3, id2 = 404



Fetching a single object
^^^^^^^^^^^^^^^^^^^^^^^^

The previos section describes how to fetch objects (rows from the
database) resuling with list of the corresponding model instances
or a complied SQLAlchemy results. The result will always be a
list (array) in that case.

Now, let's fetch a single object. This approach has more variants
to use.

As said before, there are two approachs for this situation: we can
use SQLAlchemy query or we can use Wefram helper methods. This
first one we wantn't to describe here - because it is always best
to read on the official site of the SQLAlchemy. The second will
be reviewed here with several methods to use.

There are two typical situations when we expects only one object
to be returned from the query: when we want to fetch the exactly
object (we know it's primary key value) and when we does not
knows the resulting object's primary key, but knows that only
one object about to be returned (as a maximal special case, let's
say we give ``limit=1`` in the query).


Fetching by primary key value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To do that, we about to use :py:meth:`~system.ds.orm.model.Model.get`
method of the corresponding model:

.. code-block:: python

    ...

    instance = await MyModel.get('5c9466b2-ebc1-4b76-ae89-72b7125815d8')


If the model has complex primary key, consisting of two or more
columns, then the method above calls with all primary key
columns' values, ordered by the corresponding keys been declared
in the corresponding model. For example:

.. code-block:: python

    class ComplexPkModel(ds.Model):
        """ The very useless model, but okay for the example """
        id1 = ds.Column(ds.UUID(), primary_key=True)
        id2 = ds.Column(ds.Integer(), primary_key=True)


    # Lets assume that we want to fetch a single object which
    # has next primary key values:
    # id1 = 5c9466b2-ebc1-4b76-ae89-72b7125815d8
    # id2 = 1005
    instance = await MyModel.get(
        '5c9466b2-ebc1-4b76-ae89-72b7125815d8', 1005
    )

The result will be a single object, **or None** if there is no
object with given primary key. So, Wefram does not raises any
exception when the requested object does not exists in the
database, it returns *None* result instead.


Fetch only first object
^^^^^^^^^^^^^^^^^^^^^^^

There are cases when we querying for the request not by the
object's primary key, but by another filters (or even without
any of them). But the case is that we expect only one object
to be returned, while :py:meth:`~system.ds.orm.model.Model.all`
method returns a list.

Let's assume that we have a query and we except only one
object to be returned or nothing. While using
:py:meth:`~system.ds.orm.model.Model.all` general
case we always will get a list, even empty. Not the best
options for this vase, yea?

As always, let's look an example:

.. code-block:: python

    from system import ds


    class User(ds.Model):
        id = ds.UUIDPrimaryKey()
        login = ds.Column(ds.String(100), nullable=False)
        password = ds.Column(ds.String(255), nullable=False)
        enabled = ds.Column(ds.Boolean(), nullable=False, default=True)


    async def try_to_login(login: str, password: str) -> [User, None]:
        """ We about to query the database searching for
        the given login and password
        """
        users = await User.all(login=login, password=password, enabled=True)
        if users:
            return users[0]
        return None


Here we querying for the user with given login and password,
which is enabled, and checking the resulting list for its
length and returning the first returned user.

Let's do the same query with a little simplification:

.. code-block:: python

    ...

    async def try_to_login(login: str, password: str) -> [User, None]:
        """ We about to query the database searching for
        the given login and password
        """
        return await User.first(login=login, password=password, enabled=True)


The :py:meth:`~system.ds.orm.model.Model.first` method of the
:class:`~system.ds.orm.model.Model` returns only first object
from the result (it even sets ``limit=1`` automatically to not to
fetch more than one row from the database), or returns ``None`` if
none objects with given criteria been found.


Creating the object
^^^^^^^^^^^^^^^^^^^

We will have nothing to fetch if nothing being created, right? So, the
**create** is not less important operation then the **read** one.

To create an object we about to use :py:meth:`~system.ds.orm.model.Model.create`
class method on the corresponding model. The created object will automatically be
assigned to the current default database session.

To create an object defining column's values - just pass their values
within named arguments (*kwargs*) defining the initial values.

Let's look at example:

.. code-block:: python

    from system import ds


    class Contant(ds.Model):
        id = ds.UUIDPrimaryKey()
        first_name = ds.Column(ds.String(100), nullable=False, default='')
        middle_name = ds.Column(ds.String(100), nullable=False, default='')
        last_name = ds.Column(ds.String(100), nullable=False, default='')
        avatar = ds.Column(ds.Image(), nullable=True)
        phone = ds.Column(ds.Phone(), nullable=False)
        email = ds.Column(ds.Email(), nullable=False)


    # Lets create a couple objects for this example model
    contact1 = await Contact.create(
        first_name='Mike',
        last_name='Cooper',
        phone='+71234561234'
    )
    contact2 = await Contact.create(
        first_name='Car service',
        phone='+70001112233'
    )

    # Ooh, we forgot to set email for Mike?
    contact1.email = 'mike.cooper@domain.com'

    # The next call IS NOT required. It is useful only if you
    # need the previously created objects to be flushed to the
    # database. Otherwise them will do that at the web request
    # finish.
    ds.flush()


Updating the existing object
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The **update** operation modifies the existing object to update
its corresponding values in the database. So, first of all, we need
to fetch the corresponding object first.

To modify the object it is best to call the
:py:meth:`~system.ds.orm.model.Model.update` method on
the model instance, and pass new corresponding columns' values
to it within named arguments (*kwargs*).

.. code-block:: python

    from system import ds


    class Contant(ds.Model):
        id = ds.UUIDPrimaryKey()
        first_name = ds.Column(ds.String(100), nullable=False, default='')
        middle_name = ds.Column(ds.String(100), nullable=False, default='')
        last_name = ds.Column(ds.String(100), nullable=False, default='')
        avatar = ds.Column(ds.Image(), nullable=True)
        phone = ds.Column(ds.Phone(), nullable=False)
        email = ds.Column(ds.Email(), nullable=False)


    ...

    # Lets assume that we have made some objects prior to this place

    instance = await ds.get('...')  # We have fetched a single object

    # Now let's update its columns' values
    await instance.update(
        first_name='Another',
        phone=''
    )

    # That's it, we have updated the object. As been said before, we
    # need to fetch the corresponding object(s) first.


Deleting the existing object
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To delete the object, when we have fetched that object before, we just
can call instance method :py:meth:`~system.ds.orm.model.Model.delete`.
This will delete the object from the
database.

Again, the method :py:meth:`~system.ds.orm.model.Model.delete`` is an
instance method, not the class one. This means that this method works
with previously fetched from the database object (using any selecting
method described before).

.. code-block:: python

    # Fetching a single object in this example
    instance = await MyModel.get('...')

    ...

    # And deleing that object
    await instance.delete()



Deleting many objects or without fetching
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

As been described in the section above, to use
:py:meth:`~system.ds.orm.model.Model.delete` method we
have to fetch the corresponding object or objects first. This is the
approach the SQLAlchemy tells us.

This approach has one negative variant - when we need to delete many
objetcs at a time, or even when we just do not want to fetch objects
from the database just ot delete them (why?)

In this case, Wefram offers a class method (*class*, not *instance*)
named :py:meth:`~system.ds.orm.model.Model.delete_where`.
This method waits a clause or filter to be given like for
:py:meth:`~system.ds.orm.model.Model.all` or
:py:meth:`~system.ds.orm.model.Model.first` methods (really the same
syntax).

For example:

.. code-block:: python

    # The first example with SQLAlchemy-based where clause
    await MyModel.delete_where(
        ds.and_(MyModel.email == '', MyModel.phone == '')
    )

    # The second example with filter variant
    await MyModel.delete_where(
        email='', phone=''
    )


.. attention::
    Please note that :py:meth:`~system.ds.orm.model.Model.delete_where`
    is not capable of triggering
    the history (because it uses the fast, direct SQL query like
    DELETE FROM ... WHERE ...) and the history **WILL NOT** be
    written on objects been deleted with this method! So, it is
    really bad idea to delete object with history logging enabled
    using exact this, maybe fast, but logging unfollowed method!



ORM models methods summary
--------------------------

.. autoclass:: system.ds.model.Model
    :members:

