
Herman
======
The ORM peewee is currently going through a major 3.0 rewrite and isn't accepting many (any?) new features for the 2.x line.

Because of this, I made Herman as a light fork of peewee (2.8.1) implementing some syntactic sugar and minor tweaks I've liked from other ORMs.

This is mostly for my company's internal use, but anyone else is free to use it.  I'm maintaining API compatibility as a drop in replacement for peewee 2.x.

.. image:: https://api.travis-ci.org/keredson/peewee.png?branch=master
  :target: https://travis-ci.org/keredson/peewee


How To Install
==============

.. code-block:: bash

  sudo pip install git+git://github.com/keredson/peewee.git

Or to upgrade:

.. code-block:: bash

  sudo pip install git+git://github.com/keredson/peewee.git --upgrade


Differences (Herman vs. Peewee)
===============================


SQL Generation
--------------

The SQL generated by an ORM isn't just an artifact to be consumed by the database.  Developers need to be able to read it as well.  The more readable it is and the closer it is to what one would naturally write, the less time it takes to manually evaluate / debug it.

For example, this query:

.. code-block:: python

    Article.select()
    .join(Person, join_type=pw.JOIN.LEFT_OUTER, on=(Article.author==Person.id))
    .where(Article.author==derek)

In Peewee generates this:

.. code-block:: sql

    SELECT "t1"."id", "t1"."title", "t1"."author_id", "t1"."editor_id" 
    FROM "article" AS t1 
    LEFT OUTER JOIN "person" AS t2 ON ("t1"."author_id" = "t2"."id") 
    WHERE ("t1"."author_id" = ?

Whereas Herman generates:

.. code-block:: sql

    SELECT a1.id, a1.title, a1.author_id, a1.editor_id 
    FROM article AS a1 
    LEFT OUTER JOIN person AS p1 ON (a1.author_id = p1.id) 
    WHERE (a1.author_id = ?) 

You'll notice that in Herman:

* Table and column identifiers in Herman aren't unnecessarily quoted.
* Table alias names are derived from their table names, not just "t1..tN".


The first() Method Uses a Limit
-------------------------------

.. code-block:: python

    Article.select().first()

In Peewee generates this:

.. code-block:: sql

    SELECT "t1"."id", "t1"."title", "t1"."author_id", "t1"."editor_id" FROM "article" AS t1

Where in Herman this is generated:

.. code-block:: sql

    SELECT a1.id, a1.title, a1.author_id, a1.editor_id FROM article AS a1 LIMIT 1

Notice the additional "LIMIT 1" in Herman.  Peewee 2.8.1 rather loads entire result set and returns this first object.  (It's my understanding future versions of Peewee will also add a "LIMIT 1".)


Named Table Aliases
-------------------

In Peewee you can create table aliases as a variable outside your query.  For example:

.. code-block:: python

    author_table = Person.alias()
    Article.select()
    .join(author_table)
    .where(author_table.name=='Derek')

This gets you a very respectable:

.. code-block:: sql

    SELECT "t1"."id", "t1"."title", "t1"."author_id", "t1"."editor_id" 
    FROM "article" AS t1
    INNER JOIN "person" AS t2 ON ("t1"."author_id" = "t2"."id") 
    WHERE ("t2"."name" = ?)

But in some queries it's very useful to specify what the aliases are.  In Herman you can do this:

.. code-block:: python

    author_table = Person.alias('author')
    Article.select()
    .join(author_table)
    .where(author_table.name=='Derek')

Which generates this:

.. code-block:: sql

    SELECT a1.id, a1.title, a1.author_id, a1.editor_id 
    FROM article AS a1 
    INNER JOIN person AS author ON (a1.author_id = author.id) 
    WHERE (author."name" = ?)

These two statements are equivalent in Herman:

.. code-block:: python

    author_table = Person.alias('author')
    author_table = Person.as_('author')

And because aliases are named, you're no longer required to use the external variable:

.. code-block:: python

    Article.select()
    .join(Person.as_('author'))
    .where(Person.as_('author').name=='Derek')

Whether in-lining the aliases makes it easier or harder to read is entirely dependent on the query and the code block it's in, but it's good to have the option.  Both are supported in Herman.


Alias References are Inferred When Unambiguous
----------------------------------------------

If a query is otherwise invalid and there is only one possible interpretation of the query author's intent, automatically tie the un-aliased column to the appropriate alias.  For example:

.. code-block:: python

   Person.alias('xyz').select().where(Person.name == 'Derek')

Generates invalid SQL in Peewee (which is run on the database, which throws an exception):

.. code-block:: sql

    SELECT "t1"."id", "t1"."name", "t1"."parent_id"
    FROM "person" AS t1 WHERE ("t2"."name" = ?)
    peewee.OperationalError: no such column: t2.name


But in Herman it's valid:

.. code-block:: sql

    SELECT xyz.id, xyz."name", xyz.parent_id 
    FROM person AS xyz WHERE (xyz."name" = ?)

This would NOT work if more than one Person alias were included in the query.


Herman Raises Exceptions When Invalid Columns Are Referenced
------------------------------------------------------------

Peewee will generate and run on the database SQL it knows is invalid.  For example:

.. code-block:: python

    Person.select().where(Article.title == 'xyz').first()

In Peewee will throw a database error:

.. code-block:: sql

    SELECT "t1"."id", "t1"."name", "t1"."parent_id" 
    FROM "person" AS t1 WHERE ("t2"."title" = ?)
    peewee.OperationalError: no such column: t2.title

This isn't good for two reasons.  First, I don't like relying on the database to catch easily detectable errors for us.  Second, the error is opaque and specific to the internal implementation details of peewee (the "t2").

Herman on the other hand will not generate the SQL at all, instead raising:

.. code-block:: python

    peewee.ProgrammingError: <class '__main__.Article'> is not a part of this query


The get() Method Confirms Uniqueness
------------------------------------

The get() method in Peewee adds a "LIMIT 1" and returns the first object.  I feel this is incorrect behavior.  The difference between first() and get() is get() should assert that only one matching record exists. (This is something Django got right IMO.)  If my query criteria hasn't correctly isolated a unique row the ORM should throw an exception.

This is why Herman added a "LIMIT 2" to the SQL genned from get(), and does a check on the number of object returned.  For example:

.. code-block:: python

    derek = Person.create(name='Derek')
    callie = Person.create(name='Callie')
    Person.select().get()

Will throw:

.. code-block:: sql

    peewee.DataError: Too many instances matching query exist:
    SQL: SELECT p1.id, p1."name", p1.parent_id FROM person AS p1 LIMIT 2

Rather than returning a random Person object selected by the database.


The Shortcut ALL
----------------

In Herman, this:

.. code-block:: python

  Person.ALL

Is the same as this:

.. code-block:: python

  Person.select()

It's just a nomenclature I preferred from Django.  I made it uppercase to prevent conflicts with any columns named "all", and to highlight that it's effectively a constant.


A New (Additional) Join Syntax
------------------------------

If I have a set of models:

.. code-block:: sql

    class Person(pw.Model):
      name = pw.CharField()

    class Article(pw.Model):
      title = pw.CharField()
      author = pw.ForeignKeyField(db_column='author_id', rel_model=Person, to_field='id')
      editor = pw.ForeignKeyField(db_column='editor_id', rel_model=Person, to_field='id', related_name='edited_articles', null=True)

    class Reply(pw.Model):
      text = pw.CharField()
      article = pw.ForeignKeyField(db_column='article_id', rel_model=Article, to_field='id')


And I want to do something fancy like get all replies with their articles and authors and editors, in Peewee I have to do something like this:


.. code-block:: python

  author_table = Person.alias()
  editor_table = Person.alias()
  replies = Reply.select(Reply, author_table, editor_table)
    .join(Article) 
    .join(author_table, join_type=pw.JOIN.LEFT_OUTER, on=(author_table==Article.author)) 
    .switch(Article) 
    .join(editor_table, join_type=pw.JOIN.LEFT_OUTER, on=(editor_table==Article.editor)) 
    .where(author_table.name=="Derek")

Which is all sorts of complicated.  For instance:

* I have to mentally keep track of what the join context is and manipulate it with the switch statement.
* Because Article has two FKs to the same table, I have to manually specify the on conditionals.
* Because an editor can be null, the default INNER JOIN will implicitly filter out replies to articles without editors, which is rarely what the developer wants when asking for a list of replies, so I have to use "join_type=pw.JOIN.LEFT_OUTER" a lot.

Herman offers a simpler syntax:

.. code-block:: python

  Reply.ALL
    .plus(Reply.article, Article.author.as_('author'))
    .plus(Reply.article, Article.editor)
    .where(Person.as_('author').name=="Derek")

Which generates the same SQL:

.. code-block:: sql

  SELECT r1.id, r1."text", r1.article_id, a1.id, a1.title, a1.author_id, a1.editor_id,
         author.id, author."name", author.parent_id, p1.id, p1."name", p1.parent_id 
  FROM reply AS r1 
  LEFT OUTER JOIN article AS a1 ON (r1.article_id = a1.id) 
  LEFT OUTER JOIN person AS author ON (a1.author_id = author.id) 
  LEFT OUTER JOIN person AS p1 ON (a1.editor_id = p1.id) 
  WHERE (author."name" = ?)

The plus() method takes a variable number of ForeignKeyField objects which represent a path away from the primary query object (Reply in this case).  For example:

.. code-block:: python

  Reply.ALL.plus(Reply.article)

Gets all the replies with their associated articles.

.. code-block:: python

  Reply.ALL.plus(Reply.article, Article.editor)

Gets all the replies with their associated articles and all the articles editors.  Note that 
"Reply.article" is a foreign key from Reply to Article, and Article.editor is a foreign key from Article to Person.  The list of foreign keys must create a logical path where the "to" type of one FK matches the "from" type of the next FK.

Following the same path twice is harmless.  For instance:

.. code-block:: python

  Reply.ALL.plus(Reply.article).plus(Reply.article)

Is no different than calling plus() once.  This is why we can specify multiple paths that have some overlap, for example:

.. code-block:: python

  Reply.ALL
    .plus(Reply.article, Article.author)
    .plus(Reply.article, Article.editor)

Which will join on the article table only once.

You can also alias your joined tables with:

.. code-block:: python

  Article.author.as_('author')

Which allows you to reference it later in your conditional:

.. code-block:: python

  .where(Person.as_('author').name=="Derek")

Herman's plus() also supports following foreign keys from one-to-many.  Like:

.. code-block:: python

  Article.ALL.plus(Reply.article)

Which internally does a prefetch to populate the article with all of its replies.  There will be O(k) SQL statements executed, where `k` is the number of two-many relationships.  All of these queries will be grouped into one transaction to guarantee correctness.

*IMPORTANT:*

    Remember that foreign keys represent edges in your object graph, and a call to `plus(*edges)` tells Herman to include that path from the object graph in your query.

This semantics for plus() have been co-opted from the `DKO <https://github.com/keredson/DKO>`_ project, which I authored for my former employer.  DKO's version of this syntax has been in broad production use since 2010 by hundreds of developers, accessing some of the largest (billions of rows) conventional relational databases that exist.

Calling len() Does a Database Count
-----------------------------------

If you call:

.. code-block:: python

  len(Article.select())

Peewee this will load a list of all objects, permanently cache said list, and then call `len()` on that cache.

In Herman this will call `count()` on the database and return the resulting integer.  It does not build the list of objects in python nor cache anything.  However, for backwards compatability, if something else has already populated the cached results of the query, it will call `len()` on that.


A New  DeferredRelation Syntax
------------------------------

The semantics behind Peewee's `circular foreign key dependencies <http://docs.peewee-orm.com/en/latest/peewee/model.html#circular-foreign-key-dependencies>`_ get kind of unwieldy when you have more than a few models (and they're spread over multiple files). This is because the DeferredRelation object has to be defined, used, then the other model defined in another file, then set_model has to be called on the original, and then you're left with the object reference dangling around that has no purpose. IE the example in the docs:

.. code-block:: python

  # Create a reference object to stand in for our as-yet-undefined Tweet model.
  DeferredTweet = DeferredRelation()

  class User(Model):
    username = CharField()
    # Tweet has not been defined yet so use the deferred reference.
    favorite_tweet = ForeignKeyField(DeferredTweet, null=True)
  
  class Tweet(Model):
    message = TextField()
    user = ForeignKeyField(User, related_name='tweets')
  
  # Now that Tweet is defined, we can initialize the reference.
  DeferredTweet.set_model(Tweet)

Ours happens all in the model definition with an optional parameter given to DeferredRelation. Like:

.. code-block:: python

  class User(Model):
    username = CharField()
    # Tweet has not been defined yet so use the deferred reference.
    favorite_tweet = ForeignKeyField(DeferredRelation('Tweet'), null=True)
  
  class Tweet(Model):
    message = TextField()
    user = ForeignKeyField(User, related_name='tweets')


This removes the need for the extra variable in the global namespace and the coordination of it over multiple files. And since the parameter is optional, it is fully backwards-compatible with the old syntax.

Our patch for this has been incorporated upstream, so this is forwards-compatible too, following Peewee's next release.


Passing an Empty List/Set/Tuple into IN Doesn't Gen Invalid SQL
---------------------------------------------------------------

If you try to do a IN operation on an empty list:

.. code-block:: python

  User.select().where(User.id << [])

Peewee will generate the following SQL:

.. code-block:: sql

  SELECT "t1"."id", "t1"."username" FROM "user" AS t1 WHERE ("t1"."id" IN ())

Which the database will reject as invalid, throwing an exception.  We instead generate a "false" statement:

.. code-block:: sql

  SELECT u1.id, u1.username FROM "user" AS u1 WHERE (0 = 1)

So you don't have to manually test for empty lists every time you use a SQL IN.



peewee
======

Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use.

* A small, expressive ORM
* Written in python with support for versions 2.6+ and 3.2+.
* built-in support for sqlite, mysql and postgresql
* tons of extensions available in the `playhouse <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html>`_

  * `Postgresql HStore, JSON, arrays and more <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#postgres-ext>`_
  * `SQLite full-text search, user-defined functions, virtual tables and more <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#sqlite-ext>`_
  * `Schema migrations <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#migrate>`_ and `model code generator <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pwiz>`_
  * `Connection pool <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pool>`_
  * `Encryption <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#sqlcipher-ext>`_
  * `and much, much more... <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html>`_

New to peewee? Here is a list of documents you might find most helpful when getting
started:

* `Quickstart guide <http://docs.peewee-orm.com/en/latest/peewee/quickstart.html#quickstart>`_ -- this guide covers all the essentials. It will take you between 5 and 10 minutes to go through it.
* `Guide to the various query operators <http://docs.peewee-orm.com/en/latest/peewee/querying.html#query-operators>`_ describes how to construct queries and combine expressions.
* `Field types table <http://docs.peewee-orm.com/en/latest/peewee/models.html#field-types-table>`_ lists the various field types peewee supports and the parameters they accept.

For flask helpers, check out the `flask_utils extension module <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#flask-utils>`_. You can also use peewee with the popular extension `flask-admin <http://flask-admin.readthedocs.org/en/latest/>`_ to provide a Django-like admin interface for managing peewee models.

Examples
--------

Defining models is similar to Django or SQLAlchemy:

.. code-block:: python

    from peewee import *
    from playhouse.sqlite_ext import SqliteExtDatabase
    import datetime

    db = SqliteExtDatabase('my_database.db')

    class BaseModel(Model):
        class Meta:
            database = db

    class User(BaseModel):
        username = CharField(unique=True)

    class Tweet(BaseModel):
        user = ForeignKeyField(User, related_name='tweets')
        message = TextField()
        created_date = DateTimeField(default=datetime.datetime.now)
        is_published = BooleanField(default=True)

Connect to the database and create tables:

.. code-block:: python

    db.connect()
    db.create_tables([User, Tweet])

Create a few rows:

.. code-block:: python

    charlie = User.create(username='charlie')
    huey = User(username='huey')
    huey.save()

    # No need to set `is_published` or `created_date` since they
    # will just use the default values we specified.
    Tweet.create(user=charlie, message='My first tweet')

Queries are expressive and composable:

.. code-block:: python

    # A simple query selecting a user.
    User.get(User.username == 'charles')

    # Get tweets created by one of several users. The "<<" operator
    # corresponds to the SQL "IN" operator.
    usernames = ['charlie', 'huey', 'mickey']
    users = User.select().where(User.username << usernames)
    tweets = Tweet.select().where(Tweet.user << users)

    # We could accomplish the same using a JOIN:
    tweets = (Tweet
              .select()
              .join(User)
              .where(User.username << usernames))

    # How many tweets were published today?
    tweets_today = (Tweet
                    .select()
                    .where(
                        (Tweet.created_date >= datetime.date.today()) &
                        (Tweet.is_published == True))
                    .count())

    # Paginate the user table and show me page 3 (users 41-60).
    User.select().order_by(User.username).paginate(3, 20)

    # Order users by the number of tweets they've created:
    tweet_ct = fn.Count(Tweet.id)
    users = (User
             .select(User, tweet_ct.alias('ct'))
             .join(Tweet, JOIN.LEFT_OUTER)
             .group_by(User)
             .order_by(tweet_ct.desc()))

    # Do an atomic update
    Counter.update(count=Counter.count + 1).where(
        Counter.url == request.url)

Check out the `example app <http://docs.peewee-orm.com/en/latest/peewee/example.html>`_ for a working Twitter-clone website written with Flask.

Learning more
-------------

Check the `documentation <http://docs.peewee-orm.com/>`_ for more examples.

Specific question? Come hang out in the #peewee channel on irc.freenode.net, or post to the mailing list, http://groups.google.com/group/peewee-orm . If you would like to report a bug, `create a new issue <https://github.com/coleifer/peewee/issues/new>`_ on GitHub.

Still want more info?
---------------------

.. image:: http://media.charlesleifer.com/blog/photos/wat.jpg

I've written a number of blog posts about building applications and web-services with peewee (and usually Flask). If you'd like to see some real-life applications that use peewee, the following resources may be useful:

* `Building a note-taking app with Flask and Peewee <http://charlesleifer.com/blog/saturday-morning-hack-a-little-note-taking-app-with-flask/>`_ as well as `Part 2 <http://charlesleifer.com/blog/saturday-morning-hacks-revisiting-the-notes-app/>`_ and `Part 3 <http://charlesleifer.com/blog/saturday-morning-hacks-adding-full-text-search-to-the-flask-note-taking-app/>`_.
* `Analytics web service built with Flask and Peewee <http://charlesleifer.com/blog/saturday-morning-hacks-building-an-analytics-app-with-flask/>`_.
* `Personalized news digest (with a boolean query parser!) <http://charlesleifer.com/blog/saturday-morning-hack-personalized-news-digest-with-boolean-query-parser/>`_.
* `Using peewee to explore CSV files <http://charlesleifer.com/blog/using-peewee-to-explore-csv-files/>`_.
* `Structuring Flask apps with Peewee <http://charlesleifer.com/blog/structuring-flask-apps-a-how-to-for-those-coming-from-django/>`_.
* `Creating a lastpass clone with Flask and Peewee <http://charlesleifer.com/blog/creating-a-personal-password-manager/>`_.
* `Building a web-based encrypted file manager with Flask, peewee and S3 <http://charlesleifer.com/blog/web-based-encrypted-file-storage-using-flask-and-aws/>`_.
* `Creating a bookmarking web-service that takes screenshots of your bookmarks <http://charlesleifer.com/blog/building-bookmarking-service-python-and-phantomjs/>`_.
* `Building a pastebin, wiki and a bookmarking service using Flask and Peewee <http://charlesleifer.com/blog/dont-sweat-small-stuff-use-flask-blueprints/>`_.
* `Encrypted databases with Python and SQLCipher <http://charlesleifer.com/blog/encrypted-sqlite-databases-with-python-and-sqlcipher/>`_.
* `Dear Diary: An Encrypted, Command-Line Diary with Peewee <http://charlesleifer.com/blog/dear-diary-an-encrypted-command-line-diary-with-python/>`_.
* `Query Tree Structures in SQLite using Peewee and the Transitive Closure Extension <http://charlesleifer.com/blog/querying-tree-structures-in-sqlite-using-python-and-the-transitive-closure-extension/>`_.
