.. include:: <s5defs.txt>

----------------------------------------
Content Mirroring / Relational Databases
----------------------------------------

:Author: Kapil Thangavelu
:Contact: kapilt@gmail.com
:Date: 14 May 2009
:Location: Plone Symposium Sorrento

Who am I
--------

Zope / Python / Plone / Django / Pony Consultant last 10 years, residing in DC.

For the past year, I've been primarily a Zope3 / Django Consultant using relational databases.

.. haven't done plone in two years.

.. iterate, plonepas

.. Last week, porting Zope3 to Google AppEngine ( Lighting Talk Tomorrow)

.. thank abstract.it

.. All of these engagements have been utilizing relational databases.

.. The alchemist toolkit, contains package for constructing zope3 applications, quickly with reusable components for user interface (crud and ajax), security, traversal.

.. a happy experience.


Talk
----

Deploying Content to relational databases

Not about using databases in Plone, but about databases using Plone.

Product vs. Framework
---------------------

2003, First Plone conference NOLA, paul everitt posed the question.

.. if plone was a product or a framework? arguing that in order to succeed plone needed to be a great product.



The Product Won
---------------

Plone does alot of out the box, with alot of knobs and customization possible through the web.


.. but whats the impact for development.


More Complexity
---------------

- Features are great if your using them.
- Size Matters
- Orthogonality Maters

.. Plone has incorporated, lots of sensible end user functionality, but these amount to application policies and infrastructure and management of persistent state that need to be handled for customization.

.. public facing web front ends deployments don't nesc. need that policy.

.. not product related, but the incorporation of zope3, multiple redundant ways of doing things, components in many different styles. 

.. all while deeping the learning curve.


Complexity Measured
-------------------

"I have measured out my life in coffee spoons", The Love song of J Alrefd Prufrock, TS. Eliot

(1m x 40r x 5d x 50w )/60 = 166h

The Plone Tax, aka time spent waiting on restarts


Simplicity is Good
------------------

- Developer Productivity and the Learning Curve Matters 
- Simple things have a better chance of being correct

.. pay for what you use

Using Plone
-----------

`- as a content management system Product`

`- simplify deployment by removing plone from the deployment equation.`

`- build a front end in any language / framework you choose.`

`- the question changes from how do i do xyz in plone, to what do i want to do?`

`- make it simple, make it scale`

.. caching is not the only way to scale, customizing templates (maintenance) is not the only way to make front end changes.

.. minimize your maintenance costs (ie. upgrades and customization) of Plone.


Separating Deployment
---------------------

2am December 6th 2002, Amsterdam, infrae zope3 sprint

.. alan and i talk at a internet cafe about deployment, which lead to


Static Deployment
-----------------

CMFDeployment - Flexible Static Deployment, Filtering, transforms, deployment skins, rendering, url
resolution. 

http://plone.org/products/cmfdeployment

From Plone 1.0 - Plone 3.0 (3.2?)


Data Deployment
---------------

Entransit, provides data deployment, with features like rollback in a workspace based site structure.

.. requires particular site structure/layout according to workgroups, provides advanced features like rollback of a deployment.


Too Complex
-----------

Still too complex, needs to work out of the box.

operation and setup need to be dirt simple regardless of the internal complexity (like a product ;-)

Content Mirror
--------------

"A facility for mirroring the content of a Plone site into a structured external datastore."

`- automated, transparent, synchronous serialization.`

`- default source, Archetypes Content`
  
`- default target, Relational Databases`

.. at least in the first release, extensible to asynchronous lazy sync
.. Archetypes provides uniform mechanism of addressing content

Schema Transformation
---------------------

`- transform archetypes schemas into relational tables`

`- Using SQLAlchemy`

`- Uses a default schema ( dublin core, relations, files, identity )`

`- each type gets its own table with common fields factored out`

Schema Transformation
---------------------

.. image:: ui/images/MirrorTransform.png

Peer Based
----------

Each content type's class has a corresponding, database persisted class (orm mapped), with corresponding attributes. The peer/domain class can be specified or autogenerated (default).

.. image:: ui/images/MirrorPeer.png

.. to create a content object, we create a peer, and copy the attributes
.. for example to modify a content, we fetch the peer, and copy the attributes
.. to delete an object we fetch its peer and delete it

Bootstrapping
-------------

 - Load the Archetypes
 - Transform to Database Schema
 - Create Peer  
 
.. image:: ui/images/MirrorBootstrap.png


Event Based
-----------

In order to serialize content as changes are happening in the CMS, we integrate into the application server's event stream and subscribe to content events. The subscribers in turn create sync operations via an operation factory.

.. container:: animation

   .. image:: ui/images/MirrorEntry.png

   .. image:: ui/images/MirrorSerialize.png
      :class: incremental

.. A typical problem in Plone at least, is that redundant operations and events are fairly common, as well outright spurious events from facilities like portal_factory. To combat that we aggregate events on transaction boundaries, and automatically collapse multiple operations for the same object.

.. Lifecycle Events transformed as synchronization operations, (Command Pattern). Invoke serializers to perform actual content to peer synchronization.

Frontends
---------

 - pylne - Pylons
 
 - plango - Django ( pony powered )

 - repoze.bfg 

 - appengine 

So now what
-----------
 
  $$ Profit $$

Resources
---------
 
- http://code.google.com/p/contentmirror/

- http://groups.google.com/group/contentmirror

Sprint
------

- Async ( SQS / AQMP )
 
- Audit Logs

- Reporting / Graphics ( maybe )

if your using zope3
-------------------
 - alchemist.ui - container views (listing, json w/ sorting, paging), content views
 - alchemist.catalyst - generated views, security, and more
 - ore.workflow - state machine workflows
 - ore.xapian - full text indexing
 - ore.yuiwidget ( Rich Text Editing, LiveSearch, DateTime, DataTable, Tabs, Async Forms )







 

