{
 "metadata": {
  "name": "",
  "signature": "sha256:63b9ccd54376821644c4dfaeb6aca56a28549904f30120be2b160e8764863ffb"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Install the development package directly from GitHub with:\n",
      "\n",
      "----\n",
      "\n",
      "\n",
      "\n",
      ">pip install git+ssh://github.com/paradigm4/scidb-py.git\n",
      "\n",
      "    \n",
      "\n",
      "    "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "## Connecting to SciDB\n",
      "\n",
      "\n",
      "This step requires a connection to a SciDB cluster. This notebook assumes SciDB is available at http://localhost:8080. Replace this address with one that points to a SciDB connection.\n",
      "\n",
      "\n",
      "\n",
      "Ready-to-roll SciDB AMIs for Amazon EC2 are available. Information is available on the SciDB forum at http://scidb.org/forum."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "import numpy as np\n",
      "\n",
      "from scidbpy import connect, SciDBQueryError, SciDBArray\n",
      "\n",
      "sdb = connect('http://localhost:8080')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Creating SciDB array objects...\n",
      "\n",
      "\n",
      "####...from a numpy array:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "X = np.random.random((5, 4))\n",
      "\n",
      "X_sci = sdb.from_array(X)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# SciDBArray objects provide a mix of familiar numpy things and some SciDB-specific things:\n",
      "\n",
      "X_sci.shape      # The usual shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "X_sci.name       # SciDB array name"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Let's list all the SciDB arrays in the database:\n",
      "\n",
      "sdb.list_arrays().keys()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "####... using an array generating function\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "# Create a 10x10 array of double-precision zeros:\n",
      "A = sdb.zeros( (10,10,) )\n",
      "\n",
      "# Create a 10x10 array of 64-bit signed integer ones:\n",
      "B = sdb.ones( (10,10), dtype='int64' )\n",
      "\n",
      "# Create a 10x10 array of numbers between -1 and 2 (inclusive) sampled from a uniform random distribution.\n",
      "C = sdb.random( (10,10), lower=-1, upper=2)\n",
      "\n",
      "# Create a vector of 5 equally spaced numbers between 1 and 10, including the endpoints:\n",
      "D = sdb.linspace(1,10,num=5,endpoint=True)\n",
      "\n",
      "# Create a 10x10 sparse, double-precision-valued identity matrix:\n",
      "E = sdb.identity(10, dtype='double', sparse=True)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### ...from a SciDB query\n",
      "\n",
      "\n",
      "\n",
      "The scidbpy package query method simplifies writing low-level SciDB queries involving SciDBArray objects."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tridiag = sdb.new_array((10, 10))\n",
      "\n",
      "sdb.query('store(build_sparse({A}, \\\n",
      "                 iif({A.d0}={A.d1}, 2, -1), \\\n",
      "                 {A.d0} <= {A.d1}+1 and {A.d0} >= {A.d1}-1), \\\n",
      "           {A})', A=tridiag)\n",
      "\n",
      "tridiag.nonempty()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tridiag.toarray()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### ...from an existing SciDB array\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Create an array directly in SciDB via the query interface:\n",
      "sdb.query(\"store(build(<v:double>[i=1:10,5,0,j=1:5,5,0],i+j),Z)\")\n",
      "\n",
      "# Now associate that array with a SciDBArray object in our Python session:\n",
      "Z = sdb.wrap_array(scidbname=\"Z\")\n",
      "Z"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**The scope of SciDB arrays connected to SciDBArray objects**\n",
      "\n",
      "----\n",
      "\n",
      "The `persistent` argument of the `new_array` method determines if a SciDB array should be removed when its corresponding Python reference falls out of scope."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "X = sdb.random((10,10))\n",
      "xname = X.name\n",
      "xname"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Let's delete X from the database:\n",
      "X.reap()\n",
      "\n",
      "# and check to see if its array is still in SciDB:\n",
      "sdb.list_arrays().has_key(xname)\n",
      "\n",
      "# Note: Temporary arrays are automatically reaped when python exits"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Materializing data to Python**\n",
      "\n",
      "----"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Materialize SciDB array to Python as a numpy array:\n",
      "tridiag.toarray()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Materialize SciDB array to Python as a sparse array:\n",
      "from scipy import sparse\n",
      "tridiag.tosparse('csr')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Operations on `SciDBArray` objects\n",
      "\n",
      "\n",
      "### Subarrays\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Define a 3x10 subarray (returned as a new SciDBArray object)\n",
      "X = tridiag[2:5,:]\n",
      "X.toarray()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "X = tridiag[2:5,::2]\n",
      "X.toarray()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Data aggregation\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tridiag.sum()[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tridiag.var()[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print C.min(0).toarray()  # Aggregate minimum over rows for each column\n",
      "print C.min(1).toarray()  # Aggregate minimum over columns for each row"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "...also `min` `max` `std` `mean` `approxdc` and `count`."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Pointwise application of scalar functions\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Compare Python and SciDB\n",
      "sin_tri = sdb.sin(tridiag)\n",
      "np.linalg.norm(sin_tri.toarray() - np.sin(tridiag.toarray()))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Shape and layout functions\n",
      "\n",
      "----"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tri_reshape = tridiag.reshape((20,5))\n",
      "print tri_reshape.shape\n",
      "print tri_reshape.transpose().shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Arithmetic\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "X = sdb.from_array(np.random.random((10,10)))\n",
      "Y = sdb.from_array(np.random.random((10,10)))\n",
      "\n",
      "\n",
      "S = X + Y\n",
      "D = X - Y\n",
      "M = 2 * X\n",
      "(S + D - M).sum()[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sdb.dot(X,Y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sdb.dot(X.T, Y[:,1]).toarray()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}