.. _examples-label:

*************
Examples
*************

DataFrame-like Patterns
=======================

DataFrames (e.g. from ``pandas``) are a useful abstraction for working with tabular data. While arkouda does not yet have an actual ``DataFrame`` class, it is possible to do many of the same operations. Here, we will create and use a pseudo-DataFrame: a ``dict`` of named ``pdarray`` objects, which are analogous to columns of a DataFrame. Let the following represent transactions in which a userID purchased an item on a particular day for a certain amount of money:

.. code-block:: python

   >>> userName = ak.array(['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Alice'])
   >>> userID = ak.array([111, 222, 111, 333, 222, 111])
   >>> item = ak.array([0, 0, 1, 1, 2, 0])
   >>> day = ak.array([5, 5, 6, 5, 6, 6])
   >>> amount = ak.array([0.5, 0.6, 1.1, 1.2, 4.3, 0.6])
   >>> data = {'userName': userName, 'userID': userID,
               'item': item, 'day': day, 'amount': amount}

Selection
---------
   
The ``df.loc[condition]`` syntax is useful for selecting subsets of data by value and can be emulated in arkouda. For example, here we select all transactions involving user ``111`` of an amount less than ``1.0``:

.. code-block:: python

   >>> condition = (data['userID'] == 111) & (data['amount'] < 1.0)
   >>> u1 = {col: a[condition] for col, a in data.items()}
   >>> u1
   {'userName': array(['Alice', 'Alice']),
    'userID': array([111, 111]),
    'item': array([0, 0]),
    'day': array([5, 6]),
    'amount': array([0.5, 0.59999999999999998])}
   
Description
-----------

.. code-block:: python

   >>> ak.value_counts(data['day'])
   (array([5, 6]), array([3, 3]))
   >>> ak.histogram(data['amount'], 10)
   array([3, 2, 0, 0, 0, 0, 0, 0, 0, 1])

Grouping
--------

In Pandas, groupby-aggregate is a very useful pattern that can be computationally intensive. Arkouda supports grouping by key and most aggregations in Pandas. Note that, because arkouda does not yet have a true ``DataFrame`` class, the arkouda ``GroupBy`` operation does not conform to the ``pandas`` API.

Here we group the data by item and get the number of unique users who bought the item, and the total revenue generated by the item.

.. code-block:: python

   >>> byItem = ak.GroupBy(data['item'])
   >>> byItem.nunique(data['userID'])
   (array([0, 1, 2]), array([2, 2, 1]))
   >>> byItem.sum(data['amount'])
   (array([0, 1, 2]),
    array([1.7000000000000002, 2.2999999999999998, 4.3000000000000007]))

Integration with Pandas
-----------------------

Often, it is useful to load data in arkouda and bring back a small subset of the data to explore further in Pandas. This can be done as long as each column is less than ``arkouda.maxTransferBytes`` in size (default 1 GB).

.. code-block:: python

   # Assume some filtering takes place here
   >>> subset = data
   >>> df = pd.DataFrame({col: a.to_ndarray() for col, a in subset.items()})
   >>> df
      amount  day  item  userID userName
      0     0.5    5     0     111    Alice
      1     0.6    5     0     222      Bob
      2     1.1    6     1     111    Alice
      3     1.2    5     1     333    Carol
      4     4.3    6     2     222      Bob
      5     0.6    6     0     111    Alice

Graphs
======

Arkouda can be used for constructing and performing basic analysis of graphs.

Consider the following arkouda code (from ``toys/ak_rmat.py``), which generates an RMAT graph:

.. code-block:: python

    def gen_rmat_edges(lgNv, Ne_per_v, p, perm=False):
	# number of vertices
	Nv = 2**lgNv
	# number of edges
	Ne = Ne_per_v * Nv
	# probabilities
	a = p
	b = (1.0 - a)/ 3.0
	c = b
	d = b
	# init edge arrays
	ii = ak.ones(Ne,dtype=ak.int64)
	jj = ak.ones(Ne,dtype=ak.int64)
	# quantites to use in edge generation loop
	ab = a+b
	c_norm = c / (c + d)
	a_norm = a / (a + b)
	# generate edges
	for ib in range(1,lgNv):
	    ii_bit = (ak.randint(0,1,Ne,dtype=ak.float64) > ab)
	    jj_bit = (ak.randint(0,1,Ne,dtype=ak.float64) > (c_norm * ii_bit + a_norm * (~ ii_bit)))
	    ii = ii + ((2**(ib-1)) * ii_bit)
	    jj = jj + ((2**(ib-1)) * jj_bit)
	# sort all based on ii and jj using coargsort
	# all edges should be sorted based on both vertices of the edge
	iv = ak.coargsort((ii,jj))
	# permute into sorted order
	ii = ii[iv] # permute first vertex into sorted order
	jj = jj[iv] # permute second vertex into sorted order
	# to premute/rename vertices
	if perm:
	    # generate permutation for new vertex numbers(names)
	    ir = ak.argsort(ak.randint(0,1,Nv,dtype=ak.float64))
	    # renumber(rename) vertices
	    ii = ir[ii] # rename first vertex
	    jj = ir[jj] # rename second vertex
	#
	# maybe: remove edges which are self-loops???
	#    
	# return pair of ndarrays
	return (ii,jj)

Here we generate a random-looking edge-list representing one million vertices and about 10 million edges

.. code-block:: python

   >>> src, dst = gen_rmat_edges(20, 10, 0.01, True)

Calculate out degrees using GroupBy:

.. code-block:: python

   >>> bySrc = ak.GroupBy(src)
   >>> srcID, outDeg = bySrc.count()

Breadth first search is relatively straightforward to implement using :ref:`setops-label`. This example is from ``toys/ak_bfs_conn_comp.py``.

.. code-block:: python
		
    # src and dst pdarrays hold the edge list
    # seeds pdarray with starting vertices/seeds
    def bfs(src,dst,seeds,printLayers=False):
	# holds vertices in the current layer of the bfs
	Z = ak.unique(seeds)
	# holds the visited vertices
	V = ak.unique(Z) # holds vertices in Z to start with
	# frontiers
	F = [Z]
	while Z.size != 0:
	    if printLayers:
		print("Z.size = ",Z.size," Z = ",Z)
	    fZv = ak.in1d(src,Z) # find src vertex edges 
	    W = ak.unique(dst[fZv]) # compress out dst vertices to match and make them unique
	    Z = ak.setdiff1d(W,V) # subtract out vertices already visited
	    V = ak.union1d(V,Z) # union current frontier into vertices already visited
	    F.append(Z)
	return (F,V)

Now we do a breadth-first search from the first vertex:

.. code-block:: python

   >>> layers, visited = bfs(src, dst, ak.array([src[0]]))
   >>> [l.size for l in layers]
   [1, 1, 2056, 42584, 410889, 24146, 2, 0]
   >>> visited.size
   479679

From this we see the number of new vertices in each frontier, as well as the total number of vertices reachable from the seed.
