Quickstart — intake_spark 0.0.0+3.gadf091e.dirty documentation

# Quickstart

`intake-spark` provides quick and easy access to data via Apache Spark

## Installation

To use this plugin for intake, install with the following command:

    conda install -c intake intake-spark

## Usage

### Establishing a Context

Operations on a Spark cluster are achieved via a “context”, which is a python-side client to the remote system. All use of this package require a valid context in order to work. There are two ways to establish the context:

- use the function `intake_spark.base.SparkHolder.set_class_session()`, passing for `context=` and `session=` the respective existing objects that have been created previously using conventional means. In this case, Spark connection-specific parameters are not encoded in the catalog. Since intake-spark uses `getOrCreate()`, the existence of the objects should be enough for them to be picked up at data access time, but we still recommend calling the set function explicitly. Note that if only using RDDs and no SQl functionality, the `session` need not be created or provided.
- the same function can also take a set of parameters, in which case intake-spark will attempt to create the context and session for you, passing the parameters (master, app_name, executor_env, spark_home and config parameters) on to Spark. In this case, the parameters can be stored in a catalog (the `context_kwargs` parameter, a dictionary). If providing an empty set of parameters, Spark will create the default local context, which is useful only for testing.

    # in-code example
    from intake_spark.base import SparkHolder
    import intake
    SparkHolder.set_class_session(master='spark://myhost:7077', app_name='intake', hive=True)

### Encoding Spark calls

Spark calls are often expressed as a chain of attribute look-ups with some being called as methods with arguments. To encode these for Intake, we make each stage of the chain an element in a list, starting from the Context or Session for the RDD and DataFrame versions of the driver. For example, the following encodes `sc.textfile('s3://bucket/files*.txt')`, i.e., a single element of attribute lookup

    source = intake.open_spark_rdd([
        ('textFile', ('s3://bucket/files*.txt',))
        ])
    rdd = source.to_spark()

Here `rdd` will be a pySpark RDD instance.

A more complicated example, for encoding `spark.read.option("mergeSchema", "true").parquet("data/test_table")` for use with Intake

    source = intake.open_spark_dataframe([
        ('read', ),
        ('option', ("mergeSchema", "true")),
        ('parquet', ("data/test_table",))
        ])
    df = source.to_spark()

Note that you cannot pass functions using this formalism, only literals.

### Using in a catalog

The above example could be expressed in YAML syntax as follows

    sources:
      spark_dataframe:
        args:
          args:
          - - read
          - - option
            - - mergeSchema
              - 'true'
          - - parquet
            - - data/test_table
          context_kwargs:
            master: "spark://myhost:7077"
            app_name: intake
        description: ''
        driver: spark_dataframe
        metadata: {}

Note the complex nesting pattern, and that in this case we are including arguments for creating an appropriate SparkContext and Session on the fly, if one has not already been made.