Metadata-Version: 2.1
Name: pysparrow
Version: 1.0.4
Summary: An arrow interface for PySpark RDDs
Home-page: https://github.com/petereon/sparrow
License: MIT
Keywords: spark,arrow,>>,map,mappers
Author: Peter Vyboch
Author-email: pvyboch1@gmail.com
Requires-Python: >=3.6,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: Software Development :: Libraries
Classifier: Typing :: Typed
Requires-Dist: pyspark (>=3.2.1,<4.0.0)
Project-URL: Repository, https://github.com/petereon/sparrow
Description-Content-Type: text/markdown

# Sparrow

Sparrow (a combination of _Spark_ and _arrow_) is a Python mini library that enhances Spark with an _arrow API_.

Intent is to make mappers and filters over RDD a bit more elegant and exciting. Author also feels that here developed API does have a more consitent feel.

Consider and example of few operations on an RDD in native PySpark
```python
...
rdd = spark.sparkContext.parallelize(
        [
            (1, 2.0, ["a", "b", "c"]),
            (2, 3.0, ["b", "c", "d"]),
            (3, 4.0, ["c", "d", "e"]),
            (4, 5.0, ["d", "e", "f"]),
            (5, 6.0, ["e", "f", "g"]),
        ]
    )
    
res = rdd.map(lambda x: x[2]).flatMap(lambda x: x).filter(lambda x: x == 'b')

```
and then on RDD extended with Sparrow:
```python
rdd = spark.sparkContext.parallelize(
        [
            (1, 2.0, ["a", "b", "c"]),
            (2, 3.0, ["b", "c", "d"]),
            (3, 4.0, ["c", "d", "e"]),
            (4, 5.0, ["d", "e", "f"]),
            (5, 6.0, ["e", "f", "g"]),
        ]
    )

res = (
    SparrowRDD(rdd) 
    >> (lambda x: x[2]) 
    >> Flatten(lambda x: x)
    >> Filter(lambda x: x == 'b')
)
```

