CrumblPy

# CrumblPy: Examples and Tutorials¶

## Table of Contents¶

- Installation
- Import
- SnowflakeToolKit
  - Snowflake Connection
  - Fetch Data
  - Write Data
  - Execute Query (DML)
- SlackToolKit
  - Slack Connection
  - Post Message
  - Post Reply
  - Post File
  - Push Notification
- Gmail

## Installation¶

In [ ]:

    pip install crumblpy

## Import the package¶

In [2]:

    from CrumblPy import SnowflakeToolKit, SlackToolKit, send_gmail

## SnowflakeToolKit¶

### Snowflake Connection¶

In [3]:

    # Configure the Snowflake connection

    # Option 1: Use environment variables (os.environ)
    sf = SnowflakeToolKit()

    # Option 2: Hard-code the credentials (not recommended)
    # sf = SnowflakeToolKit(user='your_username', password='your_password', role='your_role')

    # # Option 3: Prefect Secrets (deployment only)
    # sf = SnowflakeToolKit(prefect=True)

### Fetch Data¶

In [4]:

    # Fetch data from Snowflake
    sf.fetch_data(
        """
        SELECT * FROM STORES limit 10;
        """
    )

Out[4]:

| STORE_ID | BRAND_ID | WHEN_I_WORK_LOCATION_ID | PUSH_OPERATIONS_ID | SQUARE_LOCATION_ID | UBER_EATS_STORE_ID | GRUB_HUB_MERCHANT_ID | DOORDASH_NAME | NAME | STATE | ... | WEEKLY_MINI_PRICE | MINI_PRICE | MAXIMUM_COOKIES_PER_DAY | MINIMUM_LEAD_TIME_IN_HOURS | CATERING_COOKIE_IDS | SQUARE_ID | EVENT_NAME | PROCESSED_AT | UPDATED_AT | CREATED_AT
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | e0628b10-74f8-11ee-8d9f-f38f32e60e86:Store | b92e9010-74e6-11ed-9518-ab265dbfb691:Brand | NaN | 25639 | e0628b10-74f8-11ee-8d9f-f38f32e60e86:Store | 52f62611-38f2-5fcf-8518-a672693c1602 | None | Crumbl Canada | Nepean | Ontario | ... | 299.0 | 319.0 | NaN | NaN | None | None | MODIFY | 2024-10-04 19:05:53.570 | 2024-10-04 19:05:49 | 2023-10-27 18:44:36
1 | d6621dc0-e696-11ec-b408-e37cce0087a4:Store | b92e9010-74e6-11ed-9518-ab265dbfb691:Brand | NaN | None | None | None | None | None | Garden City | New York | ... | NaN | NaN | NaN | NaN | None | None | MODIFY | 2024-10-04 19:05:59.799 | 2024-10-04 19:05:49 | 2022-06-07 19:20:05
2 | 2b963950-a8d1-11ed-86d0-f7fefe7d0de0:Store | b92e9010-74e6-11ed-9518-ab265dbfb691:Brand | NaN | None | None | None | None | None | Wheat Ridge | Colorado | ... | NaN | NaN | NaN | NaN | None | None | MODIFY | 2024-10-23 17:00:59.028 | 2024-10-23 17:00:56 | 2023-02-09 23:26:25
3 | 6a7e55e0-7fd8-11eb-84bd-ab163f932224:Store | b92e9010-74e6-11ed-9518-ab265dbfb691:Brand | 5112727.0 | None | 6a7e55e0-7fd8-11eb-84bd-ab163f932224:Store | a01f9a90-83e8-5b1e-b479-4c700fc20877 | None | CRUMBL (HILLSBORO) | Hillsboro | Oregon | ... | 249.0 | 299.0 | 200.0 | 48.0 | None | None | MODIFY | 2024-10-06 06:01:52.087 | 2024-10-06 06:00:51 | 2021-03-08 06:35:04
4 | 3ff382f0-4c1d-11ee-9d49-6d7f8c129d9e:Store | b92e9010-74e6-11ed-9518-ab265dbfb691:Brand | NaN | None | None | None | None | None | Edgewater | New Jersey | ... | NaN | NaN | NaN | NaN | None | None | MODIFY | 2024-10-04 19:06:00.012 | 2024-10-04 19:05:49 | 2023-09-05 18:51:40
5 | 625b8e70-0c39-11ed-98e1-291a683e694a:Store | b92e9010-74e6-11ed-9518-ab265dbfb691:Brand | 5446711.0 | None | 625b8e70-0c39-11ed-98e1-291a683e694a:Store | ff077d6f-e754-5c00-9db7-1275d60a4c42 | None | Crumbl (Norwalk) | Norwalk | Connecticut | ... | 249.0 | 299.0 | 1000.0 | 36.0 | None | None | MODIFY | 2024-10-04 19:05:57.452 | 2024-10-04 19:05:49 | 2022-07-25 16:46:52
6 | 113ac8a0-50e6-11ef-897e-39d4e6071072:Store | be2c5570-74e6-11ed-a722-fb8c9888066e:Brand | NaN | None | 113ac8a0-50e6-11ef-897e-39d4e6071072:Store | None | None | None | Preston | Idaho | ... | NaN | NaN | NaN | NaN | None | None | MODIFY | 2024-10-04 19:06:00.781 | 2024-10-04 19:05:49 | 2024-08-02 15:44:13
7 | 71d48e70-ae22-11ed-97e0-b3effe7779ef:Store | b92e9010-74e6-11ed-9518-ab265dbfb691:Brand | NaN | None | None | None | None | None | Grove St. Station | New Jersey | ... | NaN | NaN | NaN | NaN | None | None | MODIFY | 2024-10-16 16:12:48.937 | 2024-10-16 16:12:45 | 2023-02-16 17:50:48
8 | 16f5f4d0-513f-11ec-aaed-ab35c35632a6:Store | b92e9010-74e6-11ed-9518-ab265dbfb691:Brand | NaN | None | None | None | None | None | Northgate | Washington | ... | NaN | NaN | NaN | NaN | None | None | MODIFY | 2024-10-04 19:05:57.836 | 2024-10-04 19:05:49 | 2021-11-29 18:06:35
9 | 6291c400-b219-11ed-9811-e5c137e4de84:Store | b92e9010-74e6-11ed-9518-ab265dbfb691:Brand | NaN | None | None | None | None | None | West Orange | New Jersey | ... | NaN | NaN | NaN | NaN | None | None | MODIFY | 2024-10-04 19:05:52.839 | 2024-10-04 19:05:49 | 2023-02-21 18:56:01

10 rows × 66 columns

### Write Data¶

In [5]:

    table_to_write = sf.fetch_data(
        """
        SELECT COOKIE_ID, NAME FROM COOKIES limit 10;
        """
    )
    table_to_write

Out[5]:

| COOKIE_ID | NAME
--- | --- | ---
0 | 9b2ba8a0-9875-11ef-a16e-abc1c2cf6cf5:Cookies | Chocolate Peanut Butter Sandwich
1 | e72f42b0-c4f1-11ed-a893-2158bb08e2f1:Cookies | Apple Pie
2 | dfbe4a70-e2ed-11ed-b65e-4bac356c9d69:Cookies | Chocolate Chip Test - V1-3
3 | 37b52280-8fbe-11ee-961e-e1a804ad307a:Cookies | Apple Sweet Pie
4 | 8e90dd00-a0a1-11ec-adcf-a5314440ec47:Cookie | Dirt Cake ft. OTTO
5 | 77eab600-84c1-11ef-b69d-eb08e677efd8:Cookies | Holiday Spiced Butter Cake
6 | ed449980-72bf-11ee-a2b7-319903869c9c:Cookies | Buckeye Brownie*
7 | c93898e0-63fd-11ef-acc7-8b24d42a682d:Cookies | Cookie Dough Bits
8 | 854a3ee0-7e6b-11ee-bd83-59656f65a976:Cookies | Chocolate Sprinkle Blondie
9 | d3c9b120-27e8-11ed-a43a-21c73e66cda7:Cookies | R&D Recipe Builder

In [6]:

    # Write data to Snowflake
    sf.insert_data(
        df = table_to_write,
        table_name = 'CRUMBLPY_DEV',
        auto_create_table=True # May not the best practice in production. The table better be created manually.
    )

In [ ]:

    # Check the data we just inserted
    sf.fetch_data(
        """
        SELECT * FROM DATA_SCIENCE.CRUMBLPY_DEV;
        """
    )

Out[ ]:

| COOKIE_ID | NAME
--- | --- | ---
0 | e72f42b0-c4f1-11ed-a893-2158bb08e2f1:Cookies | Apple Pie
1 | dfbe4a70-e2ed-11ed-b65e-4bac356c9d69:Cookies | Chocolate Chip Test - V1-3
2 | 37b52280-8fbe-11ee-961e-e1a804ad307a:Cookies | Apple Sweet Pie
3 | 8e90dd00-a0a1-11ec-adcf-a5314440ec47:Cookie | Dirt Cake ft. OTTO
4 | 77eab600-84c1-11ef-b69d-eb08e677efd8:Cookies | Holiday Spiced Butter Cake
5 | ed449980-72bf-11ee-a2b7-319903869c9c:Cookies | Buckeye Brownie*
6 | c93898e0-63fd-11ef-acc7-8b24d42a682d:Cookies | Cookie Dough Bits
7 | 854a3ee0-7e6b-11ee-bd83-59656f65a976:Cookies | Chocolate Sprinkle Blondie
8 | d3c9b120-27e8-11ed-a43a-21c73e66cda7:Cookies | R&D Recipe Builder
9 | ea84d370-8d22-11eb-b017-13424779f04a:Cookie | Chocolate Wafer Bar

    notebook controller is DISPOSED.

    View Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details.

### Execute DML Query¶

In [7]:

    # Drop the table
    sf.execute_query(
        """
        DROP TABLE IF EXISTS ANALYTICS.DATA_SCIENCE.CRUMBLPY_DEV;
        """
    )

In [8]:

    # # Check it again. This will return an error because the table no longer exists.
    # sf.fetch_data(
    #     """
    #     SELECT * FROM DATA_SCIENCE.CRUMBLPY_DEV;
    #     """
    # )

## SlackToolKit¶

### Slack Connection¶

In [9]:

    # Configure the Slack connection

    # Channel ID
    channel = 'C07ULKY3VD2' # CrumblPy Dev channel

    # Option 1: Use environment variables (os.environ)
    slack = SlackToolKit(default_channel=channel)

    # Option 2: Hard-code the credentials (not recommended)
    # slack = SlackToolKit(token='data_bot_token', default_channel=channel)

    # # Option 3: Prefect Secrets (deployment only)
    # sf = SnowflakeToolKit(prefect=True)

### Post Message¶

In [10]:

    slack.post_message("Hello from CrumblPy!")

### Post Reply¶

In [11]:

    latest_thread_id = slack.get_thread_id(channel=channel)
    slack.post_message("Reply from CrumblPy!", thread_id=latest_thread_id)

### Post File¶

In [12]:

    # create an example file
    sf.fetch_data(
        """
        SELECT NAME, OPENED_AT FROM STORES limit 10;
        """
    ).to_csv('example.csv', index=False)

In [13]:

    slack.post_file('example.csv', 'Here is an example file from CrumblPy!', thread_id=latest_thread_id)
    # The file will be deleted locally after being uploaded to Slack.

### Push Notification¶

In [14]:

    # Success message. Usually at the end of the script
    slack.push_notification(project="CrumblPy")

In [15]:

    # Error message
    try:
        1/0 # This will raise an error because you can't divide by zero
        slack.push_notification(project="CrumblPy")
    except Exception as e:
        slack.push_notification(project="CrumblPy", e=e)

## Gmail¶

In [16]:

    # create an example csv
    sf.fetch_data(
        """
        SELECT NAME, OPENED_AT FROM STORES limit 10;
        """
    ).to_csv('example.csv', index=False)

In [17]:

    send_gmail(
        sender = 'donation-reporting@crumbl.com',
        recipient = 'steven.wang@crumbl.com',
        subject = 'Subject from CrumblPy',
        body = 'Body from CrumblPy',
        token_path = 'token.json', # This token is not in Doppler or Prefect yet.
        image_paths = ['slack_message.png','slack_reply.png'],
        attachment_paths = ['example.csv']
        )