# PySLL
SDKs and example code for accessing the Constellation APIs that power Emerald Cloud Lab.

## The constellation API
Detailed documentation about the Constellation API can be found at www.emeraldcloudlab.com/internal-developers-api.

## Environment Variables
If you are setting these in your `.bash_profile` (or similar), make sure to `export` them or they won't be available to `os.environ`.

### Standard Credentials
- `CONSTELLATION_AUTH_TOKEN` (Optional)
- `CONSTELLATION_USERNAME` (Required if no token)
- `CONSTELLATION_PASSWORD` (Required if no token)

### Test Credentials
To run the test suite, use the `PYTEST` prefix. The login has to be service+manifold@emeraldcloudlab.com since some tests expect this user ID.
- `PYTEST_CONSTELLATION_AUTH_TOKEN` (Optional)
- `PYTEST_CONSTELLATION_USERNAME` (Required if no token)
- `PYTEST_CONSTELLATION_PASSWORD` (Required if no token)

Example for tests:
```bash
export PYTEST_CONSTELLATION_USERNAME="your_username"
export PYTEST_CONSTELLATION_PASSWORD="your_password"
```

## Development Setup

After cloning the repository, set up the development environment:

```bash
# Install dependencies
uv sync --dev

# Install git hooks for pre-commit and pre-push
uv run pre-commit install
uv run pre-commit install --hook-type pre-push
```

The pre-commit hooks will automatically:
- Run ruff formatting and linting on commit
- Run pyright type checking on commit
- Run the full test suite on push

## Quick start
Install with pip:

```
pip install pysll
```

To use the SDK:
``` python
>>> from pysll import Constellation
>>> from pysll.models import Object
>>> client = Constellation()
```

To login:
``` python
>>> client.login("scientist@science.com", "myAwesomePassword")
```

Alternatively, load credentials automatically from environment variables using `from_env()`:
``` python
>>> client = Constellation.from_env()
```

This reads `CONSTELLATION_AUTH_TOKEN` if set, otherwise falls back to `CONSTELLATION_USERNAME` and `CONSTELLATION_PASSWORD`. An optional `prefix` argument lets you namespace the variables (e.g., `prefix="PYTEST"` reads `PYTEST_CONSTELLATION_USERNAME`, etc.).

To get information about the current user once you are logged in:
``` python
>>> me = client.me()
>>> print(me)
{'Email': 'scientist@science.com', 'EmailAddress':'scientist@science.com', 'Id': 'id:abc123', 'Type': 'Object.User', 'Username': 'scientist'}
```

To download information from an object:
``` python
>>> client.download(Object(me["Id"]), ["Name", "Email"])
["scientist", "scientist@science.com"]
```

To search for objects of a specific type:
``` python
>>> client.search("Object.User.Emerald.Administrator", "")
```

### Searching with conditions

You can perform more specific searches by providing an SLL query string:

``` python
>>> client.search(
...     "Object.SupportTicket.Operations",
...     'Status="Active" AND Priority="High"'
... )
```

You can also limit the number of results:

``` python
>>> client.search("Object.SupportTicket.Operations", "", max_results=5)
```

## Downloading data

You may perform a simple single field download like:
``` python
>>> client.download(Object("id:BYDOjvG4l3Ol"), "ColumnOrientation")
'Forward'
```

You may download multiple fields in a single download like:
``` python
>>> client.download(Object("id:BYDOjvG4l3Ol"), ["SeparationMode", "ColumnOrientation"])
['ReversePhase', 'Forward']
```

You may download from multiple objects in a single download like:
``` python
>>> client.download([Object("id:o1k9jAkRM794"), Object("id:L8kPEjkw47jw")], "ColumnOrientation")
['Forward', 'Forward']
```

And finally, you may download multiple fields from multiple objects in a single download like:
``` python
>>> client.download([Object("id:o1k9jAkRM794"), Object("id:L8kPEjkw47jw")], ["SeparationMode", "ColumnOrientation"])
[['ReversePhase', 'Forward'], ['ReversePhase', 'Forward']]
```

You may also traverse links within downloads, like:
``` python
>>> client.download(Object("id:BYDOjvG4l3Ol"), "Instrument[Model[Name]]")
'Waters Acquity UPLC H-Class ELS with Pre-Column Heater'
```

You can also download all of the fields on an object by not specifying a field.  For example:
``` python
>>> client.download(Object("id:Z1lqpMzvkGMV"))
{'type': 'Object.User.Emerald.Developer', 'id': 'id:Z1lqpMzvkGMV'....}
```

Or via the "All" implicit field:
``` python
>>> client.download(Object("id:Z1lqpMzvkGMV"), "All")
{'type': 'Object.User.Emerald.Developer', 'id': 'id:Z1lqpMzvkGMV'....}
```

Note that in this case, the results will be a dictionary mapping field name to field value

## Dealing with types

There are a number of different ways to interpret field values based off the type of data stored in the object.  String, integer, and real fields are mapped to their corresponding python types - for example:
``` python
>>> client.download(Object("id:BYDOjvG4l3Ol"), ["SeparationMode", "InjectionIndex"])
['ReversePhase', 28]
```

Link fields will return objects, which you can chain downloads off of (although note that traversals will be much faster):
``` python
>>> client.download(Object("id:BYDOjvG4l3Ol"), "Instrument")
Object[Instrument[HPLC, "id:wqW9BP4ARZVw"]
>>> client.download(client.download(Object("id:BYDOjvG4l3Ol"), "Instrument"), "Name")
'Galadriel'
>>> client.download(Object("id:BYDOjvG4l3Ol"), "Instrument[Name]")
'Galadriel'
```

Date fields will be converted to native python datetime objects:
``` python
>>> client.download(Object("id:BYDOjvG4l3Ol"), "DateCreated")
datetime.datetime(2022, 1, 9, 23, 44, 31, 746154)
```

Quantity arrays will be converted to python variable unit objects:
``` python
>>> client.download(Object("id:BYDOjvG4l3Ol"), "Scattering")
[[0.0 Minutes, -87.528984 IndependentUnit[Lsus]], [0.016667 Minutes, -96.701614 IndependentUnit[Lsus]], [0.033333 Minutes, -43.93272 IndependentUnit[Lsus]], [0.05 Minutes, -132.207855 IndependentUnit[Lsus]]...
```

which you may manipulate to get their values and units:
``` python
>>> scattering_info = client.download(Object("id:BYDOjvG4l3Ol"), "Scattering")
>>> len(scattering_info)
361
>>> scattering_info[0]
[0.0 Minutes, -87.528984 IndependentUnit[Lsus]]
>>> scattering_info[0][0]
0.0 Minutes
>>> scattering_info[0][0].value
0.0
>>> scattering_info[0][0].unit
'Minutes'
```

Blob refs will be downloaded and automatically parsed in the same way:
```python
>>> client.download(Object("id:BYDOjvG4l3Ol"), "Absorbance")
[[0.0 'Minutes', 0.0 'Milli' 'AbsorbanceUnit'], [0.0008333333535119891 'Minutes', 0.0 'Milli' 'AbsorbanceUnit']...
```

Additionally, you can download multiple fields that have different units the same as you would download other fields.  For example:
```python
>>> client.download(Object("id:O81aEB16GlJ1"), "Composition")
[[4.977777777777776 Times[Power["Liters", -1], "Milligrams"], Object[Model[Molecule, "id:E8zoYvN6m61A"]], [75.11111111111111 IndependentUnit["VolumePercent"], Object[Model[Molecule, "id:vXl9j57PmP5D"]]]
```

Finally, you can download association fields and they will be automatically translated into python structures.  For example:

```python
>>> client.download(Object("id:XnlV5jKZwmp3"), "ResolvedOptions")['Instrument']
Object[Instrument[HPLC, "id:wqW9BP4ARZVw"]]
```

## Download Files

Files are controlled via the `auto_download_cloud_files` flag to the download function.  By default, they will be returned as objects and not downloaded.

For example:
```python
>>> client.download(Object("id:BYDOjvG4l3Ol"), "DataFile")
Object[EmeraldCloudFile, "id:9RdZXv1jDAZ6"]
````

These may be manually downloaded via:
```python
>>> client.download_cloud_file(client.download(Object("id:BYDOjvG4l3Ol"), "DataFile"))
'/var/folders/j_/ftdn14ms37s40j2z0h1wzxbw0000gn/T/tmp6krhb8lp/Absorbance Raw File.bin_absorbancefile'
```

or, it is possible to automatically download them by using the `auto_download_cloud_files` flag of download:
```python
>>> data_file = client.download(Object("id:BYDOjvG4l3Ol"), "DataFile", auto_download_cloud_files=True)
>>> data_file.local_path
'/var/folders/j_/ftdn14ms37s40j2z0h1wzxbw0000gn/T/tmp6krhb8lp/Absorbance Raw File_1.bin_absorbancefile'
```

The format of these files can often change, but the sdk is pretty smart about interpreting them.  Once you have downloaded
the file, you can have the sdk attempt to parse it into python structs via the following:

```python
>>> data_file = client.download(Object("id:BYDOjvG4l3Ol"), "DataFile", auto_download_cloud_files=True)
>>> from pysll.models import ConstellationFieldParser
>>> ConstellationFieldParser().parse_local_file(data_file.local_path)
[[0.0 'Minutes', 273.0 'Nanometers', 0.0 'Milli' 'AbsorbanceUnit'], [0.0008333333535119891 'Minutes', 273.0 'Nanometers', 0.0 'Milli' 'AbsorbanceUnit']...
```

If the field parser is unable to parse the file, it will return `None`.

## Development

This project uses [uv](https://docs.astral.sh/uv/) for dependency management and [pre-commit](https://pre-commit.com/) for code quality checks.

### Setup

1. **Install uv** (if not already installed):
   ```bash
   curl -LsSf https://astral.sh/uv/install.sh | sh
   ```

2. **Install dependencies**:
   ```bash
   uv sync --all-groups
   ```

   This installs all project dependencies including dev dependencies (pytest, ruff, pyright, etc.).

3. **Set up pre-commit hooks**:
   ```bash
   uv run pre-commit install
   ```

   This installs git hooks that automatically run linters and formatters before each commit.

### Running Tests

Run the full test suite:
```bash
uv run pytest
```

Run tests in parallel (faster):
```bash
uv run pytest -n auto
```

Run tests with coverage:
```bash
uv run pytest --cov=pysll --cov-report=term-missing
```

Run specific tests:
```bash
uv run pytest tests/test_constellation.py
uv run pytest tests/test_constellation.py::test_login
```

### Code Quality

Run pre-commit checks manually on all files:
```bash
uv run pre-commit run --all-files
```

Run specific checks:
```bash
# Format code with ruff
uv run ruff format

# Lint code with ruff
uv run ruff check

# Type checking with pyright
uv run pyright
```

### Adding Dependencies

Use `uv add` to add new dependencies (never edit `pyproject.toml` directly):

```bash
# Add a runtime dependency
uv add package-name

# Add a dev dependency
uv add --group dev package-name

# Add an optional dependency
uv add --optional extra-name package-name
```

## Uploading data

The `upload` function allows you to create new objects or update existing ones.

### Create a new object

To create a new object, pass `None` as the `object_id`:

```python
>>> client.upload(
...     "Object.SupportTicket.Operations",
...     None,
...     {
...         "Name": "New Support Request",
...         "Description": "Calculated results are missing",
...         "Status": "Open"
...     }
... )
```

### Update an existing object

To update an existing object, provide its ID:

```python
>>> client.upload(
...     "Object.SupportTicket.Operations",
...     "id:ticket-123",
...     {"Status": "InProgress"}
... )
```

### Advanced Upload Examples

**Linking to other objects:**

```python
>>> from pysll.utils import create_one_way_link
>>> client.upload(
...     "Object.SupportTicket.Operations",
...     "id:ticket-123",
...     {
...         "Assignee": create_one_way_link("Object.User.Emerald.Developer", "id:user-456")
...     }
... )
```

**Appending to a list field:**

```python
>>> client.upload(
...     "Object.Sample",
...     "id:sample-123",
...     {"Append[Tags]": ["tag1", "tag2"]}
... )
```

**Setting a quantity field:**

```python
>>> client.upload(
...     "Object.Sample",
...     None,
...     {"Volume": 'Quantity[0.5, "Liters"]'}
... )
```

### Batch Uploading

Use `upload_many` to create or update multiple objects in a single request. Each element is a `(object_type, object_id, fields)` tuple — pass `None` for `object_id` to create a new object.

**Create multiple objects at once:**

```python
>>> results = client.upload_many([
...     ("Object.Sample", None, {"Name": "Sample A"}),
...     ("Object.Sample", None, {"Name": "Sample B"}),
...     ("Object.Sample", None, {"Name": "Sample C"}),
... ])
>>> [r["resolved_object"]["id"] for r in results]
['id:abc', 'id:def', 'id:ghi']
```

**Mix creates and updates in one call:**

```python
>>> results = client.upload_many([
...     ("Object.Sample", None, {"Name": "New Sample"}),
...     ("Object.Sample", "id:existing-123", {"Name": "Updated Sample"}),
... ])
```

### Uploading Files

You can upload local files to the cloud:

```python
>>> client.upload_cloud_file("path/to/my_data.txt")
```

### Notebook Context

You can specify a notebook context for your uploads:

```python
>>> with client.notebook("id:notebook-123"):
...     client.upload("Object.Example.Data", None, {"Name": "Contextual Data"})
```

## Object History

Use `object_log()` to retrieve the change history for one or more objects:

```python
>>> from datetime import datetime
>>> client.object_log(
...     object_ids=["id:abc123"],
...     start_date=datetime(2024, 1, 1),
...     max_results=100
... )
```

You can also filter by type instead of by specific IDs:

```python
>>> client.object_log(types=["Object.SupportTicket.Operations"], max_results=50)
```

## Type Introspection

Use `get_type()` to retrieve field definitions for any Constellation type:

```python
>>> client.get_type("Object.SupportTicket.Operations")
```

This returns the field names, formats, and other metadata for the type, which is useful when exploring the schema or building dynamic queries.

## Optional: NumPy Integration

If you have NumPy installed, pysll can use it for more efficient handling of large numerical arrays. Enable it by setting the `PYSLL_USE_NUMPY` environment variable:

```bash
export PYSLL_USE_NUMPY=true
```

Install with NumPy support:

```bash
pip install pysll[numpy]
# or via uv:
uv add pysll --optional numpy
```
