# vdk-control-service-api
The Data Jobs API of Versatile Data Kit Control Service. Data Jobs allows Data Engineers to implement automated pull ingestion (E in ELT) and batch data transformation into a database (T in ELT). See also https://github.com/vmware/versatile-data-kit/wiki/Introduction
The API has resource-oriented URLs, JSON-encoded responses, and uses standard HTTP response codes, authentication, and verbs. The API enables creating, deploying, managing and executing Data Jobs in the runtime environment.<br> <br> ![](https://github.com/vmware/versatile-data-kit/wiki/vdk-data-job-lifecycle-state-diagram.png) <br> The API reflects the usual Data Job Development lifecycle:<br> <li> Create a new data job (webhook to further configure the job, e.g authorize its creation, setup permissions, etc). <li> Download keytab. Develop and run the data job locally. <li> Deploy the data job in cloud runtime environment to run on a scheduled basis. <br><br> If Authentication is enabled, pass OAuth2 access token in HTTP header 'Authorization: Bearer [access-token-here]' (https://datatracker.ietf.org/doc/html/rfc6750). <br
The API promotes some best practices (inspired by https://12factor.net): <li> Explicitly declare and isolate dependencies. <li> Strict separation of configurations from code. Configurations vary substantially across deploys, code does not. <li> Separation between the build, release/deploy, and run stages. <li> Data Jobs are stateless and share-nothing processes. Any data that needs to be persisted must be stored in a stateful backing service (e.g IProperties). <li> Implementation is assumed to be atomic and idempotent - should be OK for a job to fail somewhere in the middle; subsequent restart should not cause data corruption. <li> Keep development, staging, and production as similar as possible. <br><br> <b>API Evolution</b><br> In the following sections, there are some terms that have a special meaning in the context of the APIs. <br><br> <li> <i>Stable</i> - The implementation of the API has been battle-tested (has been in production for some time). The API is a subject to semantic versioning model and will follow deprecation policy. <li> <i>Experimental</i> - May disappear without notice and is not a subject to semantic versioning. Implementation of the API is not considered stable nor well tested. Generally this is given to clients to experiment within testing environment. Must not be used in production. <li> <i>Deprecated</i> - API is expected to be removed within next one or two major version upgrade. The deprecation notice/comment will say when the API will be removed and what alternatives should be used instead.

This Python package is automatically generated by the [OpenAPI Generator](https://openapi-generator.tech) project:

- API version: 1.0
- Package version: 1.0.7
- Build package: org.openapitools.codegen.languages.PythonClientCodegen

## Requirements.

Python &gt;&#x3D;3.7

## Migration from other generators like python and python-legacy

### Changes
1. This generator uses spec case for all (object) property names and parameter names.
    - So if the spec has a property name like camelCase, it will use camelCase rather than camel_case
    - So you will need to update how you input and read properties to use spec case
2. Endpoint parameters are stored in dictionaries to prevent collisions (explanation below)
    - So you will need to update how you pass data in to endpoints
3. Endpoint responses now include the original response, the deserialized response body, and (todo)the deserialized headers
    - So you will need to update your code to use response.body to access deserialized data
4. All validated data is instantiated in an instance that subclasses all validated Schema classes and Decimal/str/list/tuple/frozendict/NoneClass/BoolClass/bytes/io.FileIO
    - This means that you can use isinstance to check if a payload validated against a schema class
    - This means that no data will be of type None/True/False
        - ingested None will subclass NoneClass
        - ingested True will subclass BoolClass
        - ingested False will subclass BoolClass
        - So if you need to check is True/False/None, instead use instance.is_true_oapg()/.is_false_oapg()/.is_none_oapg()
5. All validated class instances are immutable except for ones based on io.File
    - This is because if properties were changed after validation, that validation would no longer apply
    - So no changing values or property values after a class has been instantiated
6. String + Number types with formats
    - String type data is stored as a string and if you need to access types based on its format like date,
    date-time, uuid, number etc then you will need to use accessor functions on the instance
    - type string + format: See .as_date_oapg, .as_datetime_oapg, .as_decimal_oapg, .as_uuid_oapg
    - type number + format: See .as_float_oapg, .as_int_oapg
    - this was done because openapi/json-schema defines constraints. string data may be type string with no format
    keyword in one schema, and include a format constraint in another schema
    - So if you need to access a string format based type, use as_date_oapg/as_datetime_oapg/as_decimal_oapg/as_uuid_oapg
    - So if you need to access a number format based type, use as_int_oapg/as_float_oapg
7. Property access on AnyType(type unset) or object(dict) schemas
    - Only required keys with valid python names are properties like .someProp and have type hints
    - All optional keys may not exist, so properties are not defined for them
    - One can access optional values with dict_instance['optionalProp'] and KeyError will be raised if it does not exist
    - Use get_item_oapg if you need a way to always get a value whether or not the key exists
        - If the key does not exist, schemas.unset is returned from calling dict_instance.get_item_oapg('optionalProp')
        - All required and optional keys have type hints for this method, and @typing.overload is used
        - A type hint is also generated for additionalProperties accessed using this method
    - So you will need to update you code to use some_instance['optionalProp'] to access optional property
    and additionalProperty values
8. The location of the api classes has changed
    - Api classes are located in your_package.apis.tags.some_api
    - This change was made to eliminate redundant code generation
    - Legacy generators generated the same endpoint twice if it had > 1 tag on it
    - This generator defines an endpoint in one class, then inherits that class to generate
      apis by tags and by paths
    - This change reduces code and allows quicker run time if you use the path apis
        - path apis are at your_package.apis.paths.some_path
    - Those apis will only load their needed models, which is less to load than all of the resources needed in a tag api
    - So you will need to update your import paths to the api classes

### Why are Oapg and _oapg used in class and method names?
Classes can have arbitrarily named properties set on them
Endpoints can have arbitrary operationId method names set
For those reasons, I use the prefix Oapg and _oapg to greatly reduce the likelihood of collisions
on protected + public classes/methods.
oapg stands for OpenApi Python Generator.

### Object property spec case
This was done because when payloads are ingested, they can be validated against N number of schemas.
If the input signature used a different property name then that has mutated the payload.
So SchemaA and SchemaB must both see the camelCase spec named variable.
Also it is possible to send in two properties, named camelCase and camel_case in the same payload.
That use case should be support so spec case is used.

### Parameter spec case
Parameters can be included in different locations including:
- query
- path
- header
- cookie

Any of those parameters could use the same parameter names, so if every parameter
was included as an endpoint parameter in a function signature, they would collide.
For that reason, each of those inputs have been separated out into separate typed dictionaries:
- query_params
- path_params
- header_params
- cookie_params

So when updating your code, you will need to pass endpoint parameters in using those
dictionaries.

### Endpoint responses
Endpoint responses have been enriched to now include more information.
Any response reom an endpoint will now include the following properties:
response: urllib3.HTTPResponse
body: typing.Union[Unset, Schema]
headers: typing.Union[Unset, TODO]
Note: response header deserialization has not yet been added


## Installation & Usage
### pip install

If the python package is hosted on a repository, you can install directly using:

```sh
pip install git+https://github.com/GIT_USER_ID/GIT_REPO_ID.git
```
(you may need to run `pip` with root permission: `sudo pip install git+https://github.com/GIT_USER_ID/GIT_REPO_ID.git`)

Then import the package:
```python
import taurus_datajob_api
```

### Setuptools

Install via [Setuptools](http://pypi.python.org/pypi/setuptools).

```sh
python setup.py install --user
```
(or `sudo python setup.py install` to install the package for all users)

Then import the package:
```python
import taurus_datajob_api
```

## Getting Started

Please follow the [installation procedure](#installation--usage) and then run the following:

```python

import time
import taurus_datajob_api
from pprint import pprint
from taurus_datajob_api.apis.tags import data_jobs_api
from taurus_datajob_api.model.data_job import DataJob
from taurus_datajob_api.model.data_job_query_response import DataJobQueryResponse
from taurus_datajob_api.model.data_job_query_response_with_error import DataJobQueryResponseWithError
from taurus_datajob_api.model.error import Error
# Defining the host is optional and defaults to http://localhost
# See configuration.py for a list of all supported configuration parameters.
configuration = taurus_datajob_api.Configuration(
    host = "http://localhost"
)

# The client must configure the authentication and authorization parameters
# in accordance with the API server security policy.
# Examples for each auth method are provided below, use the example that
# satisfies your auth use case.

# Configure Bearer authorization (JWT): bearerAuth
configuration = taurus_datajob_api.Configuration(
    access_token = 'YOUR_BEARER_TOKEN'
)

# Enter a context with an instance of the API client
with taurus_datajob_api.ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = data_jobs_api.DataJobsApi(api_client)
    team_name = "team_name_example" # str | The Team which owns the Data Job
data_job = DataJob(
        job_name="starshot-processing-vmc-fact-daily",
        description="Data Job responsible for transforming vmc-related fact tables on a daily basis.",
        config=DataJobConfig(
            db_default_type="TRINO",
            contacts=DataJobContacts(
                notified_on_job_failure_user_error=["auserov@example.mail.com"],
                notified_on_job_failure_platform_error=["auserov@example.mail.com"],
                notified_on_job_success=["auserov@example.mail.com"],
                notified_on_job_deploy=["auserov@example.mail.com"],
            ),
            schedule=DataJobSchedule(
                schedule_cron="0 0 13 * 5",
            ),
            generate_keytab=False,
            enable_execution_notifications=False,
            notification_delay_period_minutes=60,
        ),
        team="starshot (Optional property)",
    ) # DataJob | 
name = "name_example" # str | The Name of the Data Job (optional)

    try:
        # Creates a new Data Job | (Stable)
        api_instance.data_job_create(team_namedata_jobname=name)
    except taurus_datajob_api.ApiException as e:
        print("Exception when calling DataJobsApi->data_job_create: %s\n" % e)
```

## Documentation for API Endpoints

All URIs are relative to *http://localhost*

Class | Method | HTTP request | Description
------------ | ------------- | ------------- | -------------
*DataJobsApi* | [**data_job_create**](docs/apis/tags/DataJobsApi.md#data_job_create) | **post** /data-jobs/for-team/{team_name}/jobs | Creates a new Data Job | (Stable)
*DataJobsApi* | [**data_job_delete**](docs/apis/tags/DataJobsApi.md#data_job_delete) | **delete** /data-jobs/for-team/{team_name}/jobs/{job_name} | Delete Data Job | (Stable)
*DataJobsApi* | [**data_job_keytab_download**](docs/apis/tags/DataJobsApi.md#data_job_keytab_download) | **get** /data-jobs/for-team/{team_name}/jobs/{job_name}/keytab | Get data job keytab. | (Stable)
*DataJobsApi* | [**data_job_read**](docs/apis/tags/DataJobsApi.md#data_job_read) | **get** /data-jobs/for-team/{team_name}/jobs/{job_name} | Retrieves details of an existing Data Job by specifying the name of the Data Job. | (Stable)
*DataJobsApi* | [**data_job_team_update**](docs/apis/tags/DataJobsApi.md#data_job_team_update) | **put** /data-jobs/for-team/{team_name}/jobs/{job_name}/team/{new_team} | Update API for Data Jobs team | (Stable)
*DataJobsApi* | [**data_job_update**](docs/apis/tags/DataJobsApi.md#data_job_update) | **put** /data-jobs/for-team/{team_name}/jobs/{job_name} | Update Data Job. | (Stable)
*DataJobsApi* | [**jobs_query**](docs/apis/tags/DataJobsApi.md#jobs_query) | **get** /data-jobs/for-team/{team_name}/jobs | List Data Jobs  | (Experimental)
*DataJobsDeploymentApi* | [**deployment_delete**](docs/apis/tags/DataJobsDeploymentApi.md#deployment_delete) | **delete** /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id} | Delete Deployment of a Data Job. Currently executing Data Job will be left to finish.  | (Stable) 
*DataJobsDeploymentApi* | [**deployment_list**](docs/apis/tags/DataJobsDeploymentApi.md#deployment_list) | **get** /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments | Get Data Job deployments. | (Stable)
*DataJobsDeploymentApi* | [**deployment_patch**](docs/apis/tags/DataJobsDeploymentApi.md#deployment_patch) | **patch** /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id} | Patch a deployment of a Data Job. Use it to change the configuration of a data job. For example: to enable or disable deployment, to change the vdk version. The operation is guranteed to be synchrounous so it cannot be used to deploy new version of a data job - job_version cannot be changed using PATCH. Use POST .../deployments for this. | (Stable) 
*DataJobsDeploymentApi* | [**deployment_read**](docs/apis/tags/DataJobsDeploymentApi.md#deployment_read) | **get** /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id} | Get Data Job deployments. | (Stable)
*DataJobsDeploymentApi* | [**deployment_update**](docs/apis/tags/DataJobsDeploymentApi.md#deployment_update) | **post** /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments | Creates or updates a deployment of a Data Job. | (Stable)
*DataJobsExecutionApi* | [**data_job_deployment_execution_list**](docs/apis/tags/DataJobsExecutionApi.md#data_job_deployment_execution_list) | **get** /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/executions | Get Data Jobs (recent) executions. | (Experimental)
*DataJobsExecutionApi* | [**data_job_execution_cancel**](docs/apis/tags/DataJobsExecutionApi.md#data_job_execution_cancel) | **delete** /data-jobs/for-team/{team_name}/jobs/{job_name}/executions/{execution_id} | Cancel (if running) Data Job Execution | (Experimental)
*DataJobsExecutionApi* | [**data_job_execution_list**](docs/apis/tags/DataJobsExecutionApi.md#data_job_execution_list) | **get** /data-jobs/for-team/{team_name}/jobs/{job_name}/executions | Get Data Jobs (recent) executions. (Execution API is still experimental and must not be used in production) | (Experimental)
*DataJobsExecutionApi* | [**data_job_execution_read**](docs/apis/tags/DataJobsExecutionApi.md#data_job_execution_read) | **get** /data-jobs/for-team/{team_name}/jobs/{job_name}/executions/{execution_id} | Get Data Job Execution details. | (Experimental)
*DataJobsExecutionApi* | [**data_job_execution_start**](docs/apis/tags/DataJobsExecutionApi.md#data_job_execution_start) | **post** /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/executions | Trigger Data Job Execution. | (Experimental)
*DataJobsExecutionApi* | [**data_job_logs_download**](docs/apis/tags/DataJobsExecutionApi.md#data_job_logs_download) | **get** /data-jobs/for-team/{team_name}/jobs/{job_name}/executions/{execution_id}/logs | Download data job logs. This API is guranteed to provide logs only if the jobs is currently running. For logs from older job executions - use logsUrl field passed by GET execution API or jobsQuery API. | (Experimental) 
*DataJobsPropertiesApi* | [**data_job_properties_read**](docs/apis/tags/DataJobsPropertiesApi.md#data_job_properties_read) | **get** /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/properties | Get Data Job properties.
*DataJobsPropertiesApi* | [**data_job_properties_update**](docs/apis/tags/DataJobsPropertiesApi.md#data_job_properties_update) | **put** /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/properties | Update Data Job properties.
*DataJobsServiceApi* | [**info**](docs/apis/tags/DataJobsServiceApi.md#info) | **get** /data-jobs/for-team/{team_name}/info | Get API and Data Jobs Service info, list of supported python versions
*DataJobsSourcesApi* | [**data_job_sources_download**](docs/apis/tags/DataJobsSourcesApi.md#data_job_sources_download) | **get** /data-jobs/for-team/{team_name}/jobs/{job_name}/sources | Download data job source code. | (Not Implemented)
*DataJobsSourcesApi* | [**sources_delete**](docs/apis/tags/DataJobsSourcesApi.md#sources_delete) | **delete** /data-jobs/for-team/{team_name}/jobs/{job_name}/sources | Delete Data Job source.
*DataJobsSourcesApi* | [**sources_upload**](docs/apis/tags/DataJobsSourcesApi.md#sources_upload) | **post** /data-jobs/for-team/{team_name}/jobs/{job_name}/sources | Upload Data Job source code. | (Stable)

## Documentation For Models

 - [DataJob](docs/models/DataJob.md)
 - [DataJobApiInfo](docs/models/DataJobApiInfo.md)
 - [DataJobConfig](docs/models/DataJobConfig.md)
 - [DataJobContacts](docs/models/DataJobContacts.md)
 - [DataJobDeployment](docs/models/DataJobDeployment.md)
 - [DataJobDeploymentId](docs/models/DataJobDeploymentId.md)
 - [DataJobDeploymentStatus](docs/models/DataJobDeploymentStatus.md)
 - [DataJobExecution](docs/models/DataJobExecution.md)
 - [DataJobExecutionLogs](docs/models/DataJobExecutionLogs.md)
 - [DataJobExecutionRequest](docs/models/DataJobExecutionRequest.md)
 - [DataJobMode](docs/models/DataJobMode.md)
 - [DataJobPage](docs/models/DataJobPage.md)
 - [DataJobProperties](docs/models/DataJobProperties.md)
 - [DataJobQueryResponse](docs/models/DataJobQueryResponse.md)
 - [DataJobQueryResponseWithError](docs/models/DataJobQueryResponseWithError.md)
 - [DataJobResources](docs/models/DataJobResources.md)
 - [DataJobSchedule](docs/models/DataJobSchedule.md)
 - [DataJobSummary](docs/models/DataJobSummary.md)
 - [DataJobVersion](docs/models/DataJobVersion.md)
 - [Error](docs/models/Error.md)

## Documentation For Authorization

 Authentication schemes defined for the API:
## bearerAuth

- **Type**: Bearer authentication (JWT)


## Author








## Notes for Large OpenAPI documents
If the OpenAPI document is large, imports in taurus_datajob_api.apis and taurus_datajob_api.models may fail with a
RecursionError indicating the maximum recursion limit has been exceeded. In that case, there are a couple of solutions:

Solution 1:
Use specific imports for apis and models like:
- `from taurus_datajob_api.apis.default_api import DefaultApi`
- `from taurus_datajob_api.model.pet import Pet`

Solution 1:
Before importing the package, adjust the maximum recursion limit as shown below:
```
import sys
sys.setrecursionlimit(1500)
import taurus_datajob_api
from taurus_datajob_api.apis import *
from taurus_datajob_api.models import *
```
