Metadata-Version: 2.1
Name: desktop-env
Version: 0.1.2
Summary: The package provides a desktop environment for setting and evaluating desktop automation tasks.
Home-page: https://github.com/xlang-ai/desktop_env
Author: Tianbao Xie, Danyang Zhang, Toh Jing Hua, etc.
Author-email: tianbaoxiexxx@gmail.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy ~=1.24.3
Requires-Dist: Pillow ~=10.1.0
Requires-Dist: fabric
Requires-Dist: gymnasium ~=0.28.1
Requires-Dist: requests ~=2.31.0
Requires-Dist: transformers ~=4.35.2
Requires-Dist: torch ~=2.1.1
Requires-Dist: accelerate
Requires-Dist: opencv-python ~=4.8.1.78
Requires-Dist: matplotlib ~=3.7.4
Requires-Dist: pynput ~=1.7.6
Requires-Dist: pyautogui ~=0.9.54
Requires-Dist: psutil ~=5.9.6
Requires-Dist: tqdm ~=4.65.0
Requires-Dist: pandas ~=2.0.3
Requires-Dist: flask ~=3.0.0
Requires-Dist: requests-toolbelt ~=1.0.0
Requires-Dist: lxml
Requires-Dist: cssselect
Requires-Dist: xmltodict
Requires-Dist: openpyxl
Requires-Dist: python-docx
Requires-Dist: python-pptx
Requires-Dist: pypdf
Requires-Dist: PyGetWindow
Requires-Dist: rapidfuzz
Requires-Dist: pyacoustid
Requires-Dist: opencv-python
Requires-Dist: ImageHash
Requires-Dist: scikit-image
Requires-Dist: librosa
Requires-Dist: pymupdf
Requires-Dist: chardet
Requires-Dist: playwright
Requires-Dist: backoff
Requires-Dist: formulas
Requires-Dist: pydrive
Requires-Dist: fastdtw
Requires-Dist: odfpy
Requires-Dist: openai
Requires-Dist: func-timeout
Requires-Dist: beautifulsoup4
Requires-Dist: dashscope
Requires-Dist: google-generativeai

# DesktopEnv: An Environment towards Human-like Computer Task Mastery
<p align="center">
    <b>SLOGAN</b>
</p>

<p align="center">
<a href="">Website</a> •
<a href="">Paper</a>
</p>

![Overview](media/overview.png)

## Updates
- 2024-03-01: 

## Install
1. Install VMWare and configure `vmrun` command:
Please refer to [guidance](https://docs.google.com/document/d/1KBdeZwmZs2Vi_Wsnngb3Wf1-RiwMMpXTftwMqP2Ztak/edit#heading=h.uh0x0tkl7fuw)

2. Install the environment package, download the examples and the virtual machine image.
```bash
pip install desktop_env
mkdir -p ~/.desktop_env
wget xxxx
wget xxxx
```

## Quick Start
Run the following minimal example to interact with the environment:
```python
import json
from desktop_env.envs.desktop_env import DesktopEnv

with open("evaluation_examples/examples/gimp/f723c744-e62c-4ae6-98d1-750d3cd7d79d.json", "r", encoding="utf-8") as f:
    example = json.load(f)

env = DesktopEnv(
    path_to_vm=r"path_to_vm",
    action_space="computer_13",
    task_config=example
)
observation = env.reset()

observation, reward, done, info = env.step({"action_type": "CLICK", "parameters": {"button": "right", "num_clicks": 1}})
```

## Annotation Tool Usage
We provide an annotation tool to help you annotate the examples.

## Agent Usage
We provide a simple agent to interact with the environment. You can use it as a starting point to build your own agent.

## Road map of infra (Proposed)

- [x] Explore VMWare, and whether it can be connected and control through mouse package
- [x] Explore Windows and MacOS, whether it can be installed
  - MacOS is closed source and cannot be legally installed
  - Windows is available legally and can be installed
- [x] Build gym-like python interface for controlling the VM
- [x] Recording of actions (mouse movement, click, keyboard) for humans to annotate, and we can replay it and compress it
- [x] Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser
- [x] Set up a pipeline and build agents implementation (zero-shot) for the task
- [x] Start to design on which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public
- [x] Start to annotate the examples for ~~training~~ and testing
- [x] Error handling during file passing and file opening, etc.
- [x] Add accessibility tree from the OS into the observation space
- [x] Add pre-process and post-process action support for benchmarking setup and evaluation
- [ ] Multiprocess support, this can enable the reinforcement learning to be more efficient
- [ ] Experiment logging and visualization system
- [ ] Add more tasks, maybe scale to 300 for v1.0.0, and create a dynamic leaderboard

## Road map of benchmark, tools and resources (Proposed)
- [ ] Improve the annotation tool base on DuckTrack, make it more robust which align on accessibility tree
- [ ] Annotate the steps of doing the task
- [ ] Build a website for the project
- [ ] Crawl all resources we explored from the internet, and make it easy to access
- [ ] Set up ways for community to contribute new examples

## Citation
If you find this environment useful, please consider citing our work:
```
@article{DesktopEnv,
  title={},
  author={},
  journal={arXiv preprint arXiv:xxxx.xxxx},
  year={2024}
}
```
