# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['kosmosx',
 'kosmosx.model',
 'kosmosx.model.allModalities',
 'kosmosx.model.allModalities.audio',
 'kosmosx.model.experiments',
 'kosmosx.model.video',
 'kosmosx.model.video.imagebind',
 'kosmosx.torchscale',
 'kosmosx.torchscale.examples',
 'kosmosx.torchscale.examples.fairseq',
 'kosmosx.torchscale.examples.fairseq.criterions',
 'kosmosx.torchscale.examples.fairseq.models',
 'kosmosx.torchscale.examples.fairseq.tasks',
 'kosmosx.torchscale.examples.fairseq.tasks.data',
 'kosmosx.torchscale.examples.fairseq.utils',
 'kosmosx.torchscale.tests',
 'kosmosx.torchscale.torchscale',
 'kosmosx.torchscale.torchscale.architecture',
 'kosmosx.torchscale.torchscale.component',
 'kosmosx.torchscale.torchscale.component.xmoe',
 'kosmosx.torchscale.torchscale.model',
 'kosmosx.utils']

package_data = \
{'': ['*']}

install_requires = \
['accelerate',
 'bitsandbytes',
 'boto3',
 'datasets',
 'fairscale',
 'flamingo_pytorch',
 'lion-pytorch',
 'pillow',
 'rich',
 'tensorboard',
 'torch',
 'torchscale',
 'transformers',
 'wandb']

setup_kwargs = {
    'name': 'kosmosx',
    'version': '0.0.8',
    'description': 'Kosmos-X - PyTorch',
    'long_description': '[![Multi-Modality](images/agorabanner.png)](https://discord.gg/qUtxnK2NMf)\n\n# Kosmos-X: Advanced Multi-Modality AI Model 🚀🌌\n\n![Kosmos-X Next Generation Multi-Modality AI Model](images/kosmos-banner.png)\n\n[![GitHub issues](https://img.shields.io/github/issues/kyegomez/Kosmos-X)](https://github.com/kyegomez/Kosmos-X/issues) \n[![GitHub forks](https://img.shields.io/github/forks/kyegomez/Kosmos-X)](https://github.com/kyegomez/Kosmos-X/network) \n[![GitHub stars](https://img.shields.io/github/stars/kyegomez/Kosmos-X)](https://github.com/kyegomez/Kosmos-X/stargazers) \n[![GitHub license](https://img.shields.io/github/license/kyegomez/Kosmos-X)](https://github.com/kyegomez/Kosmos-X/blob/main/LICENSE)\n\n[![Share on Twitter](https://img.shields.io/twitter/url/https/twitter.com/cloudposse.svg?style=social&label=Share%20%40kyegomez/Kosmos-X)](https://twitter.com/intent/tweet?text=Check%20out%20this%20amazing%20AI%20project:%20Kosmos-X&url=https%3A%2F%2Fgithub.com%2Fkyegomez%2FKosmos-X) \n[![Share on Facebook](https://img.shields.io/badge/Share-%20facebook-blue)](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fgithub.com%2Fkyegomez%2FKosmos-X) \n[![Share on LinkedIn](https://img.shields.io/badge/Share-%20linkedin-blue)](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fgithub.com%2Fkyegomez%2FKosmos-X&title=&summary=&source=)\n![Discord](https://img.shields.io/discord/999382051935506503)\n[![Share on Reddit](https://img.shields.io/badge/-Share%20on%20Reddit-orange)](https://www.reddit.com/submit?url=https%3A%2F%2Fgithub.com%2Fkyegomez%2FKosmos-X&title=Kosmos-X%20-%20the%20next%20generation%20AI%20shields) \n[![Share on Hacker News](https://img.shields.io/badge/-Share%20on%20Hacker%20News-orange)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fgithub.com%2Fkyegomez%2FKosmos-X&t=Kosmos-X%20-%20the%20next%20generation%20AI%20shields) \n[![Share on Pinterest](https://img.shields.io/badge/-Share%20on%20Pinterest-red)](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fgithub.com%2Fkyegomez%2FKosmos-X&media=https%3A%2F%2Fexample.com%2Fimage.jpg&description=Kosmos-X%20-%20the%20next%20generation%20AI%20shields) \n[![Share on WhatsApp](https://img.shields.io/badge/-Share%20on%20WhatsApp-green)](https://api.whatsapp.com/send?text=Check%20out%20Kosmos-X%20-%20the%20next%20generation%20AI%20shields%20%23Kosmos-X%20%23AI%0A%0Ahttps%3A%2F%2Fgithub.com%2Fkyegomez%2FKosmos-X)\n\n\nKosmos-X is a modular Multi-Modality AI model designed to seamlessly process diverse forms of data, including:\n\n* Images\n* Videos\n* Audio\n* Any type\n\nAt lightspeed with ultra-reliability.\n\nThe unique strength of Kosmos-X lies in its ability to process extremely long sequences of these multi-modality inputs, with context lengths of up to 40,000+!\n\n## Model Roadmap\n[Help us create a Model Roadmap on Kosmos-X Figma](https://www.figma.com/file/z3sNPzuB3thdOKT7oml6NI/Kosmos-X?type=whiteboard&node-id=1%3A142&t=Z37mybFxYALukurx-1)\n\n----\n\n## Ready for Training!\n\nKosmos-X is now ready for training, and we\'re actively seeking cloud providers or grant providers to collaborate in training this revolutionary model and eventually release it open source. If you\'re interested in learning more or supporting this endeavor, please feel free to email me at `kye@apac.ai`.\n\n---\n\n## Appreciation\n* All the creators in Agora, [Join Agora](https://discord.gg/qUtxnK2NMf) the community of AI engineers changing the world with their creations.\n* LucidRains for inspiring me to devote myself to open source AI\n\n---\n\n## Installation\n```bash\npip install kosmosx\n```\n\n## Usage\n\n```python\nimport torch\nfrom kosmosx.model import Kosmos\n\n# Create a sample text token tensor\ntext_tokens = torch.randint(0, 32002, (1, 50), dtype=torch.long)\n\n# Create a sample image tensor\nimages = torch.randn(1, 3, 224, 224)\n\n# Instantiate the model\nmodel = Kosmos()\n\n# Pass the sample tensors to the model\'s forward function\noutput = model.forward(\n    text_tokens=text_tokens,\n    images=images\n)\n\n# Print the output from the model\nprint(f"Output: {output}")\n\n```\n\n# Training\n\n`accelerate config`\n\nthen: `accelerate launch train_distributed.py`\n\n## Get Involved\n\nWe\'re just at the beginning of our journey. As we continue to develop and refine Kosmos-X, we invite you to join us. Whether you\'re a developer, researcher, or simply an enthusiast, your insights and contributions can help shape the future of Kosmos-X.\n\n# Contributing to Kosmos-X\n\nWe are thrilled to invite you to be a part of the Kosmos-X project. This is not just an open source project but a community initiative, and we value your expertise and creativity. To show our appreciation, we have instituted a unique rewards system that directly compensates contributors from the revenue generated by the Kosmos-X API.\n\n## Why Contribute\n\nContributing to Kosmos-X not only enhances your skills and profile but also comes with financial rewards. When you contribute code, documentation, or any form of improvement to the Kosmos-X project, you are adding value. As such, we believe it\'s only fair that you share in the rewards.\n\n---\n\n## The model\nKOSMOS-1 uses a decoder-only Transformer architecture based on [Magneto (Foundation Transformers)](https://arxiv.org/abs/2210.06423), i.e. an architecture that employs a so called sub-LN approach where layer normilization is added both before the attention module (pre-ln) and afterwards (post-ln) combining the advantages that either approaches have for language modelling and image understanding respectively. The model is also initialized according to a specific metric also described in the paper, allowing for more stable training at higher learning rates.\n\nThey encode images to image features using a CLIP VIT-L/14 model and use a [perceiver resampler](https://github.com/lucidrains/flamingo-pytorch) introduced in [Flamingo](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model) to pool the image features from `256 -> 64` tokens. The image features are combined with the token embeddings by adding them to the input sequence surrounded by special tokens `<image>` and `</image>`. An example is `<s> <image> image_features </image> text </s>`. This allows image(s) to be interwoven with text in the same sequence.\n\nWe follow the hyperparameters described in the paper visible in the following image:\n\n![KOSMOS-1 Hyperparameters](./hyperparams.png)\n\n## Details\n### Model (decoder)\nWe use the torchscale implementation of the decoder-only Transformer architecture from Foundation Transformers:\n    \n```python\nfrom torchscale.architecture.config import DecoderConfig\nfrom torchscale.architecture.decoder import Decoder\n\nconfig = DecoderConfig(\n    decoder_layers=24,\n    decoder_embed_dim=2048,\n    decoder_ffn_embed_dim=8192,\n    decoder_attention_heads=32,\n    dropout=0.1,\n    activation_fn="gelu",\n    attention_dropout=0.1,\n    vocab_size=32002,\n    subln=True,                 # sub-LN approach\n    xpos_rel_pos=True,          # rotary positional embeddings\n    max_rel_pos=2048\n)\ndecoder = Decoder(\n    config,\n    embed_tokens=embed,\n    embed_positions=embed_positions,\n    output_projection=output_projection\n)\n```\n\n\n### CLIP VIT-L/14\nFor the image model (CLIP VIT-L/14) we use a pretrained OpenClip model:\n\n```python\nfrom transformers import CLIPModel\nclip_model = CLIPModel.from_pretrained("laion/CLIP-ViT-L-14-laion2B-s32B-b82K").vision_model\n# projects image to [batch_size, 256, 1024]\nfeatures = clip_model(pixel_values=images)["last_hidden_state"]\n```\n\n### Perceiver Resampler\nWe follow the default hyperparams for the perceiver resampler as no hyperparams are given in the paper:\n\n```python\nfrom flamingo_pytorch import PerceiverResampler\nperceiver = PerceiverResampler(\n    dim = 1024,\n    depth = 2,\n    dim_head = 64,\n    heads = 8,\n    num_latents = 64,\n    num_media_embeds = 256\n)\n# projects image features to [batch_size, 64, 1024]\nself.perceive(images).squeeze(1)\n```\n\nBecause the model expects a hidden dimension of `2048`, we use a `nn.Linear` layer to project the image features to the correct dimension and initialize it according to Magneto\'s initialization scheme:\n\n```python\nimage_proj = torch.nn.Linear(1024, 2048, bias=False)\ntorch.nn.init.normal_(\n    image_proj.weight, mean=0, std=2048**-0.5\n)\nscaled_image_features = image_proj(image_features)\n```\n\n### Tokenizer\nThe paper describes a [SentencePiece](https://github.com/google/sentencepiece) with a vocabulary of `64007` tokens. For simplicity (as we don\'t have the training corpus available), we use the next best open-source alternative which is the pretrained [T5-large tokenizer](https://huggingface.co/t5-large) from HuggingFace. This tokenizer has a vocabulary of `32002` tokens.\n\n```python\nfrom transformers import T5Tokenizer\ntokenizer = T5Tokenizer.from_pretrained(\n    "t5-large",\n    additional_special_tokens=["<image>", "</image>"],\n    extra_ids=0,\n    model_max_length=1984 # 2048 - 64 (image features)\n)\n```\nWe then embed the tokens with a `nn.Embedding` layer. We actually use a `bnb.nn.Embedding` from\n[bitandbytes](https://github.com/TimDettmers/bitsandbytes) which allows us to use 8-bit AdamW later.\n\n```python\nimport bitsandbytes as bnb\nembed = bnb.nn.Embedding(\n    32002,          # Num embeddings\n    2048,           # Embedding dim\n    padding_idx\n)\n```\n\nFor positional embeddings, we use:\n```python\nfrom torchscale.component.embedding import PositionalEmbedding\nembed_positions= PositionalEmbedding(\n    2048,           # Num embeddings\n    2048,           # Embedding dim\n    padding_idx\n)\n```\n\nAlso, we add an output projection layer to project the hidden dimension to the vocabulary size and initialize it according to Magneto\'s initialization scheme:\n```python\noutput_projection = torch.nn.Linear(\n    2048, 32002, bias=False\n)\ntorch.nn.init.normal_(\n    output_projection.weight, mean=0, std=2048**-0.5\n)\n```\n\n### Decoder changes\nI had to make some slight changes to the decoder to allow it to accept already embedded features in the forward pass. This was necessary to allow the more complex input sequence described above. The changes are visible in the following diff in line 391 of `torchscale/architecture/decoder.py`:\n\n```diff\n+if kwargs.get("passed_x", None) is None:\n+    x, _ = self.forward_embedding(\n+        prev_output_tokens, token_embeddings, incremental_state\n+    )\n+else:\n+    x = kwargs["passed_x"]\n\n-x, _ = self.forward_embedding(\n-    prev_output_tokens, token_embeddings, incremental_state\n-)\n```\n\n\n---\n\n\n### Dataset Strategy\n\n\nHere is a markdown table with metadata for the datasets mentioned in the paper:\n\n| Dataset | Description | Size | Link | \n|-|-|-|-|\n| The Pile | Diverse English text corpus | 800 GB | [Huggingface](https://huggingface.co/datasets/the_pile) |\n| Common Crawl | Web crawl data | - | [Common Crawl](https://commoncrawl.org/) |  \n| LAION-400M | Image-text pairs from Common Crawl | 400M pairs | [Huggingface](https://huggingface.co/datasets/laion400m) |  \n| LAION-2B | Image-text pairs from Common Crawl | 2B pairs | [ArXiv](https://arxiv.org/abs/2112.05251) |\n| COYO | Image-text pairs from Common Crawl | 700M pairs | [Github](https://github.com/clovaai/coyo) |  \n| Conceptual Captions | Image-alt text pairs | 15M pairs | [ArXiv](https://arxiv.org/abs/2103.01950) |\n| Interleaved CC Data | Text and images from Common Crawl | 71M docs | Custom dataset |\n| StoryCloze | Commonsense reasoning | 16k examples | [ACL Anthology](https://aclanthology.org/W17-0906/) |\n| HellaSwag | Commonsense NLI | 70k examples | [ArXiv](https://arxiv.org/abs/1905.02875) |\n| Winograd Schema | Word ambiguity | 273 examples | [PKRR 2012](https://doi.org/10.24963/kr.2012/26) |\n| Winogrande | Word ambiguity | 1.7k examples | [AAAI 2020](https://arxiv.org/abs/1907.10641) |  \n| PIQA | Physical commonsense QA | 16k examples | [AAAI 2020](https://arxiv.org/abs/1911.11641) |\n| BoolQ | QA | 15k examples | [ACL 2019](https://aclanthology.org/N19-1246/) |\n| CB | Natural language inference | 250 examples | [Sinn und Bedeutung 2019](https://semanticsarchive.net/Archive/DlZGNjZm/) | \n| COPA | Causal reasoning | 1k examples | [AAAI Spring Symposium 2011](https://www.aaai.org/ocs/index.php/SSS/SSS11/paper/download/2418/2874) |\n| RelativeSize | Commonsense reasoning | 486 pairs | [ArXiv 2016](https://arxiv.org/abs/1602.00753) |\n| MemoryColor | Commonsense reasoning | 720 examples | [ArXiv 2021](https://arxiv.org/abs/2109.11321) |\n| ColorTerms | Commonsense reasoning | 320 examples | [ACL 2012](https://aclanthology.org/P12-2018/) |\n| IQ Test | Nonverbal reasoning | 50 examples | Custom dataset |\n| COCO Captions | Image captioning | 413k images | [PAMI 2015](https://doi.org/10.1109/TPAMI.2014.2366765) |  \n| Flickr30k | Image captioning | 31k images | [TACL 2014](https://aclanthology.org/Q14-1010/) |\n| VQAv2 | Visual QA | 1M QA pairs | [CVPR 2017](https://openaccess.thecvf.com/content_cvpr_2017/papers/Goyal_Making_the_V_CVPR_2017_paper.pdf) |  \n| VizWiz | Visual QA | 31k QA pairs | [CVPR 2018](https://openaccess.thecvf.com/content_cvpr_2018/papers/Gurari_VizWiz_Grand_Challenge_CVPR_2018_paper.pdf) |\n| WebSRC | Web QA | 1.4k examples | [EMNLP 2021](https://aclanthology.org/2021.emnlp-main.261/) |  \n| ImageNet | Image classification | 1.28M images | [CVPR 2009](https://doi.org/10.1109/CVPRW.2009.5206848) |\n| CUB | Image classification | 200 bird species | [TOG 2011](https://vision.cornell.edu/se3/wp-content/uploads/2013/03/CUB_200_2011.pdf) |\n\n\n----\n\n# Training\n\n* We\'re actively seeking cloud providers or grant providers to train this all-new revolutionary model and release it open source, if you would like to learn more please email me at kye@apac.ai\n\n----\n\n### TODO\n\n* Integrate flash attention inside the `torchscale/component/multihead_attention.py`\n\n* Integrate one write head is all you need\n\n* Look into integrating qk_norm\n\n* Look into integrating Falcon LLM model tokenizer if they allow special tokens\n\n* Prepare datasets, training strategies, and infrastructure for massive production level traning\n\n* Run tests and make sure trains well with all optimizations on small dataset\n\n',
    'author': 'Kye Gomez',
    'author_email': 'kye@apac.ai',
    'maintainer': 'None',
    'maintainer_email': 'None',
    'url': 'https://github.com/kyegomez/Kosmos-X',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.6,<4.0',
}


setup(**setup_kwargs)
