# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['citizenk']

package_data = \
{'': ['*']}

install_requires = \
['certifi>=2022.12.7,<2023.0.0',
 'confluent-kafka[schema-registry]==2.0.2',
 'fastapi-utils>=0.2.1,<0.3.0',
 'fastapi>=0.92.0,<0.93.0',
 'httpx>=0.23.3,<0.24.0',
 'websockets>=10.4,<11.0']

setup_kwargs = {
    'name': 'citizenk',
    'version': '0.1.47',
    'description': 'An async Kafka Python Framework based on FastAPI and Confluent Kafka',
    'long_description': '# CitizenK\n**CitizienK** is a simple but powerful Python Library for developing reactive async Kafka microservices, built on top of [Confluent Kafka Python](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html), [FastAPI](https://fastapi.tiangolo.com/) and [Pydantic](https://docs.pydantic.dev/).\n\n**CitizenK Replicator** is an additional tool that we developed using the same technology to simplify data transfer between production and staging environments. It\'s not a substitution for Confluent\'s replicator which is a much more robust tool for replicating data between multiple production environments\n\n------------------------------------------------------------------------\n------------------------------------------------------------------------\n\n## How we got here...\nWe are a Python shop. We develop web services in Python, write ETL code in Python, and obviously use Python for data science. A few years ago, when we started working on Lanternn, we were looking for a python library that can help us build distributed, scalable processing pipelines on top of Kafka with stateless and stateful microservices. The best solution that we could find at the time was Faust. Faust is a stream processing library, porting ideas from Kafka Streams to Python. Additionally, Faust includes a powerful web server, schema validation and management, and is based on an agents/actors approach. With so many goodies, how could we resist?\n\n\nWe built dozens of services using Faust, and all in all, we were pretty happy with it. however, over time we realized that Kafka Streams is not really working for us, it\'s just too complex to manage and other alternatives for state management like Redis are simpler. We were also worried that Faust without the spirit of its creator Ask Solem, and its underlying Kafka library: aiokafka and python-kafka don\'t have a large enough community to support the stability issues that we had experienced. In parallel, frameworks like FastAPI and Confluent Kafka that we used anyway had large community backing, and we felt like we can use them together to build an alternative foundation for our pipelines.\n\n\nThe name CitizenK was chosen to reflect the idea that Python should be a first-class citizen in the Kafka world.\n\nIt is also related to Kafka\'s most famous novel: The Trial, which tells the story of Josef K. a man arrested and prosecuted by a remote, inaccessible authority, with the nature of his crime revealed neither to him nor to the reader.\n\n## Existing tools\n- Faust\n- Fastkafka\n\n------------------------------------------------------------------------\n\n## Tutorial\n\nYou can see an example of how to use CitizenK in the demo app\n\n### Creating a CitizienK app\n\nFirst, we create a CitizenK app similar to how we create a FastAPI app, but with additional arguments:\n\n- kafka_config: provides configuration for connecting and configuring the Kafka client\n- app_name: Mainly used as the consumer group name\n- app_type: SINK (consumer only), SOURCE(producer only) or TRANSFORM(producer-consumer)\n- auto_generate_apis: Will auto generate FastAPI to consume and produce to workers and topics\n- agents_in_thread: Will run the consumer agents in a thread and not in an async loop\n\n``` python\napp = CitizenK(\n    kafka_config=config.source_kafka,\n    app_name="citizenk",\n    app_type=AppType.TRANSFORM,\n    debug=True,\n    title="CitizenK Demo App",\n    auto_generate_apis=True,\n    agents_in_thread=config.AGENTS_IN_THREAD,\n    api_router_prefix=prefix,\n    api_port=config.API_PORT,\n    schema_registry_url=config.KAFKA_SOURCE_SCHEMA_REGISTRY,\n    version=config.VERSION,\n    openapi_url=prefix + "/openapi.json",\n    docs_url=prefix + "/docs",\n    license_info={\n        "name": "Apache 2.0",\n        "url": "https://www.apache.org/licenses/LICENSE-2.0.html",\n    },\n)\n```\n\n### Creating CitizienK topics\nNext, we create topics for the app and define the model for the topics using Pydantic\n\nTopics can be either INPUT, OUTPUT or BIDIR\n\n``` python\nclass Video(JSONSchema):\n    camera_id: int\n    path: str\n    timestamp: datetime\n\n\nclass ProcessedVideo(JSONSchema):\n    camera_id: int\n    path: str\n    timestamp: datetime\n    valid: bool\n\n\nt1 = app.topic(name="B", value_type=Video, topic_dir=TopicDir.BIDIR)\nt2 = app.topic(name="C", value_type=ProcessedVideo, topic_dir=TopicDir.BIDIR)\nt3 = app.topic(name="D", value_type=ProcessedVideo, topic_dir=TopicDir.OUTPUT)\n```\n\n### Creating CitizienK agents\nAnd lastly, we create gents that process the Kafka messages.\n\nAgents can listen to multiple topics and accept either values or the entire Kafka event (key, value, offset, partition, timestamp...)\n\n``` python\n@app.agent(topics=t1, batch_size=100)\nasync def process_videos_t1(events: List[KafkaEvent]):\n    # Process incoming video\n    for event in events:\n        camera_id = event.value.camera_id\n        video_counts[camera_id] += 1\n        v = ProcessedVideo(\n            camera_id=camera_id,\n            path=event.value.path,\n            timestamp=event.value.timestamp,\n            valid=bool(camera_id % 2),\n        )\n        t2.send(value=v, key=str(v.camera_id))\n\n\n@app.agent(topics=t2, batch_size=100)\nasync def process_videos_t2(values: List[BaseModel]):\n    # Process incoming video\n    for value in values:\n        if value.valid:\n            t3.send(value=value, key=str(value.camera_id))\n\n```\n\n### Auto endpoints\nTo help debug and evaluate the service, CitizenK automatically creates web endpoints that help you send messages to topics and agents.\n\n- info: get service info\n- topics: send events to topics\n- agents: send events directly to agents, bypassing topics\n- stats: get Kafka stats for producer and consumer\n\n![CitizenK Demo API](docs/citizenk_demo_api.jpg)\n\n\n### Creating additional CitizienK endpoints\nJust like any other FastAPI app, you can create get, post and put endpoints that either interact with Kafka or perform some other tasks, non Kafka related\n\n``` python\n@router.post("/events", response_class=JSONResponse)\nasync def produce_video_events(\n    values: List[Video],\n    topic: str = Query(),\n):\n    """Sends events to the given topic"""\n    if topic not in app.topics:\n        raise HTTPException(status_code=400, detail="Topic not supported by app")\n    t = app.topics[topic]\n    for v in values:\n        t.send(value=v, key=str(v.camera_id))\n    return {"status": "ok"}\n\n\n@router.get("/topics", response_class=JSONResponse)\nasync def get_source_topics():\n    """Returns the list of topics from the source kafka"""\n    admin = KafkaAdapter(config.source_kafka)\n    topics = sorted(list(admin.get_all_broker_topics().keys()))\n    return {"topics": topics}\n```\n\n### Multiple workers behind a load balancer\nCitizenK includes two special decorators for scenarios where the service has multiple workers behind a load balancer and the web request needs to reach a specific worker that holds a partition.\n\n- topic_router: forwards the request based on the topic and key (JSON / HTML)\n- broadcast_router: aggregates the responses from all workers into a single JSON\n\nBoth routers support GET, POST, PUT and DELETE commands\n\n``` python\n@router.get("/topic_test", response_class=JSONResponse)\n@app.topic_router(topic=t1, match_info="camera_id")\nasync def test_topic_router(request: Request, camera_id: int):\n    """Returns the list of groups from the target kafka"""\n    return {"key": camera_id, "count": video_counts[camera_id]}\n\n\n@router.get("/broadcast_test", response_class=JSONResponse)\n@app.broadcast_router()\nasync def test_broadcast_router(request: Request):\n    """Returns the list of groups from the target kafka"""\n    return video_counts\n```\n\n### Websocket\nCitizenK support for Websocket agents\n\n``` python\n@app.agent(topics=t2, batch_size=100, websocket_route=prefix + "/ws")\nasync def websocket_agent(values: List[BaseModel]) -> str:\n    values = [json.loads(v.json()) for v in values if not v.valid]\n    return json.dumps(values, indent=4)\n```\n\nThis agent exposes a WebSocket endpoint for one or more clients to connect to. It then processes incoming Kafka messages from topic t2 and sends the returned string value to all the existing live WebSocket "/ws" connections. The main use case for this is to bridge between Kafka and Websocket. One possible use case for this feature is to send filtered Kafka events to a web app or mobile app.\n\n\nThe other direction frontend --> Kafka is probably easier to implement with a normal REST post endpoint and is not supported yet.\n\n## Things to be aware of...\nCitizenK is a single-threaded async app. i.e. If a coroutine spends too much time in processing without awaiting IO, it will block other coroutines from running. Specifically, when using a load balancer with health checks, it\'s important to pay attention to the time between health checks and see that it\'s higher than the longest-running agent. Fixed using:agents_in_thread\n\n\nTo help tune the service. CitizenK includes the concept of batch size:i.e. how many events to consume and process every batch across all agents.\n\nAdditionally like any other Kafka service. it\'s important to tune several kafka [consumer](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#fetch-max-bytes) and [producer](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html) configs. Specifically ensure rebalancing is not triggered unintentionally:\n\n\nConsumer:\n- fetch.max.bytes (50 Mbytes): The maximum amount of data the server should return for a fetch request. Reduce if processing each record takes significant time.\n- max.poll.records(500): The maximum number of records returned in a single call to poll().\n- max.poll.interval.ms (5 min): The maximum delay between invocations of poll() when using consumer group management\n\nProducer:\n- linger.ms(0): Important to set to 0/5/10/50/200 on moderate/high load\n- batch.size(16K): Increase if sending large buffers to Kafka\n\nBoth:\n- compression.type(none): gzip, snappy, or lz4\n\nMore explanation in here: [Solving My Weird Kafka Rebalancing Problems & Explaining What Is Happening and Why?](https://medium.com/bakdata/solving-my-weird-kafka-rebalancing-problems-c05e99535435)\n\n## Limitations\nAt the moment the library only supports JSON schema\n\n------------------------------------------------------------------------\n\n\n# CitizenK Replicator\n## Scenarios\n\n### Staging environment\n1. I have a staging environment and I want to replicate some production topics to it\n\n2. At some point I want to produce the staging topics in the staging environment using a staging service. So I switch off the replication and populate the same staging topic with real data.\n\n3. When I finish the testing in staging, I want to switch back to production, so that I can save on costs.\n\n4. If the workload is high, I want to replicate most (i.e. 90%) of the messages from production and only produce just a little (i.e. 10%) of the data from staging. This way in the same topic, I will have mixed data and potential schema from the two environments\n\n5. When switching between environments (i.e. on configuration change), I want to change the offset to the latest on the new topic, so that the handover is not too chaotic\n\n6. I also want to delete the consumer group of the service in staging, so that when it come back up again, it won\'t see a lag.\n\n7. Additionally sometimes, I want to migrate data between production and staging due to schema change or different identities.\n\n![Replicator](docs/replicator.jpg)\n\n### Dev environment + live data\n1. When I test a service locally or in a dev environment, possibly with a local Kafka, I want the local Kafka to have real data, so that I can test the service for a long period of time with live data.\n\n2. Theoretically, I can connect the dev service to the staging or production Kafka cluster, however, this presents a stability/security risk to the remote cluster. There is also a risk that the service will join a consumer group and participate accidentally in the remote workload. This approach also prevents parallel testing as there can be a conflict between the consumers.\n\n3. So one solution would be to replicate the topics from staging to the local/dev Kafka, maybe with some filtering to reduce the load, so that the local service is not overwhelmed with too much data\n\n### On premise kafka -- cloud kafka bridge\n1. I have a local Kafka and I want to replicate some topics to the remote cloud\n\n2. You can use this tool for this scenario, however Confluent replicator or Kafka MirrorMaker are probably more suitable\n\n### Dev environment + replayed data\n1. When I test a service locally or in a dev environment, possibly with a local Kafka, I want the local Kafka to replay historical/simulated messages from a file.\n\n2. this scenario is a bit different from the previous ones, as there is no Kafka consumer, just a producer. And you can say that it is more of a tool than a service.\n\n3. The messages are read from a file with a timestamp (one file for each topic), and injected into the right topic with the correct timing keeping the same gap between now, and the initial timestamp.\n\n\n### Cluster -- Cluster replication\n1. You can use this tool for this scenario, however Confluent replicator or Kafka MirrorMaker are probably more suitable\n\n## Existing tools\n1. Confluent replicator: Looks like a good tool, but not open source, expensive\n2. Kafka Mirror Maker: Open source but doesn\'t support filtering\n3. kcat: Nice tool, but not for these scenarios\n\n\n## Implementation details\n1. Using containerised python\n2. Based on Confluent Kafka API + FastAP\n3. Does not create the topics or partitions automatically. It assumes they exists and configured\n3. Deployed as a distributed service\n4. Filter based on JMESPath for JSON messages\n5. Allow two consumer options: with consumer group, without consumer group\n6. write code following DDD principles\n\n## Configuration\n- LOG_LEVEL: service log level\n- JSON_LOGGING: Use json logging\n- API_PREFIX: API prefix\n- FILE_DATA_PATH: Location of json files when reading and writing topics from file\n- KAFKA_SOURCE_SERVER_URL: Source Bootstrap Servers\n- KAFKA_SOURCE_USE_TLS: Enable Source SSL: 0,1\n- KAFKA_SOURCE_SASL_MECHANISM: Source SASL mechanism: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512\n- KAFKA_SOURCE_SASL_USERNAME: Source SASL username\n- KAFKA_SOURCE_SASL_PASSWORD: Source SASL password\n- KAFKA_SOURCE_GROUP_NAME: Source group name, or leave empty to consume without a consumer group\n- KAFKA_SOURCE_EXTRA_CONFIG_<KAFKA_CONFIG_KEY>: Any valid kafka consumer config (uppercase, replace . with _)\n- KAFKA_TARGET_SERVER_URL: Target Bootstrap Servers\n- KAFKA_TARGET_USE_TLS: Enable Target SSL\n- KAFKA_TARGET_SASL_MECHANISM: Target SASL mechanism: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512\n- KAFKA_TARGET_SASL_USERNAME: Target SASL username\n- KAFKA_TARGET_SASL_PASSWORD: Target SASL password\n- KAFKA_TARGET_EXTRA_CONFIG_<KAFKA_CONFIG_KEY>: Any valid kafka producer config (uppercase, replace . with _)\n- READ_MAPPINGS_EVERY_SECONDS: How often to check for new mappings in the file system\n- CACULATE_STATS_EVERY_SECONDS: How often to calculate stats\n- DELETE_GROUPS_EVERY_SECONDS: How often to check for new group deletion\n\n## Current solution limitations\n1. Currently only supports JSON schema.\n\n## API\n[API Description](docs/replicator_openapi.md)\n\n![Replicator API](docs/replicator_api.jpg)\n\n## User Interface\n![Replicator User Interface](docs/replicator_ui.jpg)\n\nThe user interface allows you to add a new mapping and edit/delete an existing mapping.\n\n## Usage\nProvide a JSON list of topic mappings in this format:\n```json\n[\n    {\n        "name": "File A to B",\n        "source_topic_name": "A",\n        "target_topic_name": "B",\n        "source_is_file": true\n    },\n    {\n        "name": "Topic B to C",\n        "source_topic_name": "B",\n        "target_topic_name": "C"\n    },\n    {\n        "name": "Topic C to D Using filter",\n        "source_topic_name": "C",\n        "target_topic_name": "D",\n        "valid_jmespath": "key == \'hello\' && value.msg == \'world\'",\n        "enabled": true\n    },\n    {\n        "name": "TopicCtoD",\n        "source_topic_name": "C",\n        "target_topic_name": "D",\n        "enabled": false,\n        "target_service_consumer_group": "service"\n    },\n    {\n        "name": "Topic D to File E",\n        "source_topic_name": "D",\n        "target_topic_name": "E",\n        "target_is_file": true\n    },\n    {\n        "name": "Disabled Topic U to File V",\n        "source_topic_name": "U",\n        "target_topic_name": "V",\n        "target_is_file": true,\n        "enabled": false\n    }\n]\n```\n\n- name: The mapping name\n- enabled: Enable / disable mapping\n- source_topic_name: The topic to read from in the source cluster\n- target_topic_name: The topic to write to in the target cluster\n- valid_jmespath: filter criteria\n- source_is_file: If the source is a json file\n- target_is_file: If the target is a json file\n- target_service_consumer_group: The service consumer group to delete when replication is enabled\n\n## Topic Level Mapping\nTopic level mappings allows mapping of key/values when replicating a topic. This might be useful if for example the schema / enums / keys are different between the environments\n\nTo support this, the replicator supports value mappings for each topic that it consumes from the source in the following JSON format:\n\n```json\n{\n    "key":{\n        "1":10,\n        "2":12\n    },\n    "value.payload.product_id":{\n        "1001":1,\n        "1002":12,\n        "1003":14\n    },\n    "value.payload.user_name":{\n        "A":"A name",\n        "B":"B name",\n        "C":"C name"\n    },\n    "partition":{\n        "0":0,\n        "1":0,\n        "2":0,\n        "3":1,\n        "4":1,\n        "5":1,\n    }\n}\n```\n\n## Stats\nReturns a list of JSON stats for each mapping in the following format:\n```json\n{\n    "time": "2023-05-25 18:20:43.875557",\n    "started": "2023-05-25 08:08:35.728313",\n    "queue": 180,\n    "mappings": [\n        {\n            "name": "Topic B to C",\n            "source_topic_name": "B",\n            "target_topic_name": "C",\n            "valid_jmespath": null,\n            "target_service_consumer_group": null,\n            "consumer_group_up": false,\n            "assignments": [0,1,2],\n            "lag": 1258,\n            "source_count": 27739,\n            "target_count": 27739\n        }\n    ]\n}\n```\n\n## Grafana Integration\nTo view the stats in Grafana, use the Infinity data source with the following settings:\n\n![Replicator Grafana Interface](docs/replicator_grafana.jpg)\n\n## Consumer API\nTo simplify debug and to support other use cases, the replicator also includes an end point to consume messages from a given topic.\n\n\n## License\n[Apache License v2.0](https://www.apache.org/licenses/LICENSE-2.0)\n',
    'author': 'Valerann',
    'author_email': 'info@valerann.com',
    'maintainer': 'Valerann',
    'maintainer_email': 'info@valerann.com',
    'url': 'https://pypi.org/user/valerann/',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.7,<4.0',
}


setup(**setup_kwargs)
