---
title: Introducing RealtimeAgent Capabilities in AG2
authors: [marklysze, sternakt, davorrunje, davorinrusevljan]
tags: [Realtime API, Swarm]

---

<iframe width="560" height="315" src="https://www.youtube.com/embed/NbZW2goinKM?si=Lp3Znzz2aIMfsUNp" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

**TL;DR:**
- [**`RealtimeAgent`**](/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent) is coming in the AG2 0.6 release, enabling real-time conversational AI.
- Features include real-time voice interactions, seamless task delegation to Swarm teams, and Twilio-based telephony integration.
- Learn how to integrate [Twilio](https://www.twilio.com/) and [**`RealtimeAgent`**](/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent) into your swarm in this blogpost.

### **Realtime API Support: What's New?**

We're thrilled to announce the release of [**`RealtimeAgent`**](/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent), extending AG2's capabilities to support **real-time conversational AI tasks**. This new experimental feature makes it possible for developers to build agents capable of handling voice-based interactions with minimal latency, integrating [OpenAI’s Realtime API](https://openai.com/index/introducing-the-realtime-api/), [Twilio](https://www.twilio.com/) for telephony, and [AG2’s Swarm orchestration](/docs/user-guide/advanced-concepts/orchestration/swarm/deprecation).

### **Why Realtime API Support Matters**

import RealtimeAgent from "/snippets/advanced-concepts/realtime-agent/twilio.mdx";

<RealtimeAgent/>

### **Caveats and Future Improvements**

While the [**`RealtimeAgent`**](/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent) and Swarm integration is a powerful tool, there are a few things to keep in mind as we continue to refine the system:

- **Work in Progress**: The agent is still evolving, and we’re actively polishing its details for a smoother experience in the coming weeks.
- **Transcription Challenges**: Occasionally, the agent may mishear inputs, particularly when dictating complex information like flight numbers or letters.
- **Response Misdirection**: At times, the agent might respond directly instead of relaying information to the customer. We’re addressing this with prompt optimizations.
- **Simpler Setup Coming Soon**: Setting up Twilio can be time-consuming, but we’re developing a **LocalAdapter** that will let you interact with the agent directly via your web browser audio—perfect for quick testing without the long setup.

We’re excited about what’s to come and look forward to your feedback as we refine and expand these capabilities!

For more updates, tutorials, and discussions, join our [Discord community](https://discord.com/invite/pAbnFJrkgZ).

---
