IA.COMédia: A Loka-made Lab for Theatre with Live AI ActorsA Portuguese playwright wanted to push AI beyond the assistant role and onto the theatre stage. Here’s how LOKA built a live system that brings a cast of AI actors to perform alongside humans.

Last year, a Portuguese actor and playwright challenged Loka to embark on an unusual endeavor — he wanted to produce a lab-meets-stage experience fully centered around AI-human interaction. Not a mere scene with a human actor interacting with a Siri-like personal assistant. Not pre-recorded AI-generated speech. Oh, and not a movie nor a pre-screened show — a real, ticketed, live comedy play in Oeiras, Portugal, already scheduled to premiere 7 months ahead in a theater venue.

From inception, the playwright wanted to go as far as current-day AI technology could take us — and, with many, many questions in our minds — we dove deep and embraced the challenge.

This is a blogpost about engineering, art, how they can work together and the many, many times this team of engineers found themselves in an AI uncanny valley — we hope you enjoy it.

So… let me see if I got you straight — Gathering Requirements#

The first conversation between Loka and the theatre team happened over a nice espresso, sitting on a street balcony under the warm July sun. When we asked the playwright for his ideal scenario, he described an entirely new paradigm for live performance — human actors would be constantly speaking to each other and to AI actors, who would listen and respond accordingly — whatever accordingly would end up meaning.

First, we talked about latency (or speed, or silence). While most AI products can tolerate a pause, theater can’t, at least not accidentally. In a play, silence is emotional punctuation. It’s tension, or comedy, or discomfort. So one of our earliest nonnegotiables was responsiveness. Not “pretty fast” in a technical sense, but “fast enough that it feels like a character is present.”

Another minor detail: the play was to be performed in Portugal, in Portuguese — which made language an issue. We have all seen impressive communication abilities from commercial AI Voice systems, but they tend to be available in reduced sets of languages, with European Portuguese not making the cut too often, let alone in high-quality, stable formats.

There’s also the reality of runtime. Theatre happens… in person, in real time. We knew whatever system we built would have to hold up through a full play — something like 60 to 90 minutes — without crashing, drifting, or turning into a silent statue halfway through Act Two. This meant designing many fallback systems to allow mid-play adjustments or even full restarts if needed — by a new backstage operator equivalent to an AI stage manager.

Interacting with a control dashboard, the new AI Operator would be able to control the play’s flow and intervene if an AI latched onto the wrong thread, misunderstood a moment, or simply failed to hand off to the right character. No one wants to see a live tech support session on a Saturday night.

The Stumble-Through — Initial Prototype and Choosing a Platform#

Realistically, we were on a tight schedule — the play was set out to premiere in 7 months, and the whole production team would need to rehearse the play (which was, by then, still unwritten — it would evolve based on what the system could do). Our goal was to produce a prototype in about 1 month, which meant we had to prioritize our design choices and iterate fast.

Our first milestone was to build a simple system that could support voice conversations between a single human and a single AI model. With it, we wanted to test our main requirements — live responsiveness, Portuguese stable voices, and some adherence to conversation topics.

Although our instinct was to build the entire system from scratch, the reality of our deadline got us scrambling for a pre-built solution that could serve as a base in establishing streamed audio conversations. If we found such a tool, we could then focus on customizing the AI actors, help the playwright define the play’s narrative logic and develop any logistical requirements — we don’t always need to re-invent the wheel.

We explored OpenAI’s real-time options at the time, but ran into practical constraints around voice selection — especially for Portuguese accents — and model flexibility. In the end, we built on ElevenLabs because it gave us what this project required immediately: a large voice library (and the capacity to create our own custom voices), solid tooling, and a pipeline that supported real-time conversational use cases without us rebuilding the entire streaming stack.

So, we had the skeleton figured out — the actors would have microphones on stage, these microphones would route the audio stream into an open channel establishing connection to ElevenLabs’ SDK. When silence was detected, the AI actor(s) would generate their next line, which would then be broadcast in the theatre’s sound system, allowing the human actors to listen and reply.

Theatre is not a Call-Center#

We’ve all been there. We call some service to ask for help, and are greeted by something we can quickly tell is AI. There’s a rhythm to it — you speak, it listens, it responds. You speak again, it listens again, it responds again.

Under the hood, this pattern relies on something called turn detection: traditionally, AI systems need to wait for silence — for the person speaking to finish — before they can begin generating a response. It’s waiting for its cue. In a call center, this is perfectly fine. Customers don’t want to be interrupted by an AI while they explain their problem.

In theatre, this is a problem.

An AI actor that is strictly on cue — that can only enter the scene once the stage has gone completely quiet — has no agency over when it speaks, only over what it says. So if we couldn’t give our AI actors genuine instinct, we had to do the next best thing: fake it well enough that no one in the audience would notice.

A Bit of Movie Magic — The Mute Button#

If AI voice systems could only generate responses after detecting silence, then we had to find a way to fake this silence for the AI actors, while still allowing human actors to speak — remember, silence is awkward in theatre.

The operator could mute the human audio feed that the AI was listening to before the actor finished their sentence. The actor was still speaking physically on stage; the audience never knew. But the AI system “heard” silence earlier, started processing earlier, and responded faster. The result was a convincing illusion of interruption, and far fewer moments where the room waited for the machine to catch up.

It took a while for us, as an engineering team, to accept we were allowed to do this sort of “movie magic” — we initially felt like we were cheating, since typical enterprise applications do not have room for this type of system design — there, the goal is automation and scale, so there is no way we could include a human by default.

We had to quickly accept that our goal here wasn’t to build a perfectly autonomous system, it was to build a perfect way to entertain the spectator, even if that means letting go of purist notions and pulling a trick — something that has always been done in cinema and theatre.

So, we were set to build a system that was expecting human intervention — to mimic this illusion of interruption — as a way for us to get the intended creative effect with the technology we had at our disposal.

The Show Must Go On — The Operator Screen#

With the play’s script taking shape, we now had a prototype being built into a stable version, a proto-script if you will, and a team of actors and producers eager to get their hands dirty. Our goal was to make the theatre team as independent as possible during rehearsals and the actual sessions, once the play premiered.

We also knew we’d need an interface to control the system in case of malfunction. The “mute” fix to mimic interruptions. Do you see where we’re going?

All of this was calling for us to bring out our best design skills to the table. If this was already a multi-disciplinary project, the decision of building a visual interface opened the doors for even more fun and collaboration between Loka teams.

The design challenge here was to find ways to support different usage patterns. We knew this visual interface would be used by one AI operator — yes, we accidentally created a new job title — both during rehearsals and actual sessions. This was unusual for us because the interface had only one user and one environment: a single operator running a live performance.

In rehearsal mode, the system needed to be flexible and forgiving. The team had to jump between scenes, force characters to speak, and edit scripts quickly as rehearsals revealed what worked and what didn’t.

In action mode, the system needed discipline. Scenes followed the final order. Scripts were locked. Controls were tighter. The UI stayed clean because, during a live show, “clean” is not aesthetics — it’s safety.

All of the fallback mechanisms we ended up building were available and especially relevant in action mode. If the conversation suddenly crashed, the AI operator would turn it back up. If the wrong agent spoke (we’ll cover that soon enough), the AI operator would fix it. But the role of the AI Operator wasn’t just a safety net. It was timing, judgment, and rehearsal. During improvised sections, especially, muting required an intuitive sense of when a thought was complete enough for the AI to respond, and when cutting too early would break meaning.

Two Humans and Five AIs walk into a bar… Prompting the AI Actors#

By the time we were done with our first prototype, the playwright was able to start outlining the play’s narrative. He had a concept in mind: a comedy built around two neighbors talking from their balconies. Set in a near-future, these characters would interact with AI assistants who were somehow fully integrated into their lives — suggesting, advising, and adjusting their humans into enhanced versions of themselves — all under a guise of efficiency.

Our challenge then was — how do we build AI actors and instruct them to follow a narrative, a set of character personalities? And how would the interaction between the humans on stage and the AI actors take place?

The playwright said it outright — the play should be different each time they ran it. Well, but different how? He envisioned that AIs should have ways to improvise their appearances in the story — if they didn’t, then why were we using live AI and not a recording? Yet, he wanted the play to have a skeleton, a narrative and a point it was trying to convey. This is obviously a hard balance to even define, let alone successfully implement.

The answer was — let’s play with it and see what we can do with these AI actors.

To Improv or Not To Improv#

Our initial architecture expected a play script, split into scenes, which would then be embedded into each AI actor’s system prompt. Alongside the script, each agent received a set of personality instructions uniquely crafted for their role in the play — the subservient AI assistant, the perpetually complaining AI of the upstairs neighbor, and the controlling, megalomaniacal AI hell-bent on taking over the building.

Each AI actor was prompted to deliver their pre-written line and then add a touch of their own character to it — following the narrative structure, but with room to improvise. At the end of each scene, the playwright would let the conversation drift without explicit instructions, opening the floor to fully unscripted exchanges.

This was the version that went live. A month passed. The play ran on Fridays and Saturdays, the audience left the room happy and engaged, the human actors found their footing. By any measure, it was working.

The playwright, naturally, wanted to push further.

So, the playwright had yet another challenge for us: can we make it even more unpredictable? So, we removed most of the pre-written AI lines from the script. Instead of telling the AI actors what to say, we left those lines deliberately empty — a fill-in-the-blanks challenge. Each agent was given the context of the scene and a sense of their character’s role within it, but the actual words were left entirely to them. The script became a skeleton; the AI actors had to find the meat.

The effect on stage was immediate. The human actors were never quite sure what was coming next, and the play became a live improvisation exercise, deepening the kind of AI-theatre-lab experimental energy the playwright had been chasing from the start.

What this version revealed, though, was that while Large Language Models (LLMs) can improvise quickly, fast isn’t the same as good, or interesting. LLMs are, at their core, sophisticated next-word prediction engines, and that shows when you ask them to be surprising. The AI actors could fill silence and give pertinent answers to the human actors’ cues, but they couldn’t lead a scene. They couldn’t feel when a moment needed to be stretched, subverted, or handed off. That job — the real creative steering of the play — remained entirely with the human actors.

So, version 2 wasn’t just a prompt upgrade. It was the experiment clarifying itself: AI brings speed and consistency, but the soul of the performance still lives with the humans on stage.

Wait, are we doing Karaoke?! — The AI Transfer Tool#

Once you decide you want multiple AI characters, you hit a staging problem that’s less “ML” and more “directing”: how do you make five AI agents share a conversation without talking over each other, going quiet, or stepping on the scene?

We considered a rigid keyword-based routing system. Hear “report,” trigger the “leader” agent. Hear “politics,” trigger a different one. It worked in a technical sense, but it felt uncomfortably mechanical. The playwright wanted something that could flow like performance, not like an if/then tree.

So we went with agent-driven handoffs. We gave agents a tool that let them transfer the “microphone” to another agent when it wasn’t their turn. The play was written as a scripted skeleton, split into scenes, and each agent received the relevant scene script as part of its prompt. The agent listened to what just happened, inferred where the scene was, and decided whether it should speak next. If not, it transferred control to the right agent. If yes, it delivered the next beat.

Splitting the script into scenes turned out to be one of the biggest stability wins. The show might run for an hour, but we didn’t treat it as one hour-long conversation. We treated it as several smaller conversations, each with a reset point and tighter control. That reduced drift, reduced chaos, and gave the operator more predictable breakpoints.

Beyond the Code — Engineers as Screenwriters#

Throughout the project, we worked closely with the play’s creative team to make sure the comedy served a purpose beyond the laughs. The goal was to make people think — and that required keeping the AI characters’ behavior tethered to something plausible, not cartoonish.

At a certain point, the technical team became unofficial screenwriting consultants, nudging the script away from the spectacular (an AI autonomously hacking into a country’s central bank) and toward the quietly unsettling (a government outsourcing its national strategy to an AI system).

That shift mattered to us. This project was, unexpectedly, a rare opportunity to speak to a general audience as engineers — not through a white paper or a conference talk, but through a comedy show on a Friday night in Lisbon. And it made us think seriously about our own responsibility in how we communicate what we actually know. It’s easy to stay behind a screen and then feel frustrated when people don’t understand the real risks of AI — the undramatic, structural ones. If we had a stage, we were going to use it.

Custom Voices & Expression Instructions#

A big concern for the playwright was the voices of the AI actors — after all, it was the main way the audience would infer which AI character was speaking. The characters should have distinct voice pitches, different levels of raspiness and eagerness in their voices. Oh, and they all needed to speak European Portuguese.

This is where our choice of platform — ElevenLabs — proved to be really helpful, since it had a wide voice library, and also allowed custom voice creation from relatively short recordings.

At one point we generated a test voice from about 90 seconds of audio. It sounded so convincingly like the original speaker that the reaction was half laughter, half discomfort. Hearing “yourself” say things you’ve never said is a uniquely surreal experience, and it’s the kind of moment that makes you appreciate how seriously voice consent has to be treated — especially in public-facing work.

The next challenge was instructing voice expressions to our AI actors — instructing how a line or moment should be delivered is crucial in theatre. While we initially assumed we’d shape tone with SSML-style markup, we had to use a faster proprietary model (from ElevenLabs) that didn’t support that approach, due to latency reasons.

So, we tuned what we could: speaking speed, stability (how variable each generation could be), and similarity (how tightly the output matched the original voice sample). Those parameters were the ones exposed by ElevenLabs and ended up being our emotional controls. With the right tuning, an agent could feel calmer or sharper, more chaotic or more measured — without needing explicit markup in every line.

A Character in Every Color#

Voice alone can carry a lot — but in a room full of people watching a stage, it helps to have somewhere to look. So, besides having their own voice and personalities, each AI actor in IA.Comédia had their own color.

We used ElevenLabs’ Orb component as a visual presence for the AI characters — a pulsing interface element displayed on screen that the audience could anchor to while listening. Each agent was assigned a color, and whenever the speaking character changed, so did the orb.

The interesting part was getting the orb to actually know who was speaking. ElevenLabs handles agent-to-agent transfers internally, which meant we needed a way to surface that state to our own server — the one running the operator screen and managing the conversation pipeline. The solution was a client-side tool: a function that lives and executes on our infrastructure, but that the ElevenLabs agent itself can invoke. Whenever a transfer happened, the newly active agent would call the tool, passing its own agent ID. Our server would catch that, look up the corresponding color, and update the orb accordingly.

Do You Believe?#

In a pre-premiere showing, an audience member pulled us aside with a question that stopped us cold: “Is this really AI, or is it prerecorded?” All our work on latency, on voice quality, on keeping the system alive through a full show — and the result was too polished to feel real. We flew too close to the sun and engineered ourselves into a credibility problem.

The answer was to open the system up to direct contact. In the foyer before the show, the audience could approach a microphone and speak directly to LOKA — the receptionist AI character — getting their first taste of the system before the curtain even went up. During the performance itself, a microphone handoff let audience members address the characters on stage.

Somewhere in this process we stumbled onto something that cuts against most engineering instinct — a small, visible failure can do more for believability than a seamless experience ever could. A system that occasionally trips over itself feels present. A system that never does starts to feel like a recording.

The greatest artistic payoff happened after the curtain came down. At the exit door, we had Anathol — the play’s antagonist AI — responding to the audience as they filed out. Some people were curious. Some were genuinely irritated. One person muttered “You AIs should just leave us alone”; another said “I want you to make my life easier, not control it.”

They weren’t talking to an LLM. They were talking to a character — one they’d formed a real opinion about. For a team of engineers who’d spent months obsessing over latency and voice pipelines, that was a strange and wonderful thing to witness.

A Round of Applause!#

Speaking of believing, this testimonial wouldn’t be complete without thanking the creative team who worked alongside LOKA to bring this play to life, and believed in our ability to deliver a working system in a short amount of time.

Paulo Matos — playwright, creative director, and the actor behind ARY — was the one who pointed us toward the moon in the first place. His belief that AI technology and live performance could speak to each other was the foundation everything else was built on.

Carlos D’Almeida Ribeiro, who manages Teatro Independente de Oeiras, was the one who said yes — taking Paulo’s vision seriously enough to put a stage behind it, and bringing VLAD to life in the process.

Gonçalo Lima was the AI Operator — muting mics, switching scenes, and holding his nerve through every live performance so the rest of us didn’t have to. Tomás Palma and Catarina Jalles kept the lights and sound running while cheerfully fielding every technical question we threw at them from the control booth during rehearsals.

And to Teatro Independente de Oeiras — thank you for opening your doors to us. You gave a team of engineers a stage, and that turned out to mean more than we expected.

Originally published on Loka Engineering on Medium.