All work · Retrieval and agents
The Lantern
Voice ordering for restaurants
- Built with the Pyrium team
- Team of 3
- September 2026
- AssemblyAI Voice Agent Hackathon · lablab.ai
What it is
A voice maître d' for restaurants, built by a team of three at the AssemblyAI Voice Agent Hackathon. It takes a whole table's order by voice, checks every change against the live menu, and sends nothing to the kitchen until the guest places the order — on local models. The order flow is tested end to end in English and Spanish.
Evaluation
- Correct orders
- 10 of 10
- team result — six-turn ordering scenario, direct runs, qwen3:4b on an RTX 3060
- Full voice sessions with the correct basket
- 5 of 5
- team result — synthetic 16 kHz speech streamed in real time, five sessions on one machine
- Model time after the final transcript, median
- 1,341 ms
- team result — the same five sessions
- Time to first audio
- 42.9 s → 5.8 s
- average over a six-turn conversation with real AssemblyAI, Gemini and Cartesia — the first version, since replaced; Yoshio's rate-limiter fix
How it works
A small model proposes one intent per turn; code owns the order and the conversation state
Speech is transcribed by AssemblyAI's streaming model. A local Qwen3 4B model on Ollama returns one intent per turn in a fixed JSON format; code then resolves references such as "those two" or "that instead", applies the change to the order, and builds the reply from templates, which Kokoro speaks locally. Conversation state lives in SQLite rather than in the prompt, so the prompt stays about the same size every turn. Most of the code is a teammate's; Yoshio's part is the dialogue logic and reliability — the intent schema, reference resolution in code, explicit order placement, and the first version's rate-limiter fix.