Spatial Guide: an AI agent that answers questions about what you are looking at

Overview

Spatial Guide is a self-initiated research prototype that lets a person wearing a headset look at an object or an area, ask a question out loud and get an answer about that specific thing. A vision model identifies what the user is looking at, a large language model composes an answer using only a curated knowledge base about the objects, and the answer is pinned to the object as a spatial annotation. We built it to understand what it takes to make a conversational agent trustworthy in a space, rather than to ship an assistant. It underpins our AI and XR service.

Challenge

Language models answer fluently about things they know nothing about. In a museum or a showroom that is a liability: a confident wrong date on a reconstructed statue is worse than no answer. The challenge had four parts: make the agent answer only from a vetted knowledge base and say so when it cannot; keep the exchange hands-free and natural in a headset; deal honestly with the latency of a vision model, a language model and speech synthesis in sequence; and respect the privacy of the camera frames the system needs to see.

Research

We mapped the pipeline latency first. Speech recognition, a vision call, a retrieval step, a language model response and speech synthesis each add time, and in sequence they sum to several seconds. We studied how people tolerate waiting when they can see that something is happening, drawing on conversational interface research and our note on AI in immersive experiences. We also tested grounding strategies: a model allowed to use general knowledge, a model constrained to retrieved passages, and a model required to cite the passage it used. Only the last two were candidates.

Concept

Ask the thing in front of you. The user looks at an object, says a question, and the answer appears beside the object with a thin line connecting the two, read aloud at the same time. If the question is about something outside the knowledge base, the guide says so plainly and offers what it does know. The space itself is the interface: no wake word, no menu, no tablet.

Design

Latency drove the design. The moment a question is heard, an annotation appears on the object showing a transcript of what was understood, so the user can see that the target and the question were right before the answer arrives. A subtle, non-looping indicator shows progress through recognition, retrieval and composition. Answers are short by default, with a "tell me more" gesture to expand. Refusals are designed as first-class responses: a distinct, calm phrasing and a short note on what the guide can answer about that object, so that a refusal does not feel like a failure. Turn-taking is hands-free and gaze-gated: the guide listens when the user looks at a recognised object and speaks, and stops listening when they look away for a few seconds. We relied on the ideas in our spatial UX note and on the broader principles of spatial computing.

Photo: Collision Conf · CC BY 2.0

Technology

The headset application is built in Unity with OpenXR and the Meta Quest SDK for passthrough, gaze direction and hand tracking. Speech recognition runs on the device where possible, and a lightweight on-device ML classifier gives an immediate first guess at the target object while a stronger vision model confirms it. A Node.js service orchestrates the pipeline: it receives a cropped frame and the transcript, retrieves passages from the curated knowledge base, calls a large language model with strict instructions to answer only from those passages or decline, and returns the answer with the identifiers of the passages used. Speech synthesis plays the answer spatially from the object's position.

Interaction

Look, ask, listen. The user looks at an object; a faint marker confirms it is recognised. They ask a question; the transcript appears, then the answer, pinned to the object and spoken. A pinch expands the answer; a palm dismisses it. Looking at another object and speaking starts a new exchange. The guide never speaks unprompted.

Implementation

Grounding is enforced in two places: the prompt, which forbids answers outside the retrieved passages, and a check on the returned passage identifiers, which hides any answer that cites nothing. Camera frames are cropped to the recognised object before leaving the device and are discarded after the vision call; nothing is stored, and a visible indicator shows when a frame is being sent. We built a small evaluation set of questions, including ones designed to tempt the model outside the knowledge base, and ran them repeatedly while tuning the prompt and the retrieval step. Latency masking was tuned by watching testers in the studio rather than by numbers alone.

Photo: waketechcc · CC0

Result

The prototype demonstrated that a conversational guide can be constrained to a curated knowledge base and still feel natural, that showing the transcript and target immediately makes a multi-second wait acceptable to studio testers, and that a well-designed refusal is read as honesty rather than as failure. It also showed where the approach is fragile: recognition of small or similar objects, questions that span several objects, and noisy rooms where speech recognition degrades. What remains open is evaluation with members of the public, multilingual behaviour, and the cost and reliability of the pipeline in a venue without dependable connectivity.

Lessons

Design the wait, do not hide it. Grounding needs enforcement in code, not only in the prompt. Refusal is an interaction to design with as much care as an answer. Privacy decisions are architectural: crop on device, send little, keep nothing, and show when the camera is in use. Gaze-gated turn-taking removes the wake word but needs generous timeouts. And a curated knowledge base is the product; the model is the interface to it. If you are weighing a conversational guide for a space, talk to us about what to prototype first.