Object Stories: a mixed reality guide prototype for museum collections

Overview

Object Stories is a self-initiated mixed reality prototype for museum collections. A visitor wearing a Meta Quest 3 looks at an object on a plinth; the headset recognises it and anchors a short story, an exploded view and a provenance timeline to the object itself, in passthrough, with spatial audio and text in the visitor's language. We built and tested it in the studio with replica objects on plinths. It is the reference project for our mixed reality and cultural heritage services.

Challenge

Museum interpretation competes with the object for attention. Wall labels pull the eye away, audio guides are linear, and tablet AR puts a screen between visitor and artefact. The challenge was to keep the object at the centre: content should appear where the object is, respond to where the visitor looks and stands, and disappear when it is not wanted. It also had to be editable by a curator rather than a developer, and realistic for a loan-device programme in a real gallery.

Research

We reviewed how museums currently use AR and XR, which became our note on museums and AR/XR, and listed what a label cannot hold about an object by going through published interpretation guidelines and our own gallery visits. The recurring gaps were the inside of the object, where it came from and what it looked like when new. We also tested recognition approaches side by side: image targets on the plinth, object recognition from a photogrammetry model, and a hybrid where the plinth target bootstraps the pose and the model refines it.

Concept

Each object carries its own stories. Approaching a plinth reveals a small anchored marker; looking at it for a moment opens a story card beside the object. From the card the visitor can expand an exploded view that lifts components off the object in place, or a provenance line that draws a path from the object outward through the places it has been. Content stays with the object as the visitor moves around the plinth. Nothing floats in the middle of the room.

Design

Passthrough changes the rules of legibility. Text that is readable on an opaque display becomes hard to read over a mid-grey plinth in gallery lighting, so we settled on a limited palette, large type and panels with a soft backing that adapts its opacity to the brightness of what is behind it. Panels anchor to the object but turn to face the visitor, and they keep a minimum distance from the object so the artefact is never covered. Hand tracking replaces controllers: a pinch to select, an open palm to dismiss. Multilingual text is held as structured content so layouts reflow for longer languages. We applied the principles in our spatial UX note and revised them where passthrough proved them wrong.

Photo: Portable Antiquities Scheme · CC BY 2.0

Technology

The prototype runs in Unity on Quest 3 through OpenXR and the Meta Quest SDK, which provide passthrough, hand tracking and spatial anchors. Objects are recognised by a computer vision step: image targets on the plinth give the initial pose, and a feature-matching pass against a photogrammetry capture of the object refines it. Exploded views are modelled in Blender from the same captures. Stories, translations and provenance entries live in Firebase and are edited through a plain web form we built as the curator tool. Audio is spatialised at the object's position.

Interaction

Gaze opens, hands act. Looking at an object's marker for about a second opens its card, which avoids accidental triggers while walking past. A pinch selects a story, a palm dismisses everything, and a slow pinch-drag on the exploded view rotates it. Audio follows the visitor's attention: moving to another object fades the previous narration. There is no menu system; the gallery is the menu.

Implementation

We built the studio test with a small set of replica objects on plinths under adjustable lights to simulate spot-lit and diffuse gallery conditions, and one object inside a glass case. The recognition pipeline is a Unity C# layer over the vision component, with a confidence gate that holds content until the pose is stable for several frames. The curator tool writes to Firebase; the headset caches a collection on start so the app works without Wi-Fi during a session. We also prototyped the loan-device flow: a kiosk mode that resets state, clears the language selection and returns to an idle screen when the headset is removed.

Photo: Kai Kowalewski · CC BY-SA 4.0

Result

The prototype demonstrated that an object-anchored guide in passthrough is workable with current hardware: recognition is reliable for matte, textured objects under stable light, stories stay attached to the object as a visitor moves, and a curator can change text and translations without touching the application. It also exposed the hard parts. Glass cases defeat the refinement step in most lighting; reflective and dark objects are recognised slowly or not at all; and text legibility in passthrough needs more contrast than desktop guidelines suggest. What remains open is a test in a real gallery with real visitors, and the operational side of a loan programme: hygiene, charging, fitting and staff time.

Lessons

Recognition robustness is a lighting problem before it is a model problem, and a plinth target as fallback is worth its visual cost. Glass changes everything and needs its own approach, probably a plinth-relative anchor with no refinement. Legibility rules from screens do not survive passthrough. Loan devices are a logistics project, and the software should assume a headset will be handed over mid-session. Finally, curator tooling is not an add-on: it decides whether a museum can keep the experience alive after the developers leave. If you are planning a mixed reality guide for a collection, talk to us about what to test first.