xr-simulator/docs/DESIGN.md

OpenXR Simulator Runtime — Design

A Windows OpenXR runtime that pretends to be a headset. OpenXR applications (especially the gl3-vulkan / "Get Beaned" engine) run unmodified and render into runtime-owned swapchains; the runtime displays those swapchain images in a desktop window built with GLFW + Dear ImGui (docking).

This document is the architecture of record for the scaffold in this repository.

---

1. What an OpenXR runtime is on Windows

The application links openxr_loader.dll (from OpenXR-SDK). The loader is not the runtime; it locates one of the installed runtimes and forwards every xr* call to it.

Resolution order for the active runtime:

1. XR_RUNTIME_JSON environment variable → path to a runtime manifest JSON.
2. Per-user registry HKCU\SOFTWARE\Khronos\OpenXR\1\ActiveRuntime.
3. Machine registry HKLM\SOFTWARE\Khronos\OpenXR\1\ActiveRuntime.

The manifest points at a DLL that exports xrNegotiateLoaderRuntimeInterface. That function hands the loader an XrNegotiateRuntimeRequest containing a PFN_xrGetInstanceProcAddr. The loader uses that GIPA to resolve every core and extension entry point.

{
  "file_format_version": "1.0.0",
  "runtime": {
    "library_path": "openxr_simulator.dll",
    "name": "OpenXR Simulator"
  }
}

The negotiation structs are in third_party/openxr/openxr_loader_negotiation.h (vendored from OpenXR-SDK-Source src/common/loader_interfaces.h).

Development shortcut

The gl3-vulkan engine already supports overriding the runtime without touching the registry: ApplicationContext::createOpenXrInstance(customOpenXrRuntimePath) sets XR_RUNTIME_JSON (code/Engine/src/Engine/Core/ApplicationContext.cpp:904). So the whole simulator can be tested by launching the game with that env var pointing at openxr_simulator.json. No system-wide install required.

---

2. How gl3-vulkan uses OpenXR (the contract we must satisfy)

The engine is the primary target. Key facts, with source references:

2.1 The runtime owns Vulkan

The engine does not create its own VkInstance/VkDevice. It asks the runtime to:

Call	Location
xrCreateVulkanInstanceKHR	ApplicationContext.cpp:1442
xrGetVulkanGraphicsDevice2KHR	ApplicationContext.cpp:177
xrCreateVulkanDeviceKHR	ApplicationContext.cpp:478

Consequence: the runtime creates the instance and device, so it can create a Win32 VkSurfaceKHR/present queue on the same objects and display the app's images with zero cross-device copies. This is the single most important design lever.

2.2 Graphics binding and swapchains

• Session is created with XrGraphicsBindingVulkan2KHR (app's instance, physical device, device, graphics queue family/index): Headset.cpp:59.
• One multiview swapchain, arraySize = 2 (Headset.cpp:657). Two eye layers in one image, submitted as two projection views with imageArrayIndex = 0/1 (Headset.cpp:762).
• Format negotiated via xrEnumerateSwapchainFormats; engine prefers VK_FORMAT_R8G8B8A8_SRGB (Headset.cpp:23), fallback R8G8B8A8_UNORM + MUTABLE_FORMAT (Headset.cpp:24,667).
• recommendedSwapchainSampleCount, recommendedImageRectWidth/Height come from our view configuration.
• Reference space is XR_REFERENCE_SPACE_TYPE_STAGE (Headset.cpp:20).

2.3 Frame loop

Per frame: xrWaitFrame → xrBeginFrame → xrLocateViews → xrAcquireSwapchainImage → xrWaitSwapchainImage → [app renders] → xrReleaseSwapchainImage → xrEndFrame with one XrCompositionLayerProjection (2 views). See Headset.cpp:1070-1161.

2.4 Required extensions and runtime behaviour

• XR_KHR_vulkan_enable2 is hard-required (ApplicationContext.cpp:963,979).
• Debug builds also require XR_EXT_debug_utils and call xrCreateDebugUtilsMessengerEXT (ApplicationContext.cpp:965-967,1090).
• XR_ENVIRONMENT_BLEND_MODE_OPAQUE must be advertised (ApplicationContext.cpp:66,1194).
• Session state events drive startup: the engine waits for XR_SESSION_STATE_READY to call xrBeginSession (Engine.cpp:284, Headset.cpp:925), and only renders in READY/SYNCHRONIZED/VISIBLE/FOCUSED (Headset.cpp:990). We must enqueue XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED.
• Input: the engine builds actions for ~16 interaction profiles and attaches them (XrInputHandler.cpp). We must implement the action API and at least one profile (khr/simple_controller and/or oculus/touch_controller).

2.5 Gotcha: Vulkan requirement version encoding

getVulkanGraphicsRequirements2KHR must report minApiVersionSupported / maxApiVersionSupported using XR_MAKE_VERSION (major in bits 48-63), not VK_MAKE_VERSION. The engine decodes with XR_VERSION_MAJOR/MINOR (ApplicationContext.cpp:1289-1297) and rejects the runtime otherwise. Report XR_MAKE_VERSION(1,0,0) .. XR_MAKE_VERSION(1,4,0).

---

3. Architecture

 app process (Game.exe)
 ┌─────────────────────────────────────────────────────────────┐
 │  engine  ── xr* calls ──►  openxr_loader.dll                 │
 │                                   │                          │
 │                                   ▼                          │
 │                            openxr_simulator.dll   (this repo)│
 │  ┌───────────────────────────────────────────────────────┐   │
 │  │ xrNegotiateLoaderRuntimeInterface  (DLL export)       │   │
 │  │ xrGetInstanceProcAddr              (dispatch table)   │   │
 │  │                                                        │   │
 │  │ Instance ─ System ─ Session ─ Swapchain                │   │
 │  │      │                                │                │   │
 │  │      ▼                                ▼                │   │
 │  │ Vulkan backend (owns VkInstance/VkDevice)              │   │
 │  │      │                                                 │   │
 │  │      ▼                                                 │   │
 │  │ Compositor (GLFW window + ImGui docking)               │   │
 │  │  - eye textures drawn in dockable ImGui windows        │   │
 │  └───────────────────────────────────────────────────────┘   │
 └─────────────────────────────────────────────────────────────┘

3.1 Layers

Layer	Responsibility
Negotiation	Export the interface, build the GIPA table.
Instance	State per XrInstance; registry of handles.
System / View config	Advertise HMD system, stereo view config, resolution, formats, blend modes.
Vulkan backend	Create/hold VkInstance/VkDevice, allocate swapchain images, own the present queue.
Session / Frame loop	Session state events, xrWaitFrame pacing, xrEndFrame submission.
Tracking	Head pose + controller poses (MVP: mouse/WASD/keyboard).
Actions	Action sets, bindings, xrSyncActions, state queries.
Compositor	GLFW+ImGui window; blit submitted eye layers into ImGui-visible textures.

3.2 Object model and handles

OpenXR handles are opaque pointers (XR_DEFINE_HANDLE). We allocate small structs and reinterpret-cast to the handle type, keeping them in a global registry guarded by a mutex:

struct RuntimeInstance; struct RuntimeSession; struct RuntimeSwapchain; ...
RuntimeInstance* GetInstance(XrInstance);
RuntimeSession*  GetSession(XrSession);

3.3 Vulkan ownership

Because the engine delegates instance/device creation to us:

• xrCreateVulkanInstanceKHR: call the app-supplied pfnGetInstanceProcAddr to get vkCreateInstance, create the instance (augmenting the create info with our own required extensions such as VK_KHR_surface, VK_KHR_win32_surface).
• xrGetVulkanGraphicsDevice2KHR: enumerate devices, pick one that supports graphics + present to our window surface, return it.
• xrCreateVulkanDeviceKHR: vkCreateDevice with the app's create info plus any queue families/extensions we need (VK_KHR_swapchain).
• Swapchain images are allocated on this device; the compositor blits them on the same device/queue. No interop needed.

3.4 Compositor (GLFW + ImGui docking)

Decision (MVP): the simulator window is a normal GLFW window rendered with the Dear ImGui Vulkan backend, using ImGui docking so the user can freely arrange windows. Rather than presenting to a raw swapchain, each submitted eye is shown in a dockable ImGui::Image window.

Thread affinity (important). The gl3-vulkan engine calls xrEndFrame from a background thread (EngineKern::completeFrameAsync) and xrWaitFrame from a worker pool, but calls xrBeginFrame from its main thread. GLFW/ImGui are thread-affine, so the compositor splits into two phases:

• Present() — called from xrEndFrame on any thread. Only records the resolved eye images (ResolveSubImage, using the swapchain's lastReleasedIndex because release already cleared the acquired index). No GLFW/ImGui/Vulkan work.
• Pump() — called from xrBeginFrame and xrWaitFrame; does the real work only when the caller is the thread that created the window (guarded by windowThreadId_), otherwise returns immediately.

Pump steps:

1. Wait for the app's rendering to finish on the shared queue (MVP: vkQueueWaitIdle on the app graphics queue; later: timeline/fence sync).
2. Copy the two array layers of the submitted XR swapchain image into two persistent runtime "display" images (TRANSFER_DST | SAMPLED, kept in GENERAL layout).
3. ImGui::NewFrame(); draw a menu + two image windows (Eye L, Eye R) sampling the display images, plus simulator controls (tracking sliders).
4. Render ImGui into the window swapchain and present.

Registering two persistent display images as ImGui textures decouples the ImGui texture set from the XR swapchain image count and lets us scale each eye to fit its dock window.

Window/queue notes:

• The simulator window surface is created on the runtime-owned VkInstance.
• Use the app's graphics queue (from XrGraphicsBindingVulkan2KHR) as the ImGui queue so ordering on the same queue provides implicit synchronization, and ensure that family supports present to our surface. If it does not, do a queue family ownership transfer (or fall back to a dedicated runtime present queue + explicit waits).
• gl3-vulkan also creates its own GLFW mirror window on the same instance. GLFW is statically linked into both; each copy keeps independent state, so both can coexist. If this proves fragile, the ImGui Win32 backend is the fallback (no GLFW dependency).

3.5 Frame pacing and events

• xrWaitFrame throttles to a configurable virtual refresh rate (default 90 Hz) and returns predictedDisplayTime + shouldRender.
• Session state event sequence we enqueue: IDLE → READY at session creation, then after xrBeginSession SYNCHRONIZED → VISIBLE → FOCUSED. This matches what the engine's event handler expects.

3.6 Tracking and input

• Head: mouse look (yaw/pitch) + WASD translation, integrated into an XrPosef returned from xrLocateViews. STAGE space; y- is up.
• Controllers: the runtime advertises an emulated oculus/touch_controller profile from xrGetCurrentInteractionProfile for both hands, so the app activates that profile and wires up its aim/grip pose spaces.
• Action state (xrGetActionState*) is classified from the action name suffix (trigger_value, grip_value, thumbstick, a_button, …) and the subaction path (hand), then read from a global InputState updated by the compositor. xrLocateSpace on an action space returns the emulated hand pose.

Keyboard mapping (shown in the ImGui Controls window):

Control	Left hand	Right hand
Head look / move	right-drag, WASD, Q/E
Trigger	F	H
Grip	G	U
Thumbstick	I/K/J/L	arrow keys
Stick click	R	O
Face buttons	C (X), V (Y)	N (A), M (B)
Menu	Tab

The hand poses are placed relative to the head, so they follow look direction. A TrackingProvider interface is planned so a real HMD / OpenTrack / gamepad bridge can be plugged in later without touching the session code.

3.7 Configuration

The runtime reads a JSON config (src/Config.{h,cpp}) from XRSIM_CONFIG, or else openxr_simulator_config.json next to the DLL. It drives the emulated headset: system name/vendor, per-eye resolution, refresh rate, FOV, IPD, swapchain image count, and the compositor window (title/size/present mode/display mode). The configurator tool (GLFW + ImGui, tools/configurator/) edits this file and ships presets for common headsets; it does not render — it only writes the config the DLL reads.

Presentation mode matters for throughput: fifo presents with vsync and, because presentation runs on the app's thread inside xrBeginFrame, caps the app to the monitor refresh. The default mailbox (or immediate) is uncapped. refreshRateHz sets the simulated strobe used by xrWaitFrame, independent of presentation.

---

4. Phasing

Phase	Deliverable	Exit criterion
0	Negotiation + GIPA + metadata functions (this scaffold)	Loader loads the DLL; xrCreateInstance/xrGetSystem succeed.
1	Vulkan passthrough	Engine creates its VkInstance/VkDevice through us.
2	Session, swapchain, frame loop, GLFW+ImGui compositor	Eye images (test pattern, then real) appear in the ImGui window.
3	Tracking	Mouse/WASD head pose drives xrLocateViews.
4	Actions/controllers	Game hands/inputs work; a profile is bound.
5	Polish	Session state edges, visibility/focus, haptics no-op, settings UI.

Phases 0–4 are implemented. Remaining: phase 5 polish.

5. Risks / open questions

• Loader strictness: GIPA must return non-null for every function the app loads. Extension functions the engine loads (xrGetVulkanGraphicsDevice2KHR, xrGetVulkanGraphicsRequirements2KHR, xrCreateVulkanInstanceKHR, xrCreateVulkanDeviceKHR) must be real, or OpenXrHelper::LoadXrFunction throws.
• GPU sync without app-provided semaphores: core OpenXR gives no sync primitive for swapchain images; MVP uses vkQueueWaitIdle.
• Present family mismatch between the app's graphics family and our window surface.
• Duplicate GLFW instances in one process (engine + runtime); fallback to ImGui Win32 backend.
• Format negotiation: only advertise formats we can actually create and sample.
• Version encoding for Vulkan requirements (§2.5).

6. References

• OpenXR-SDK hello_xr — what a runtime must answer.
• Monado — reference open-source runtime (negotiation + compositor).
• openxr_loader_negotiation.h — exact negotiation structs.
• Engine source: code/Engine/src/Engine/Core/ApplicationContext.cpp, code/Engine/src/Engine/Renderer/Headset.cpp, code/Engine/src/Engine/Input/XrInputHandler.cpp.