xr-simulator/docs/DESIGN.md

# OpenXR Simulator Runtime — Design

A Windows OpenXR **runtime** that pretends to be a headset. OpenXR applications
(especially the `gl3-vulkan` / "Get Beaned" engine) run unmodified and render
into runtime-owned swapchains; the runtime displays those swapchain images in a
desktop window built with GLFW + Dear ImGui (docking).

This document is the architecture of record for the scaffold in this repository.

---

## 1. What an OpenXR runtime is on Windows

The application links `openxr_loader.dll` (from OpenXR-SDK). The loader is not
the runtime; it locates one of the installed runtimes and forwards every `xr*`
call to it.

Resolution order for the active runtime:

1. `XR_RUNTIME_JSON` environment variable → path to a runtime manifest JSON.
2. Per-user registry `HKCU\SOFTWARE\Khronos\OpenXR\1\ActiveRuntime`.
3. Machine registry `HKLM\SOFTWARE\Khronos\OpenXR\1\ActiveRuntime`.

The manifest points at a DLL that exports `xrNegotiateLoaderRuntimeInterface`.
That function hands the loader an `XrNegotiateRuntimeRequest` containing a
`PFN_xrGetInstanceProcAddr`. The loader uses that GIPA to resolve every core and
extension entry point.

```json
{
  "file_format_version": "1.0.0",
  "runtime": {
    "library_path": "openxr_simulator.dll",
    "name": "OpenXR Simulator"
  }
}
```

The negotiation structs are in `third_party/openxr/openxr_loader_negotiation.h`
(vendored from OpenXR-SDK-Source `src/common/loader_interfaces.h`).

### Development shortcut

The `gl3-vulkan` engine already supports overriding the runtime without touching
the registry: `ApplicationContext::createOpenXrInstance(customOpenXrRuntimePath)`
sets `XR_RUNTIME_JSON` (`code/Engine/src/Engine/Core/ApplicationContext.cpp:904`).
So the whole simulator can be tested by launching the game with that env var
pointing at `openxr_simulator.json`. No system-wide install required.

---

## 2. How `gl3-vulkan` uses OpenXR (the contract we must satisfy)

The engine is the primary target. Key facts, with source references:

### 2.1 The runtime owns Vulkan

The engine does **not** create its own `VkInstance`/`VkDevice`. It asks the
runtime to:

| Call | Location |
| --- | --- |
| `xrCreateVulkanInstanceKHR` | `ApplicationContext.cpp:1442` |
| `xrGetVulkanGraphicsDevice2KHR` | `ApplicationContext.cpp:177` |
| `xrCreateVulkanDeviceKHR` | `ApplicationContext.cpp:478` |

Consequence: the runtime creates the instance and device, so it can create a
Win32 `VkSurfaceKHR`/present queue on the same objects and display the app's
images with **zero cross-device copies**. This is the single most important
design lever.

### 2.2 Graphics binding and swapchains

- Session is created with `XrGraphicsBindingVulkan2KHR` (app's instance, physical
  device, device, graphics queue family/index): `Headset.cpp:59`.
- One **multiview** swapchain, `arraySize = 2` (`Headset.cpp:657`). Two eye
  layers in one image, submitted as two projection views with
  `imageArrayIndex = 0/1` (`Headset.cpp:762`).
- Format negotiated via `xrEnumerateSwapchainFormats`; engine prefers
  `VK_FORMAT_R8G8B8A8_SRGB` (`Headset.cpp:23`), fallback `R8G8B8A8_UNORM` +
  `MUTABLE_FORMAT` (`Headset.cpp:24,667`).
- `recommendedSwapchainSampleCount`, `recommendedImageRectWidth/Height` come
  from our view configuration.
- Reference space is `XR_REFERENCE_SPACE_TYPE_STAGE` (`Headset.cpp:20`).

### 2.3 Frame loop

Per frame: `xrWaitFrame` → `xrBeginFrame` → `xrLocateViews` →
`xrAcquireSwapchainImage` → `xrWaitSwapchainImage` → [app renders] →
`xrReleaseSwapchainImage` → `xrEndFrame` with one
`XrCompositionLayerProjection` (2 views). See `Headset.cpp:1070-1161`.

### 2.4 Required extensions and runtime behaviour

- `XR_KHR_vulkan_enable2` is hard-required (`ApplicationContext.cpp:963,979`).
- Debug builds also require `XR_EXT_debug_utils` and call
  `xrCreateDebugUtilsMessengerEXT` (`ApplicationContext.cpp:965-967,1090`).
- `XR_ENVIRONMENT_BLEND_MODE_OPAQUE` must be advertised
  (`ApplicationContext.cpp:66,1194`).
- Session state events drive startup: the engine waits for
  `XR_SESSION_STATE_READY` to call `xrBeginSession`
  (`Engine.cpp:284`, `Headset.cpp:925`), and only renders in
  READY/SYNCHRONIZED/VISIBLE/FOCUSED (`Headset.cpp:990`). We must enqueue
  `XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED`.
- Input: the engine builds actions for ~16 interaction profiles and attaches
  them (`XrInputHandler.cpp`). We must implement the action API and at least one
  profile (`khr/simple_controller` and/or `oculus/touch_controller`).

### 2.5 Gotcha: Vulkan requirement version encoding

`getVulkanGraphicsRequirements2KHR` must report `minApiVersionSupported` /
`maxApiVersionSupported` using **`XR_MAKE_VERSION`** (major in bits 48-63), not
`VK_MAKE_VERSION`. The engine decodes with `XR_VERSION_MAJOR/MINOR`
(`ApplicationContext.cpp:1289-1297`) and rejects the runtime otherwise. Report
`XR_MAKE_VERSION(1,0,0)` .. `XR_MAKE_VERSION(1,4,0)`.

---

## 3. Architecture

```
 app process (Game.exe)
 ┌─────────────────────────────────────────────────────────────┐
 │  engine  ── xr* calls ──►  openxr_loader.dll                 │
 │                                   │                          │
 │                                   ▼                          │
 │                            openxr_simulator.dll   (this repo)│
 │  ┌───────────────────────────────────────────────────────┐   │
 │  │ xrNegotiateLoaderRuntimeInterface  (DLL export)       │   │
 │  │ xrGetInstanceProcAddr              (dispatch table)   │   │
 │  │                                                        │   │
 │  │ Instance ─ System ─ Session ─ Swapchain                │   │
 │  │      │                                │                │   │
 │  │      ▼                                ▼                │   │
 │  │ Vulkan backend (owns VkInstance/VkDevice)              │   │
 │  │      │                                                 │   │
 │  │      ▼                                                 │   │
 │  │ Compositor (GLFW window + ImGui docking)               │   │
 │  │  - eye textures drawn in dockable ImGui windows        │   │
 │  └───────────────────────────────────────────────────────┘   │
 └─────────────────────────────────────────────────────────────┘
```

### 3.1 Layers

| Layer | Responsibility |
| --- | --- |
| Negotiation | Export the interface, build the GIPA table. |
| Instance | State per `XrInstance`; registry of handles. |
| System / View config | Advertise HMD system, stereo view config, resolution, formats, blend modes. |
| Vulkan backend | Create/hold `VkInstance`/`VkDevice`, allocate swapchain images, own the present queue. |
| Session / Frame loop | Session state events, `xrWaitFrame` pacing, `xrEndFrame` submission. |
| Tracking | Head pose + controller poses (MVP: mouse/WASD/keyboard). |
| Actions | Action sets, bindings, `xrSyncActions`, state queries. |
| Compositor | GLFW+ImGui window; blit submitted eye layers into ImGui-visible textures. |

### 3.2 Object model and handles

OpenXR handles are opaque pointers (`XR_DEFINE_HANDLE`). We allocate small
structs and reinterpret-cast to the handle type, keeping them in a global
registry guarded by a mutex:

```cpp
struct RuntimeInstance; struct RuntimeSession; struct RuntimeSwapchain; ...
RuntimeInstance* GetInstance(XrInstance);
RuntimeSession*  GetSession(XrSession);
```

### 3.3 Vulkan ownership

Because the engine delegates instance/device creation to us:

- `xrCreateVulkanInstanceKHR`: call the app-supplied `pfnGetInstanceProcAddr` to
  get `vkCreateInstance`, create the instance (augmenting the create info with
  our own required extensions such as `VK_KHR_surface`,
  `VK_KHR_win32_surface`).
- `xrGetVulkanGraphicsDevice2KHR`: enumerate devices, pick one that supports
  graphics + present to our window surface, return it.
- `xrCreateVulkanDeviceKHR`: `vkCreateDevice` with the app's create info plus any
  queue families/extensions we need (`VK_KHR_swapchain`).
- Swapchain images are allocated on this device; the compositor blits them on
  the same device/queue. No interop needed.

### 3.4 Compositor (GLFW + ImGui docking)

Decision (MVP): the simulator window is a normal GLFW window rendered with the
Dear ImGui Vulkan backend, using ImGui docking so the user can freely arrange
windows. Rather than presenting to a raw swapchain, each submitted eye is shown
in a dockable `ImGui::Image` window.

**Thread affinity (important).** The `gl3-vulkan` engine calls `xrEndFrame` from
a **background thread** (`EngineKern::completeFrameAsync`) and `xrWaitFrame` from
a worker pool, but calls `xrBeginFrame` from its main thread. GLFW/ImGui are
thread-affine, so the compositor splits into two phases:

- `Present()` — called from `xrEndFrame` on any thread. Only *records* the
  resolved eye images (`ResolveSubImage`, using the swapchain's
  `lastReleasedIndex` because release already cleared the acquired index). No
  GLFW/ImGui/Vulkan work.
- `Pump()` — called from `xrBeginFrame` and `xrWaitFrame`; does the real work
  only when the caller is the thread that created the window (guarded by
  `windowThreadId_`), otherwise returns immediately.

Pump steps:

1. Wait for the app's rendering to finish on the shared queue (MVP:
   `vkQueueWaitIdle` on the app graphics queue; later: timeline/fence sync).
2. Copy the two array layers of the submitted XR swapchain image into two
   persistent runtime "display" images (`TRANSFER_DST | SAMPLED`, kept in
   `GENERAL` layout).
3. `ImGui::NewFrame()`; draw a menu + two image windows (`Eye L`, `Eye R`)
   sampling the display images, plus simulator controls (tracking sliders).
4. Render ImGui into the window swapchain and present.

Registering two persistent display images as ImGui textures decouples the ImGui
texture set from the XR swapchain image count and lets us scale each eye to fit
its dock window.

Window/queue notes:

- The simulator window surface is created on the runtime-owned `VkInstance`.
- Use the app's graphics queue (from `XrGraphicsBindingVulkan2KHR`) as the ImGui
  queue so ordering on the same queue provides implicit synchronization, and
  ensure that family supports present to our surface. If it does not, do a queue
  family ownership transfer (or fall back to a dedicated runtime present queue +
  explicit waits).
- `gl3-vulkan` also creates its own GLFW mirror window on the same instance.
  GLFW is statically linked into both; each copy keeps independent state, so
  both can coexist. If this proves fragile, the ImGui **Win32** backend is the
  fallback (no GLFW dependency).

### 3.5 Frame pacing and events

- `xrWaitFrame` throttles to a configurable virtual refresh rate (default 90 Hz)
  and returns `predictedDisplayTime` + `shouldRender`.
- Session state event sequence we enqueue:
  `IDLE → READY` at session creation, then after `xrBeginSession`
  `SYNCHRONIZED → VISIBLE → FOCUSED`. This matches what the engine's event
  handler expects.

### 3.6 Tracking and input

- Head: mouse look (yaw/pitch) + WASD translation, integrated into an
  `XrPosef` returned from `xrLocateViews`. STAGE space; y- is up.
- Controllers: the runtime advertises an emulated `oculus/touch_controller`
  profile from `xrGetCurrentInteractionProfile` for both hands, so the app
  activates that profile and wires up its aim/grip pose spaces.
- Action state (`xrGetActionState*`) is classified from the action name suffix
  (`trigger_value`, `grip_value`, `thumbstick`, `a_button`, …) and the
  subaction path (hand), then read from a global `InputState` updated by the
  compositor. `xrLocateSpace` on an action space returns the emulated hand pose.

Keyboard mapping (shown in the ImGui `Controls` window):

| Control | Left hand | Right hand |
| --- | --- | --- |
| Head look / move | right-drag, WASD, Q/E | |
| Trigger | `F` | `H` |
| Grip | `G` | `U` |
| Thumbstick | `I`/`K`/`J`/`L` | arrow keys |
| Stick click | `R` | `O` |
| Face buttons | `C` (X), `V` (Y) | `N` (A), `M` (B) |
| Menu | `Tab` | |

The hand poses are placed relative to the head, so they follow look direction.
A `TrackingProvider` interface is planned so a real HMD / OpenTrack / gamepad
bridge can be plugged in later without touching the session code.

### 3.7 Configuration

The runtime reads a JSON config (`src/Config.{h,cpp}`) from `XRSIM_CONFIG`, or
else `openxr_simulator_config.json` next to the DLL. It drives the emulated
headset: system name/vendor, per-eye resolution, refresh rate, FOV, IPD,
swapchain image count, and the compositor window (title/size/present mode/display
mode). The `configurator` tool (GLFW + ImGui, `tools/configurator/`) edits this
file and ships presets for common headsets; it does not render — it only writes
the config the DLL reads.

Presentation mode matters for throughput: `fifo` presents with vsync and, because
presentation runs on the app's thread inside `xrBeginFrame`, caps the app to the
monitor refresh. The default `mailbox` (or `immediate`) is uncapped. `refreshRateHz`
sets the simulated strobe used by `xrWaitFrame`, independent of presentation.

---

## 4. Phasing

| Phase | Deliverable | Exit criterion |
| --- | --- | --- |
| 0 | Negotiation + GIPA + metadata functions (this scaffold) | Loader loads the DLL; `xrCreateInstance`/`xrGetSystem` succeed. |
| 1 | Vulkan passthrough | Engine creates its `VkInstance`/`VkDevice` through us. |
| 2 | Session, swapchain, frame loop, GLFW+ImGui compositor | Eye images (test pattern, then real) appear in the ImGui window. |
| 3 | Tracking | Mouse/WASD head pose drives `xrLocateViews`. |
| 4 | Actions/controllers | Game hands/inputs work; a profile is bound. |
| 5 | Polish | Session state edges, visibility/focus, haptics no-op, settings UI. |

Phases 0–4 are implemented. Remaining: phase 5 polish.

## 5. Risks / open questions

- **Loader strictness**: GIPA must return non-null for every function the app
  loads. Extension functions the engine loads (`xrGetVulkanGraphicsDevice2KHR`,
  `xrGetVulkanGraphicsRequirements2KHR`, `xrCreateVulkanInstanceKHR`,
  `xrCreateVulkanDeviceKHR`) must be real, or `OpenXrHelper::LoadXrFunction`
  throws.
- **GPU sync without app-provided semaphores**: core OpenXR gives no sync
  primitive for swapchain images; MVP uses `vkQueueWaitIdle`.
- **Present family mismatch** between the app's graphics family and our window
  surface.
- **Duplicate GLFW instances** in one process (engine + runtime); fallback to
  ImGui Win32 backend.
- **Format negotiation**: only advertise formats we can actually create and
  sample.
- **Version encoding** for Vulkan requirements (§2.5).

## 6. References

- OpenXR-SDK `hello_xr` — what a runtime must answer.
- Monado — reference open-source runtime (negotiation + compositor).
- `openxr_loader_negotiation.h` — exact negotiation structs.
- Engine source: `code/Engine/src/Engine/Core/ApplicationContext.cpp`,
  `code/Engine/src/Engine/Renderer/Headset.cpp`,
  `code/Engine/src/Engine/Input/XrInputHandler.cpp`.