Initial commit: OpenXR simulator runtime, compositor, configurator, docs
This commit is contained in:
Konstantin Uni
+323
@@ -0,0 +1,323 @@
|
||||
# OpenXR Simulator Runtime — Design
|
||||
|
||||
A Windows OpenXR **runtime** that pretends to be a headset. OpenXR applications
|
||||
(especially the `gl3-vulkan` / "Get Beaned" engine) run unmodified and render
|
||||
into runtime-owned swapchains; the runtime displays those swapchain images in a
|
||||
desktop window built with GLFW + Dear ImGui (docking).
|
||||
|
||||
This document is the architecture of record for the scaffold in this repository.
|
||||
|
||||
---
|
||||
|
||||
## 1. What an OpenXR runtime is on Windows
|
||||
|
||||
The application links `openxr_loader.dll` (from OpenXR-SDK). The loader is not
|
||||
the runtime; it locates one of the installed runtimes and forwards every `xr*`
|
||||
call to it.
|
||||
|
||||
Resolution order for the active runtime:
|
||||
|
||||
1. `XR_RUNTIME_JSON` environment variable → path to a runtime manifest JSON.
|
||||
2. Per-user registry `HKCU\SOFTWARE\Khronos\OpenXR\1\ActiveRuntime`.
|
||||
3. Machine registry `HKLM\SOFTWARE\Khronos\OpenXR\1\ActiveRuntime`.
|
||||
|
||||
The manifest points at a DLL that exports `xrNegotiateLoaderRuntimeInterface`.
|
||||
That function hands the loader an `XrNegotiateRuntimeRequest` containing a
|
||||
`PFN_xrGetInstanceProcAddr`. The loader uses that GIPA to resolve every core and
|
||||
extension entry point.
|
||||
|
||||
```json
|
||||
{
|
||||
"file_format_version": "1.0.0",
|
||||
"runtime": {
|
||||
"library_path": "openxr_simulator.dll",
|
||||
"name": "OpenXR Simulator"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The negotiation structs are in `third_party/openxr/openxr_loader_negotiation.h`
|
||||
(vendored from OpenXR-SDK-Source `src/common/loader_interfaces.h`).
|
||||
|
||||
### Development shortcut
|
||||
|
||||
The `gl3-vulkan` engine already supports overriding the runtime without touching
|
||||
the registry: `ApplicationContext::createOpenXrInstance(customOpenXrRuntimePath)`
|
||||
sets `XR_RUNTIME_JSON` (`code/Engine/src/Engine/Core/ApplicationContext.cpp:904`).
|
||||
So the whole simulator can be tested by launching the game with that env var
|
||||
pointing at `openxr_simulator.json`. No system-wide install required.
|
||||
|
||||
---
|
||||
|
||||
## 2. How `gl3-vulkan` uses OpenXR (the contract we must satisfy)
|
||||
|
||||
The engine is the primary target. Key facts, with source references:
|
||||
|
||||
### 2.1 The runtime owns Vulkan
|
||||
|
||||
The engine does **not** create its own `VkInstance`/`VkDevice`. It asks the
|
||||
runtime to:
|
||||
|
||||
| Call | Location |
|
||||
| --- | --- |
|
||||
| `xrCreateVulkanInstanceKHR` | `ApplicationContext.cpp:1442` |
|
||||
| `xrGetVulkanGraphicsDevice2KHR` | `ApplicationContext.cpp:177` |
|
||||
| `xrCreateVulkanDeviceKHR` | `ApplicationContext.cpp:478` |
|
||||
|
||||
Consequence: the runtime creates the instance and device, so it can create a
|
||||
Win32 `VkSurfaceKHR`/present queue on the same objects and display the app's
|
||||
images with **zero cross-device copies**. This is the single most important
|
||||
design lever.
|
||||
|
||||
### 2.2 Graphics binding and swapchains
|
||||
|
||||
- Session is created with `XrGraphicsBindingVulkan2KHR` (app's instance, physical
|
||||
device, device, graphics queue family/index): `Headset.cpp:59`.
|
||||
- One **multiview** swapchain, `arraySize = 2` (`Headset.cpp:657`). Two eye
|
||||
layers in one image, submitted as two projection views with
|
||||
`imageArrayIndex = 0/1` (`Headset.cpp:762`).
|
||||
- Format negotiated via `xrEnumerateSwapchainFormats`; engine prefers
|
||||
`VK_FORMAT_R8G8B8A8_SRGB` (`Headset.cpp:23`), fallback `R8G8B8A8_UNORM` +
|
||||
`MUTABLE_FORMAT` (`Headset.cpp:24,667`).
|
||||
- `recommendedSwapchainSampleCount`, `recommendedImageRectWidth/Height` come
|
||||
from our view configuration.
|
||||
- Reference space is `XR_REFERENCE_SPACE_TYPE_STAGE` (`Headset.cpp:20`).
|
||||
|
||||
### 2.3 Frame loop
|
||||
|
||||
Per frame: `xrWaitFrame` → `xrBeginFrame` → `xrLocateViews` →
|
||||
`xrAcquireSwapchainImage` → `xrWaitSwapchainImage` → [app renders] →
|
||||
`xrReleaseSwapchainImage` → `xrEndFrame` with one
|
||||
`XrCompositionLayerProjection` (2 views). See `Headset.cpp:1070-1161`.
|
||||
|
||||
### 2.4 Required extensions and runtime behaviour
|
||||
|
||||
- `XR_KHR_vulkan_enable2` is hard-required (`ApplicationContext.cpp:963,979`).
|
||||
- Debug builds also require `XR_EXT_debug_utils` and call
|
||||
`xrCreateDebugUtilsMessengerEXT` (`ApplicationContext.cpp:965-967,1090`).
|
||||
- `XR_ENVIRONMENT_BLEND_MODE_OPAQUE` must be advertised
|
||||
(`ApplicationContext.cpp:66,1194`).
|
||||
- Session state events drive startup: the engine waits for
|
||||
`XR_SESSION_STATE_READY` to call `xrBeginSession`
|
||||
(`Engine.cpp:284`, `Headset.cpp:925`), and only renders in
|
||||
READY/SYNCHRONIZED/VISIBLE/FOCUSED (`Headset.cpp:990`). We must enqueue
|
||||
`XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED`.
|
||||
- Input: the engine builds actions for ~16 interaction profiles and attaches
|
||||
them (`XrInputHandler.cpp`). We must implement the action API and at least one
|
||||
profile (`khr/simple_controller` and/or `oculus/touch_controller`).
|
||||
|
||||
### 2.5 Gotcha: Vulkan requirement version encoding
|
||||
|
||||
`getVulkanGraphicsRequirements2KHR` must report `minApiVersionSupported` /
|
||||
`maxApiVersionSupported` using **`XR_MAKE_VERSION`** (major in bits 48-63), not
|
||||
`VK_MAKE_VERSION`. The engine decodes with `XR_VERSION_MAJOR/MINOR`
|
||||
(`ApplicationContext.cpp:1289-1297`) and rejects the runtime otherwise. Report
|
||||
`XR_MAKE_VERSION(1,0,0)` .. `XR_MAKE_VERSION(1,4,0)`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Architecture
|
||||
|
||||
```
|
||||
app process (Game.exe)
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ engine ── xr* calls ──► openxr_loader.dll │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ openxr_simulator.dll (this repo)│
|
||||
│ ┌───────────────────────────────────────────────────────┐ │
|
||||
│ │ xrNegotiateLoaderRuntimeInterface (DLL export) │ │
|
||||
│ │ xrGetInstanceProcAddr (dispatch table) │ │
|
||||
│ │ │ │
|
||||
│ │ Instance ─ System ─ Session ─ Swapchain │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ ▼ ▼ │ │
|
||||
│ │ Vulkan backend (owns VkInstance/VkDevice) │ │
|
||||
│ │ │ │ │
|
||||
│ │ ▼ │ │
|
||||
│ │ Compositor (GLFW window + ImGui docking) │ │
|
||||
│ │ - eye textures drawn in dockable ImGui windows │ │
|
||||
│ └───────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 3.1 Layers
|
||||
|
||||
| Layer | Responsibility |
|
||||
| --- | --- |
|
||||
| Negotiation | Export the interface, build the GIPA table. |
|
||||
| Instance | State per `XrInstance`; registry of handles. |
|
||||
| System / View config | Advertise HMD system, stereo view config, resolution, formats, blend modes. |
|
||||
| Vulkan backend | Create/hold `VkInstance`/`VkDevice`, allocate swapchain images, own the present queue. |
|
||||
| Session / Frame loop | Session state events, `xrWaitFrame` pacing, `xrEndFrame` submission. |
|
||||
| Tracking | Head pose + controller poses (MVP: mouse/WASD/keyboard). |
|
||||
| Actions | Action sets, bindings, `xrSyncActions`, state queries. |
|
||||
| Compositor | GLFW+ImGui window; blit submitted eye layers into ImGui-visible textures. |
|
||||
|
||||
### 3.2 Object model and handles
|
||||
|
||||
OpenXR handles are opaque pointers (`XR_DEFINE_HANDLE`). We allocate small
|
||||
structs and reinterpret-cast to the handle type, keeping them in a global
|
||||
registry guarded by a mutex:
|
||||
|
||||
```cpp
|
||||
struct RuntimeInstance; struct RuntimeSession; struct RuntimeSwapchain; ...
|
||||
RuntimeInstance* GetInstance(XrInstance);
|
||||
RuntimeSession* GetSession(XrSession);
|
||||
```
|
||||
|
||||
### 3.3 Vulkan ownership
|
||||
|
||||
Because the engine delegates instance/device creation to us:
|
||||
|
||||
- `xrCreateVulkanInstanceKHR`: call the app-supplied `pfnGetInstanceProcAddr` to
|
||||
get `vkCreateInstance`, create the instance (augmenting the create info with
|
||||
our own required extensions such as `VK_KHR_surface`,
|
||||
`VK_KHR_win32_surface`).
|
||||
- `xrGetVulkanGraphicsDevice2KHR`: enumerate devices, pick one that supports
|
||||
graphics + present to our window surface, return it.
|
||||
- `xrCreateVulkanDeviceKHR`: `vkCreateDevice` with the app's create info plus any
|
||||
queue families/extensions we need (`VK_KHR_swapchain`).
|
||||
- Swapchain images are allocated on this device; the compositor blits them on
|
||||
the same device/queue. No interop needed.
|
||||
|
||||
### 3.4 Compositor (GLFW + ImGui docking)
|
||||
|
||||
Decision (MVP): the simulator window is a normal GLFW window rendered with the
|
||||
Dear ImGui Vulkan backend, using ImGui docking so the user can freely arrange
|
||||
windows. Rather than presenting to a raw swapchain, each submitted eye is shown
|
||||
in a dockable `ImGui::Image` window.
|
||||
|
||||
**Thread affinity (important).** The `gl3-vulkan` engine calls `xrEndFrame` from
|
||||
a **background thread** (`EngineKern::completeFrameAsync`) and `xrWaitFrame` from
|
||||
a worker pool, but calls `xrBeginFrame` from its main thread. GLFW/ImGui are
|
||||
thread-affine, so the compositor splits into two phases:
|
||||
|
||||
- `Present()` — called from `xrEndFrame` on any thread. Only *records* the
|
||||
resolved eye images (`ResolveSubImage`, using the swapchain's
|
||||
`lastReleasedIndex` because release already cleared the acquired index). No
|
||||
GLFW/ImGui/Vulkan work.
|
||||
- `Pump()` — called from `xrBeginFrame` and `xrWaitFrame`; does the real work
|
||||
only when the caller is the thread that created the window (guarded by
|
||||
`windowThreadId_`), otherwise returns immediately.
|
||||
|
||||
Pump steps:
|
||||
|
||||
1. Wait for the app's rendering to finish on the shared queue (MVP:
|
||||
`vkQueueWaitIdle` on the app graphics queue; later: timeline/fence sync).
|
||||
2. Copy the two array layers of the submitted XR swapchain image into two
|
||||
persistent runtime "display" images (`TRANSFER_DST | SAMPLED`, kept in
|
||||
`GENERAL` layout).
|
||||
3. `ImGui::NewFrame()`; draw a menu + two image windows (`Eye L`, `Eye R`)
|
||||
sampling the display images, plus simulator controls (tracking sliders).
|
||||
4. Render ImGui into the window swapchain and present.
|
||||
|
||||
Registering two persistent display images as ImGui textures decouples the ImGui
|
||||
texture set from the XR swapchain image count and lets us scale each eye to fit
|
||||
its dock window.
|
||||
|
||||
Window/queue notes:
|
||||
|
||||
- The simulator window surface is created on the runtime-owned `VkInstance`.
|
||||
- Use the app's graphics queue (from `XrGraphicsBindingVulkan2KHR`) as the ImGui
|
||||
queue so ordering on the same queue provides implicit synchronization, and
|
||||
ensure that family supports present to our surface. If it does not, do a queue
|
||||
family ownership transfer (or fall back to a dedicated runtime present queue +
|
||||
explicit waits).
|
||||
- `gl3-vulkan` also creates its own GLFW mirror window on the same instance.
|
||||
GLFW is statically linked into both; each copy keeps independent state, so
|
||||
both can coexist. If this proves fragile, the ImGui **Win32** backend is the
|
||||
fallback (no GLFW dependency).
|
||||
|
||||
### 3.5 Frame pacing and events
|
||||
|
||||
- `xrWaitFrame` throttles to a configurable virtual refresh rate (default 90 Hz)
|
||||
and returns `predictedDisplayTime` + `shouldRender`.
|
||||
- Session state event sequence we enqueue:
|
||||
`IDLE → READY` at session creation, then after `xrBeginSession`
|
||||
`SYNCHRONIZED → VISIBLE → FOCUSED`. This matches what the engine's event
|
||||
handler expects.
|
||||
|
||||
### 3.6 Tracking and input
|
||||
|
||||
- Head: mouse look (yaw/pitch) + WASD translation, integrated into an
|
||||
`XrPosef` returned from `xrLocateViews`. STAGE space; y- is up.
|
||||
- Controllers: the runtime advertises an emulated `oculus/touch_controller`
|
||||
profile from `xrGetCurrentInteractionProfile` for both hands, so the app
|
||||
activates that profile and wires up its aim/grip pose spaces.
|
||||
- Action state (`xrGetActionState*`) is classified from the action name suffix
|
||||
(`trigger_value`, `grip_value`, `thumbstick`, `a_button`, …) and the
|
||||
subaction path (hand), then read from a global `InputState` updated by the
|
||||
compositor. `xrLocateSpace` on an action space returns the emulated hand pose.
|
||||
|
||||
Keyboard mapping (shown in the ImGui `Controls` window):
|
||||
|
||||
| Control | Left hand | Right hand |
|
||||
| --- | --- | --- |
|
||||
| Head look / move | right-drag, WASD, Q/E | |
|
||||
| Trigger | `F` | `H` |
|
||||
| Grip | `G` | `U` |
|
||||
| Thumbstick | `I`/`K`/`J`/`L` | arrow keys |
|
||||
| Stick click | `R` | `O` |
|
||||
| Face buttons | `C` (X), `V` (Y) | `N` (A), `M` (B) |
|
||||
| Menu | `Tab` | |
|
||||
|
||||
The hand poses are placed relative to the head, so they follow look direction.
|
||||
A `TrackingProvider` interface is planned so a real HMD / OpenTrack / gamepad
|
||||
bridge can be plugged in later without touching the session code.
|
||||
|
||||
### 3.7 Configuration
|
||||
|
||||
The runtime reads a JSON config (`src/Config.{h,cpp}`) from `XRSIM_CONFIG`, or
|
||||
else `openxr_simulator_config.json` next to the DLL. It drives the emulated
|
||||
headset: system name/vendor, per-eye resolution, refresh rate, FOV, IPD,
|
||||
swapchain image count, and the compositor window (title/size/present mode/display
|
||||
mode). The `configurator` tool (GLFW + ImGui, `tools/configurator/`) edits this
|
||||
file and ships presets for common headsets; it does not render — it only writes
|
||||
the config the DLL reads.
|
||||
|
||||
Presentation mode matters for throughput: `fifo` presents with vsync and, because
|
||||
presentation runs on the app's thread inside `xrBeginFrame`, caps the app to the
|
||||
monitor refresh. The default `mailbox` (or `immediate`) is uncapped. `refreshRateHz`
|
||||
sets the simulated strobe used by `xrWaitFrame`, independent of presentation.
|
||||
|
||||
---
|
||||
|
||||
## 4. Phasing
|
||||
|
||||
| Phase | Deliverable | Exit criterion |
|
||||
| --- | --- | --- |
|
||||
| 0 | Negotiation + GIPA + metadata functions (this scaffold) | Loader loads the DLL; `xrCreateInstance`/`xrGetSystem` succeed. |
|
||||
| 1 | Vulkan passthrough | Engine creates its `VkInstance`/`VkDevice` through us. |
|
||||
| 2 | Session, swapchain, frame loop, GLFW+ImGui compositor | Eye images (test pattern, then real) appear in the ImGui window. |
|
||||
| 3 | Tracking | Mouse/WASD head pose drives `xrLocateViews`. |
|
||||
| 4 | Actions/controllers | Game hands/inputs work; a profile is bound. |
|
||||
| 5 | Polish | Session state edges, visibility/focus, haptics no-op, settings UI. |
|
||||
|
||||
Phases 0–4 are implemented. Remaining: phase 5 polish.
|
||||
|
||||
## 5. Risks / open questions
|
||||
|
||||
- **Loader strictness**: GIPA must return non-null for every function the app
|
||||
loads. Extension functions the engine loads (`xrGetVulkanGraphicsDevice2KHR`,
|
||||
`xrGetVulkanGraphicsRequirements2KHR`, `xrCreateVulkanInstanceKHR`,
|
||||
`xrCreateVulkanDeviceKHR`) must be real, or `OpenXrHelper::LoadXrFunction`
|
||||
throws.
|
||||
- **GPU sync without app-provided semaphores**: core OpenXR gives no sync
|
||||
primitive for swapchain images; MVP uses `vkQueueWaitIdle`.
|
||||
- **Present family mismatch** between the app's graphics family and our window
|
||||
surface.
|
||||
- **Duplicate GLFW instances** in one process (engine + runtime); fallback to
|
||||
ImGui Win32 backend.
|
||||
- **Format negotiation**: only advertise formats we can actually create and
|
||||
sample.
|
||||
- **Version encoding** for Vulkan requirements (§2.5).
|
||||
|
||||
## 6. References
|
||||
|
||||
- OpenXR-SDK `hello_xr` — what a runtime must answer.
|
||||
- Monado — reference open-source runtime (negotiation + compositor).
|
||||
- `openxr_loader_negotiation.h` — exact negotiation structs.
|
||||
- Engine source: `code/Engine/src/Engine/Core/ApplicationContext.cpp`,
|
||||
`code/Engine/src/Engine/Renderer/Headset.cpp`,
|
||||
`code/Engine/src/Engine/Input/XrInputHandler.cpp`.
|
||||
Reference in New Issue
Block a user