Pointer Input And Joystick

ReGame routes mouse, touch, and pen through one pointer system. Author gameplay and RSX controls once; platform adapters preserve the device kind and pointer id while the shared runtime owns hit testing, capture, events, and cleanup.

Joystick is a first-class retained RSX element. It is authored in .rsx, styled with RStyle, laid out by UISurface, and rendered by the active native backend. It is not a React Native view and it does not run gameplay input through JavaScript each frame.

Type Hierarchy

The script-facing hierarchy is intentionally small and searchable:

UIEvent
└── UIPointerEvent
    └── UIJoystickEvent

UIElement
└── Joystick

GameObject
└── onPointerDown / onPointerMove / onPointerUp / onPointerCancel / onClick
    └── PointerHit3DEvent

UIElement and GameObject are separate roots. UISurface elements are retained native UI objects; they are not hidden scene GameObjects. A world object receives pointer events through its 3D physics collider, while an RSX element receives pointer events through UISurface layout and hit testing.

Routing Order

For each platform pointer event, ReGame uses this order:

1. Update authoritative PointerState once.
2. Route to the captured target, when that pointer already owns one.
3. Otherwise hit test Screen InterfaceHosts from front to back.
4. Project the camera ray onto World InterfaceHosts from nearest to farthest.
5. Route to the topmost actionable scene ButtonComp, if present.
6. If UI and scene buttons did not handle the event, raycast 3D bodies and areas.
7. If the Down event is still unhandled, offer it to a Dynamic or Following Joystick on the frontmost Screen InterfaceHost.
8. Project the routed result into legacy TouchState compatibility events.
9. Emit the Game-wide onPointer... notification with its handled state.

This makes ordinary RSX controls and gameplay ButtonComp objects win over movement. A layout-only View or ordinary 2D render item does not swallow input merely because it covers the viewport. An interactive Pressable, Button, field, slider, toggle, color picker, Fixed Joystick, or ButtonComp does capture or consume the pointer. A held ButtonComp also consumes a second Down at its location without accepting a second capture, so that finger cannot accidentally start movement. Dynamic and Following Joysticks are deliberately deferred: actionable screen UI, World InterfaceHosts, gameplay buttons, and the closest 3D collider receive the Down decision first. TouchState is a post-routing compatibility projection, not a competing input listener, so legacy buttons cannot activate alongside a deferred Joystick. The closest 3D collider is also the occluder. If it has pointer listeners, it captures the world event; if it has none, the world lane stays unhandled and ReGame offers the Down to the deferred Joystick. ReGame never makes walls, floors, or props pointer-transparent merely because they have no callback, so an actionable object behind an inert collider cannot be clicked through it.

Screen hosts that overlap use scene traversal order: the later host is visually and interactively above the earlier host. World hosts use ray distance and their authored Depth Test setting.

Pointer Identity And Capture

Every pointer is identified by both device and platform id. Mouse id 0 and touch id 0 are different contacts. Multiple touches remain independent, so one finger can hold a Joystick while another presses a button.

Capture begins when an interactive target handles Down. Move, Up, and Cancel continue to that same target even when the pointer leaves its rectangle. Up releases capture. Cancel releases capture without clicking. Invalidating a captured scene button or 3D target terminates that capture instead of letting the same pointer fall through to movement; hierarchy deactivation also releases a ButtonComp held-key state immediately. A state-driven UIComponent commit retains compatible elements so visual feedback can update during a drag without losing the eventual Up event. Disabling, structurally replacing, source-rebuilding, unmounting, or destroying the captured InterfaceHost emits cancellation before its retained surface disappears. Runtime pause, native surface detach, viewport resize, and desktop window focus loss also cancel all active pointers through the same target event lane. This covers operating systems that do not deliver a terminal touch event and prevents a cached non-zero Joystick vector from surviving a lifecycle boundary.

Mouse, touch, and pen use the same coordinate mapping. Touch does not synthesize hover. Mouse and pen can update hover without beginning a contact.

RSX Taps, Double-Taps, And Swipes

Use onClick for a completed tap, onDoubleClick for two completed taps on the same semantic element within 500 ms, and the pointer lifecycle when a control needs drag or swipe behavior:

class PuzzleCell extends UIComponent {
  private dragX: number = 0
  private marked: bool = false
  private revealed: bool = false

  public beginSwipe(event: UIPointerEvent): void {
    dragX = 0
  }

  public updateSwipe(event: UIPointerEvent): void {
    dragX = dragX + event.deltaX
    if (abs(dragX) >= 24) {
      marked = true
    }
  }

  public finishSwipe(event: UIPointerEvent): void {
    dragX = 0
  }

  public toggleMark(event: UIPointerEvent): void {
    marked = !marked
  }

  public reveal(event: UIPointerEvent): void {
    revealed = true
  }

  render {
    <Pressable
      automationId="puzzle.cell.4"
      accessibilityLabel="Puzzle cell"
      onClick={toggleMark}
      onDoubleClick={reveal}
      onPointerPress={beginSwipe}
      onPointerMove={updateSwipe}
      onPointerRelease={finishSwipe}
    >
      <Text>{revealed ? "Found" : (marked ? "X" : "")}</Text>
    </Pressable>
  }
}

Pointer, click, double-click, and context-menu handlers may receive one UIPointerEvent; they do not receive separate positional arguments. Zero-argument click handlers remain valid. deltaX and deltaY are the movement since the previous native pointer event. targetId and currentTargetId let one reusable VirtualList or VirtualGrid item handler identify the generated item that was activated. State changed by any number of move, release, cancellation, or imperative element writes is committed once on the next InterfaceHost frame.

An InterfaceHost commit has separate structure, layout, paint, and semantic invalidation:

• compatible component output patches the retained element tree without unmounting the host or releasing pointer capture;
• layout updates only dirty nodes in the retained Yoga tree;
• paint regenerates render data only for dirty elements and reuses cached output for clean elements and clean InterfaceHosts;
• semantic changes republish accessibility independently of paint;
• a structural change replaces the surface and follows the normal cancellation lifecycle.

Imperative setters update their authoritative retained value before returning, but they do not force layout or rendering in the setter. This allows several writes or pointer moves in one frame to produce one immutable render snapshot. ReGame also preserves double-click identity by semantic id across state-driven commits, so the first click may change selection or appearance without erasing the second-click candidate.

The normal onClick handler runs for each activation before onDoubleClick. When a control assigns different meanings to a single tap and a double-tap, make the double-tap result authoritative over the two preceding click results.

Joystick

import styles from "./MovementControls.rstyle"

class MovementControls extends UIComponent {
  render {
    <View style={styles.root}>
      <Joystick
        automationId="controls.move"
        accessibilityLabel="Move player"
        mode="dynamic"
        visibility="whenTouched"
        deadZone={0.12}
        knobRatio={0.42}
        style={styles.joystick}
      />
    </View>
  }
}

extends StyleSet

export default StyleSet.create({
  root: {
    width: Project.viewport.width,
    height: Project.viewport.height,
    position: "relative"
  },
  joystick: {
    position: "absolute",
    left: 24,
    bottom: 34,
    width: 112,
    height: 112,
    backgroundColor: "#17203370",
    borderColor: "#FFFFFFB8",
    borderWidth: 2,
    borderRadius: 56,
    color: "#FFFFFFFF"
  }
})

Properties

Property	Type	Default	Meaning
valueX	number	0	Current horizontal value in -1…1. Readable from retained script handles.
valueY	number	0	Current vertical value in -1…1; positive points upward.
deadZone	number	0.12	Radial center region that resolves to zero, clamped below 1.
knobRatio	number	0.42	Knob diameter as a fraction of the base diameter.
mode	"fixed" | "dynamic" | "following"	"fixed"	Selects an authored base, a new origin at each accepted Down, or a dynamic origin that follows beyond the clamp radius.
visibility	"always" | "whenTouched"	"always"	Controls whether the base is visible while idle, independently of input mode.
disabled	boolean	false	Keeps the control visible but prevents pointer capture.
enabled / interactable	boolean	true	Inverse script-facing aliases of disabled.

The output is radial, not square. Values outside the base are clamped to magnitude 1. Values after the dead zone are rescaled smoothly, so movement does not jump when the pointer first leaves the center.

mode and visibility are independent. mode="fixed" uses the authored base position and participates in normal UISurface hit testing. mode="dynamic" places the base at the first otherwise-unhandled primary Down. mode="following" starts like Dynamic, then shifts the base only when the pointer would move beyond the base radius, keeping the tip clamped without making the player lift and retouch.

Dynamic and Following activation uses the Joystick's effective parent bounds, clipped by every ancestor. Put a single movement Joystick directly under a full-screen root to accept a first press anywhere on screen. For twin sticks, wrap each Joystick in a separate left/right region; those clipped parent regions deterministically choose which stick receives each Down. A Following base never follows outside its captured activation region, though its output remains radially clamped while the pointer is held beyond that boundary.

For Dynamic and Following modes, the authored width and height define the base diameter, while the authored position is the idle render and accessibility fallback. Down captures one pointer and starts at (0, 0). Render and accessibility bounds follow the active center. Up or Cancel resets to zero, clears capture, and returns to the authored idle position. With visibility="whenTouched", that idle position is not drawn. A second pointer cannot steal an active Joystick, so it remains available for buttons and other multitouch actions.

Dynamic and Following modes are Screen InterfaceHost behaviors. Use Fixed mode for a World InterfaceHost; world UI and 3D picking must keep their spatial hit-test semantics.

Retained script handles can read and set mode and visibility. Changing mode, disabling the Joystick, hiding it, rebuilding its InterfaceHost, or making that host unavailable while a pointer is held first resets the vector and emits canonical onChange(0, 0) plus onCancel, then applies the configuration transition. Invalid mode or visibility strings are rejected without mutation. These configuration writes do not add separate onModeChanged or onVisibilityChanged events: the setter's synchronous return is authoritative for imperative writes, while authored RSX changes replace the retained surface through the existing InterfaceHost unmount/layout/geometry/mount lifecycle.

Events

Event	Timing
onStart	After this Joystick captures a pointer and computes its initial value. Dynamic and Following modes begin at zero.
onChange	After the authoritative normalized value changes, including reset to zero.
onEnd	After a normal release resets the value to zero.
onCancel	After capture loss resets the value to zero.

Each callback receives UIJoystickEvent, which inherits the pointer fields and adds valueX and valueY:

class PlayerMovement extends Component {
  let stick: Joystick
  let moveX: number = 0
  let moveY: number = 0

  function _ready() {
    stick = UI.findById<Joystick>("controls.move")
    if (stick != null) {
      stick.onChange((event: UIJoystickEvent) => {
        moveX = event.valueX
        moveY = event.valueY
      })
    }
  }
}

Subscriptions return EventController and are owner-bound. ReGame cancels them when the subscribing script, target InterfaceHost, or retained element is removed. You can also keep the controller and call cancel() explicitly.

Desktop Accessibility And Automation

Screen InterfaceHost controls publish one semantic tree and one action dispatcher. The ReGame MCP bridge, macOS Accessibility, and Windows UI Automation all consume that same UIAccessibleNode data and invoke the same UIAccessibilityAction; a control must not need a separate Computer Use handler. In embedded play, the player sends semantic snapshots to the visible editor host and the editor transforms their bounds into the embedded viewport. A standalone desktop player publishes the same trees directly. World InterfaceHosts are excluded from this native screen-coordinate adapter because their bounds require camera and occlusion-aware spatial semantics.

A Joystick exposes the setValue action even though it is not a text field. Native macOS presents that action as a settable accessibility value, and Windows presents it through UI Automation's Value pattern. Supply the normalized vector as "x,y", for example "1,0" to move right and "0,0" to release:

{"id":"controls.move","action":"setValue","value":"1,0"}

The zero-to-nonzero transition emits onStart and onChange. Returning to zero emits onChange and onEnd. If focus, the native surface, or the InterfaceHost disappears while an automation value remains nonzero, ReGame resets the value first and emits onChange followed by onCancel. This gives automation the same safe movement lifecycle as a physical pointer and prevents a worker from continuing to walk after control is lost.

Pointer Event Payloads

UIPointerEvent contains:

• pointerId, device, button, primary, and canceled;
• x, y, deltaX, and deltaY in the target UISurface coordinate space;
• targetId and currentTargetId for stable semantic lookup. During a captured pointer move, targetId is the interactive element currently under the pointer while currentTargetId remains the pressed element whose handler is receiving the move;
• name and the authored bindingKey.

UIJoystickEvent adds valueX and valueY.

PointerHit3DEvent contains the device fields plus screenPosition, screenDelta, world position, world surface normal, and ray distance.

Tapping 3D World Objects

A GameObject needs an active 3D physics body or area shape to participate in world pointer picking. Subscribe on the object that owns that collider:

class Balloon extends Component {
  function _ready() {
    self.onPointerDown((event: PointerHit3DEvent) => {
      self.trigger("balloon.pressed", event.position)
    })

    self.onClick((event: PointerHit3DEvent) => {
      self.trigger("balloon.popped", event.position)
      self.destroy()
    })
  }
}

onClick requires Down and Up to resolve to the same closest collider. An inert collider in front of the captured object cancels click eligibility instead of being skipped. Use onPointerDown for immediate tap behavior, and onClick when dragging off or becoming occluded should cancel activation.

The 3D camera projection supports perspective and orthographic cameras. World InterfaceHosts use the same ray, transform chain, front/back-face rule, surface bounds, and depth occlusion as their rendered rectangle.

Platform Behavior

Platform	Pointer source	Shared runtime result
Desktop	Mouse, supported pen devices	UISurface capture, Joystick, World InterfaceHost projection, 3D picking
Android	MotionEvent pointer ids	The same capture, Joystick, projection, and 3D picking path
iOS	UITouch identities	The same capture, Joystick, and 3D picking path through Metal rendering
Web	Browser Pointer Events	Mouse, touch, and pen use the same pointer ids and capture model

Native iOS currently renders Screen InterfaceHosts, including Joystick. World InterfaceHost drawing still depends on the native Metal 3D world-surface renderer; until that renderer is complete, the shared projection path exists but an invisible World host should not be used as an interaction target on iOS.

Performance Notes

• Pointer adapters submit events only when input changes; they do not poll React Native or rebuild the RSX tree every frame.
• Joystick motion mutates its retained native state and marks only UISurface rendering dirty.
• Dynamic and Following Joystick activation uses the Game's unhandled-pointer lane; it does not add a full-screen hit-test element or collider.
• Camera projection and physics picking run only for relevant pointer events.
• Captures use stable integer keys and direct target lookup rather than repeated full hit tests during a drag.
• Use one InterfaceHost for controls that share lifecycle and stacking. Split hosts only when transforms, visibility, or lifecycle genuinely differ.