Screen mirroring MVP (ReplayKit broadcast extension) - #16
Merged
Merged
Conversation
Adds an iOS screen-mirror path alongside the camera: a ReplayKit broadcast-upload extension (LensLinkBroadcast) captures the whole screen plus system audio and streams it to the plugin over the existing wire protocol — so USB via usbmuxd works with no plugin transport change. iOS: - New app-extension target sharing Protocol/StreamClient/VideoEncoder with the app (no code duplication). SampleHandler encodes screen frames with VideoToolbox H.264 (rebuilds the encoder on rotation) and converts .audioApp buffers to canonical 48 kHz stereo S16 PCM. - StreamClient gains sourceKind + sendScreenAudio; HELLO/video-config carry "kind": "camera"|"screen". - In-app "Mirror screen to OBS" button (RPSystemBroadcastPickerView) for discoverability. System audio only — mic omitted by design. Plugin: - OBSC_PKT_SCREEN_AUDIO (type 10): 48 kHz stereo S16LE, played via obs_source_output_audio (source now advertises OBS_SOURCE_AUDIO); pts shares the video clock so A/V stay aligned. - Reads the kind field; labels the source "(screen)". Protocol/README docs updated. Plugin builds clean with -Wall -Wextra -Werror. iOS half is validated by CI compile; needs on-device testing (broadcast extensions can't be exercised in the simulator build). Release-Skip: true
…non-failable Release-Skip: true
MyNamesEMurray
marked this pull request as ready for review
July 13, 2026 11:52
Sideloading tools re-sign with a remapped bundle id, so the hardcoded extension id stopped matching and the broadcast picker fell back to listing every provider on the phone instead of pre-selecting LensLink. Release-Skip: true
The extension has no UI, so a listener failure (port taken by the camera stream) or OBS never dialing in looked like 'recording but nothing happens'. End the broadcast with a descriptive error instead: iOS shows it to the user as an alert. Release-Skip: true
'Waiting for iPhone on USB' with a live broadcast means the mux connect is refused device-side. The alert now reports the listener's actual state and how many connections ever arrived, separating listener-never- ready / nothing-arrived / handshake-died without console access. Release-Skip: true
… listening' 'Waiting for iPhone on USB' covered both, which made it impossible to tell whether a failing screen broadcast meant the phone wasn't enumerated at all or its listener refused the connection. Release-Skip: true
Apple Mobile Device Service (Windows) pretty-prints its ListDevices reply, and region_string required <key> and <string> to be adjacent — so SerialNumber never parsed there: the USB device dropdown stayed empty forever and UDID pinning silently never engaged. Match the two tags separately (as the integer parser already did). Unit-tested against pretty-printed and compact replies. Release-Skip: true
- Installer now deletes the pre-1.0 ios-camera-source plugin: if left in
place it loads first, registers the source id, and OBS rejects the
current plugin as a duplicate — users silently keep running old code
('Source ios_camera_source already exists' in the log). README
troubleshooting entry added for manual installs.
- App gains a 'Check broadcast link' diagnostic that connects to the
broadcast extension's listener locally, separating extension-side
failures from OBS/transport ones without console access.
Release-Skip: true
No red broadcast indicator means the extension never launches; the usual cause on sideloaded installs is the signing tool stripping PlugIns (while the picker shows a stale replayd cache entry). The screen-mirror section now inspects the installed bundle and says whether the .appex is actually there. Release-Skip: true
New workflow builds, signs, and uploads to TestFlight on published releases (or manual dispatch), using an App Store Connect API key from repo secrets — no certificates in the repo. project.yml gains MARKETING_VERSION/CURRENT_PROJECT_VERSION (CI drives the build number from the run number). docs/TESTFLIGHT.md covers the one-time setup. Canonical Apple provisioning should also resolve the broadcast extension failing to launch under third-party re-signing. Release-Skip: true
Release-Skip: true
…evices) Automatic signing archives with a development profile by default, which requires a registered device — none exist on a CI runner, so the archive failed with 'No profiles for ... were found'. Forcing Apple Distribution at archive time needs no device list and lets the API key mint the cloud-managed cert. Release-Skip: true
…matic signing) Release-Skip: true
…vice) Automatic signing during archive defaults to a development profile (get-task-allow=1), which requires a registered device the CI account lacks. Archive with CODE_SIGNING_ALLOWED=NO and let -exportArchive (signingStyle automatic + API key) create the distribution profile and sign — distribution profiles need no device. Release-Skip: true
com.exaltedpixels.LensLink is registered to another Apple account (globally-unique bundle ids), so it can't be used for the app record. Set an explicit unique id on the app + its .broadcast child; the runtime-derived broadcast-picker id follows automatically. Docs and the TestFlight workflow comment updated. Release-Skip: true
- All four interface orientations (iPad multitasking requirement). - Select Xcode 26 on the runner: uploads must be built with the iOS 26 SDK; the image default (16.4) is rejected at validation. Release-Skip: true
The browser panel at localhost:9980 exposes zoom/exposure/focus/flashlight, none of which apply to a screen mirror. Track a per-source is_screen flag (set from the HELLO kind), expose it via ios_camera_is_screen(), and emit it in /api/status. The panel now swaps the controls for a short note when the connected source is a screen. Release-Skip: true
…bounding box Adds a Screen mirroring section explaining the broadcast picker, how to pin the source size with OBS's Scale-to-inner-bounds (Fit) transform (with a screenshot), and that the browser panel hides camera controls for screens. Release-Skip: true
…gnostics Fixes the intermittent black screen: with hardware decoding on, some screen-mirror resolutions (tall, non-16-aligned, e.g. 886x1918) are silently rejected by the GPU decoder on Windows — avcodec_send_packet succeeds but avcodec_receive_frame never yields a frame, so decode 'works' while the source stays black and the plugin looks connected. The plugin now detects this (a keyframe went in but no frame came out within ~10 packets), destroys the hardware decoder, recreates it in software, and asks the device for a fresh keyframe (new CONTROL command, honoured by the screen extension) so the picture returns in a fraction of a second instead of waiting up to ~2s for the next periodic keyframe. Hardware decoding stays the default and recovers itself. Diagnostics to pinpoint any remaining stalls, all funnelled to the OBS log: - Plugin: per-connection heartbeat (packets, keyframes, decoded frames, errors, bitrate, decoded size/format, GPU/CPU) every ~2s; a 'first decoded frame' line; and a 'Verbose diagnostics' source toggle (on by default). - h264-decoder: frames-output and last-frame getters. - Extension/app: matching heartbeat (screen samples, encoded/sent/dropped frames, connection state) forwarded over a new DIAG wire packet so the phone's own counters appear in the OBS log too — the extension has no console of its own. - New docs/DEBUGGING-SCREEN-MIRROR.md maps log patterns to causes. Also adds a one-line HEVC toggle in the extension (preferHEVC) to A/B whether HEVC's lower bitrate reduces backpressure drops; falls back to H.264 when unsupported, and the plugin already decodes either. Release-Skip: true
The audio transport can be perfectly healthy while carrying pure silence — DRM apps (Apple Music, Spotify, Netflix) mute or pause themselves the moment a broadcast starts, so 'no audio in OBS' can look identical to a broken pipeline in the packet counters. The heartbeat now includes the loudest sample since the last beat (peak=NN%), which separates 'real audio arriving' from 'a healthy stream of silence' at a glance. Doc table updated with the DRM and OBS audio-monitoring rows. Release-Skip: true
MyNamesEMurray
added a commit
that referenced
this pull request
Jul 14, 2026
The add-source menu now offers LensLink Camera and LensLink Screen as distinct source types with their own property sheets, sharing one connection/decode engine: - Screen source: connection/decoding/diagnostics only — no lip-sync group, no web panel; screen-specific help text. - Strict kind enforcement: each type accepts only its own stream, rejecting at HELLO with an actionable status and a 3 s redial backoff. - Camera sources no longer advertise audio (no more dead mixer meter); only Screen sources appear in the mixer. - New optional "Disconnect when this source isn't shown anywhere" toggle (off by default) on both types: hidden sources release the phone entirely and reconnect when shown. - Existing scenes keep working: the camera source id is unchanged. Together with the screen-mirroring MVP (#16) already on main, this makes the first minor release since v1.0. Release-Bump: minor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MVP for mirroring the whole iPhone/iPad screen (video + system audio) into the same OBS source, alongside the camera. Draft — needs on-device testing before merge (broadcast extensions can't run in the CI simulator/compile build; CI only proves it compiles).
How it works
A ReplayKit broadcast-upload extension (
LensLinkBroadcast) captures the screen and app audio and streams it to the plugin over the existing wire protocol — so USB via usbmuxd works with no plugin transport change. The plugin dials port 9979 as always; the HELLO's newkind: "screen"tells it what it's receiving.Scope (as agreed)
obs_source_output_audio)Changes
iOS — new app-extension target sharing
Protocol/StreamClient/VideoEncoder(no duplication);SampleHandlerdoes capture→encode→send + audio conversion;StreamClientgainssourceKind+sendScreenAudio; in-app Mirror screen to OBS button (RPSystemBroadcastPickerView) for discoverability.Plugin —
OBSC_PKT_SCREEN_AUDIO(type 10); source advertisesOBS_SOURCE_AUDIO; audio pts shares the video clock so A/V stay aligned; source labels itself "(screen)". Builds clean with-Wall -Wextra -Werror.Docs — PROTOCOL.md (kind field + packet 10) and README updated.
Known MVP limitations to verify on device
Tagged
Release-Skip: trueso merging won't auto-cut a release until you've tested it and are ready.🤖 Generated with Claude Code
https://claude.ai/code/session_019Qrp83VmgTR1fJmSeBRtHR
Generated by Claude Code