diff --git a/README.md b/README.md index 79c4085..1f04c2e 100644 --- a/README.md +++ b/README.md @@ -5,13 +5,15 @@ geometry, raise, and engagement lifecycle — the mechanism behind an infinite-canvas "window stream" tile, extracted as a federated plugin so a second platform (Windows) can be added without touching consumers. -> **Status.** macOS backend is complete and in production use (extracted from -> the Campus `work_canvas` Runner): per-window capture, geometry/raise/engage, -> and the engagement-event push. A runnable `example/` app exercises the API -> (list windows → capture → engage). The **Windows** backend -> (`flutter_window_portal_windows`) implements window enumeration + control + -> the engagement events today; per-window **capture** (Windows.Graphics.Capture -> → Flutter texture) is the remaining slice — see the roadmap below. +> **Status.** Both backends implement the full v1 contract. **macOS** is in +> production use (extracted from the Campus `work_canvas` Runner): +> ScreenCaptureKit capture, geometry/raise/engage, and the engagement-event +> push. **Windows** implements the same surface over Windows.Graphics.Capture +> (frames reach the Flutter texture as DXGI shared handles, so they never leave +> the GPU), Win32 window control, and `SetWinEventHook` engagement events — see +> [its README](packages/flutter_window_portal_windows/README.md) for the +> platform-specific conventions. A runnable `example/` app exercises the API on +> both (list windows → capture → engage). ## Why a plugin (and why federated) @@ -23,8 +25,8 @@ consumer's tile code doesn't change a line. Unlike a CEF-style plugin, there is **no bundled runtime binary**: every backend compiles from source against OS frameworks (macOS: ScreenCaptureKit -+ Accessibility + CoreGraphics; Windows: Windows.Graphics.Capture + Win32 + -UI Automation). So there is nothing to fetch, content-hash-match, or drift — ++ Accessibility + CoreGraphics; Windows: Windows.Graphics.Capture + Direct3D ++ Win32). So there is nothing to fetch, content-hash-match, or drift — the failure mode that a fetched prebuilt (like `cef_host`) introduces simply does not exist here. @@ -35,7 +37,7 @@ flutter_window_portal/ app-facing API — WindowPortal packages/ flutter_window_portal_platform_interface/ the contract + method-channel default flutter_window_portal_macos/ endorsed macOS impl (ScreenCaptureKit + AX) - flutter_window_portal_windows/ endorsed Windows impl (planned) + flutter_window_portal_windows/ endorsed Windows impl (WGC + Win32) ``` ## The contract diff --git a/packages/flutter_window_portal_windows/CHANGELOG.md b/packages/flutter_window_portal_windows/CHANGELOG.md index 41cc7d8..802acb7 100644 --- a/packages/flutter_window_portal_windows/CHANGELOG.md +++ b/packages/flutter_window_portal_windows/CHANGELOG.md @@ -1,3 +1,12 @@ -## 0.0.1 +## 0.1.0 -* TODO: Describe initial release. +* Per-window capture via Windows.Graphics.Capture, delivered into a Flutter + texture as a DXGI shared handle: `startStream`, `stopStream`, + `reconfigureStream`, `streamStatus`. +* One-shot PNG grabs for a window (`captureSingleFrame`) and a display + (`captureSingleDisplayFrame`), encoded with WIC off the platform thread. +* Engagement lifecycle events pushed from `SetWinEventHook`: `alignmentBroken` + on external move/resize, `disengaged(windowClosed)` on destroy, and + `disengaged(thirdApp | hostActivated)` on focus change. +* `detectSeparateWindows` reports same-process windows overlapping the source. +* Window enumeration and geometry/raise/engage control over Win32. diff --git a/packages/flutter_window_portal_windows/README.md b/packages/flutter_window_portal_windows/README.md index 5d705e7..92764f8 100644 --- a/packages/flutter_window_portal_windows/README.md +++ b/packages/flutter_window_portal_windows/README.md @@ -1,15 +1,84 @@ # flutter_window_portal_windows -A new Flutter plugin project. +The Windows implementation of [`flutter_window_portal`][portal] — capture a +native OS window into a Flutter `Texture` and control its geometry, raise, and +engagement lifecycle. -## Getting Started +This package is **endorsed**: depend on `flutter_window_portal` and this +implementation is used automatically on Windows. You never import it directly. -This project is a starting point for a Flutter -[plug-in package](https://flutter.dev/to/develop-plugins), -a specialized package that includes platform-specific implementation code for -Android and/or iOS. +## How it works -For help getting started with Flutter development, view the -[online documentation](https://docs.flutter.dev), which offers tutorials, -samples, guidance on mobile development, and a full API reference. +| Concern | Windows API | +| --- | --- | +| Window enumeration | `EnumWindows` + `QueryFullProcessImageName` | +| Capture | `Windows.Graphics.Capture` (`Direct3D11CaptureFramePool`) | +| Frame delivery | D3D11 → DXGI shared handle → Flutter `GpuSurfaceTexture` | +| One-shot PNG | WGC → staging texture → WIC encoder | +| Geometry / raise | `SetWindowPos` · `SetForegroundWindow` · `SetCursorPos` | +| Engagement events | `SetWinEventHook` | +Captured frames never leave the GPU: each frame is copied into one of two +BGRA textures created with `D3D11_RESOURCE_MISC_SHARED`, and the engine opens +that shared handle directly. The two textures rotate so the engine is never +reading the surface being overwritten. + +Everything compiles from source against the OS — there is no bundled runtime +binary to fetch or version-match. + +## Requirements + +- Windows 10 1803 (build 17134) or newer for capture. `startStream` and the + one-shot grabs return a typed `capture_unsupported` error on older systems; + window control and enumeration still work. +- Cursor omission needs Windows 10 2004; suppressing the yellow capture border + needs Windows 11. Both are applied when the OS exposes them and skipped + cleanly when it does not. +- No permission prompt. Unlike macOS Screen Recording, Windows gates none of + this, so `checkAccessibilityPermission` always reports `granted`. + +## Platform notes + +These are the places where the Windows backend must make a choice the shared +contract does not dictate: + +- **Window ids** are `HWND` values, stringified. They are not stable across + restarts of the source app. +- **Display ids** are `HMONITOR` values. Windows has no `CGDirectDisplayID` + equivalent, so `captureSingleDisplayFrame` takes the monitor handle — and + accepts `'0'` to mean the primary display, which is what a consumer with no + display list can ask for. Display thumbnails are capped at 480px wide, the + same cap the macOS backend applies. +- **Sizes** cross the channel as logical units (physical pixels ÷ the window's + DPI scale), matching what the macOS backend reports in points. Capture itself + runs at the source's native pixel size. +- **`getContentMeasurement`** always reports zero insets. The transparent + drop-shadow margin it exists to crop is an Electron-on-macOS artifact; DWM + composites shadows outside the window, so there is nothing to trim. +- **`reconfigureStream`** reports rather than applies. The capture pool resizes + itself when a frame's content size changes, so this call tells the consumer + whether the source has outgrown the frames published so far. + +## Engagement events + +The backend pushes the same three events the macOS backend does, driven by +`SetWinEventHook` rather than an `AXObserver`: + +- `alignmentBroken` — `EVENT_OBJECT_LOCATIONCHANGE` moved the engaged window + away from where engage left it. Reported once per engagement; engage and + conform mask their own moves for 1.5s so only genuinely external motion + counts. +- `disengaged(windowClosed)` — `EVENT_OBJECT_DESTROY`. +- `disengaged(thirdApp | hostActivated)` — `EVENT_SYSTEM_FOREGROUND` moved + focus to another app, or back to the Flutter host. Focus landing on another + window of the *source's own* process (a modal dialog, a save sheet) keeps the + engagement alive. + +Location and destroy hooks are scoped to the source's process — a system-wide +`EVENT_OBJECT_LOCATIONCHANGE` hook fires constantly. + +Minimize is deliberately *not* an event: like macOS, it is surfaced through the +`isWindowMinimized` poll so the consumer can show an overlay over the frozen +stream rather than tear the tile down. + +[portal]: https://github.com/FlutterFlow/flutter_window_portal diff --git a/packages/flutter_window_portal_windows/pubspec.yaml b/packages/flutter_window_portal_windows/pubspec.yaml index 6092073..eaeb8d9 100644 --- a/packages/flutter_window_portal_windows/pubspec.yaml +++ b/packages/flutter_window_portal_windows/pubspec.yaml @@ -1,5 +1,5 @@ name: flutter_window_portal_windows -description: "Windows implementation of flutter_window_portal: per-window capture via Windows.Graphics.Capture into a Flutter texture, plus Win32/UIA geometry, raise, and engagement control." +description: "Windows implementation of flutter_window_portal: per-window capture via Windows.Graphics.Capture into a Flutter texture, plus Win32 geometry, raise, and engagement control." version: 0.1.0 homepage: https://github.com/FlutterFlow/flutter_window_portal repository: https://github.com/FlutterFlow/flutter_window_portal @@ -32,7 +32,7 @@ flutter: platforms: windows: # C++ host plugin — capture (Windows.Graphics.Capture) + window control - # (Win32 / UI Automation). + # and engagement events (Win32). pluginClass: FlutterWindowPortalWindowsPluginCApi # Dart entrypoint: endorses the default method-channel platform instance. dartPluginClass: FlutterWindowPortalWindows diff --git a/packages/flutter_window_portal_windows/windows/CMakeLists.txt b/packages/flutter_window_portal_windows/windows/CMakeLists.txt index 4c78948..785db96 100644 --- a/packages/flutter_window_portal_windows/windows/CMakeLists.txt +++ b/packages/flutter_window_portal_windows/windows/CMakeLists.txt @@ -20,6 +20,20 @@ set(PLUGIN_NAME "flutter_window_portal_windows_plugin") list(APPEND PLUGIN_SOURCES "flutter_window_portal_windows_plugin.cpp" "flutter_window_portal_windows_plugin.h" + "engagement_watcher.cpp" + "engagement_watcher.h" + "platform_task_runner.cpp" + "platform_task_runner.h" + "capture/capture_controller.cpp" + "capture/capture_controller.h" + "capture/d3d_context.cpp" + "capture/d3d_context.h" + "capture/png_encoder.cpp" + "capture/png_encoder.h" + "capture/texture_sink.cpp" + "capture/texture_sink.h" + "capture/wgc_session.cpp" + "capture/wgc_session.h" ) # Define the plugin library target. Its name must not be changed (see comment @@ -35,18 +49,59 @@ add_library(${PLUGIN_NAME} SHARED # full control over build settings. apply_standard_settings(${PLUGIN_NAME}) +# C++/WinRT routes its coroutine support through when +# building as C++17, and MSVC 14.51 hard-errors on that header (STL1011) ahead +# of removing it. Under C++20 it selects the standard instead. +# apply_standard_settings above asks for C++17; this raises the floor for this +# target only, leaving the rest of the build alone. +target_compile_features(${PLUGIN_NAME} PRIVATE cxx_std_20) + # Symbols are hidden by default to reduce the chance of accidental conflicts # between plugins. This should not be removed; any symbols that should be # exported should be explicitly exported with the FLUTTER_PLUGIN_EXPORT macro. set_target_properties(${PLUGIN_NAME} PROPERTIES CXX_VISIBILITY_PRESET hidden) -target_compile_definitions(${PLUGIN_NAME} PRIVATE FLUTTER_PLUGIN_IMPL) +# NOMINMAX: defines min/max as macros, which breaks std::min and +# std::max in the capture sources. +target_compile_definitions(${PLUGIN_NAME} PRIVATE FLUTTER_PLUGIN_IMPL NOMINMAX) # Source include directories and library dependencies. Add any plugin-specific # dependencies here. target_include_directories(${PLUGIN_NAME} INTERFACE "${CMAKE_CURRENT_SOURCE_DIR}/include") -target_link_libraries(${PLUGIN_NAME} PRIVATE flutter flutter_wrapper_plugin dwmapi shcore) + +# The C++/WinRT projection headers (winrt/Windows.Graphics.Capture.h and +# friends) ship in the Windows SDK under Include//cppwinrt. The MSVC +# toolchain normally puts that on the include path alongside um/ and shared/, +# but which SDK components are installed varies, so locate it and add it when +# it is not already reachable. Nothing is added when the compiler can already +# find the headers. +find_path(FWP_CPPWINRT_INCLUDE_DIR + NAMES "winrt/Windows.Graphics.Capture.h" + PATHS + "$ENV{WindowsSdkDir}Include/$ENV{WindowsSDKVersion}cppwinrt" + "$ENV{WindowsSdkDir}Include/${CMAKE_VS_WINDOWS_TARGET_PLATFORM_VERSION}/cppwinrt" + "C:/Program Files (x86)/Windows Kits/10/Include/${CMAKE_VS_WINDOWS_TARGET_PLATFORM_VERSION}/cppwinrt" + NO_DEFAULT_PATH +) +if(FWP_CPPWINRT_INCLUDE_DIR) + target_include_directories(${PLUGIN_NAME} PRIVATE "${FWP_CPPWINRT_INCLUDE_DIR}") +endif() + +# d3d11/dxgi: the capture device and the shared textures Flutter opens. +# windowscodecs + shlwapi: PNG encoding for the one-shot grabs. +# runtimeobject: the WinRT activation/string entry points C++/WinRT calls. +target_link_libraries(${PLUGIN_NAME} PRIVATE + flutter + flutter_wrapper_plugin + dwmapi + shcore + d3d11 + dxgi + windowscodecs + shlwapi + runtimeobject +) # List of absolute paths to libraries that should be bundled with the plugin. # This list could contain prebuilt libraries, or libraries created by an diff --git a/packages/flutter_window_portal_windows/windows/capture/capture_controller.cpp b/packages/flutter_window_portal_windows/windows/capture/capture_controller.cpp new file mode 100644 index 0000000..c887885 --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/capture/capture_controller.cpp @@ -0,0 +1,558 @@ +#include "capture_controller.h" + +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "../platform_task_runner.h" +#include "d3d_context.h" +#include "png_encoder.h" +#include "texture_sink.h" +#include "wgc_session.h" + +namespace flutter_window_portal_windows { + +namespace { + +using flutter::EncodableMap; +using flutter::EncodableValue; + +// Matches the macOS one-shot grabber's bound: a source that never produces a +// frame must not hang the picker forever. +constexpr auto kOneShotTimeout = std::chrono::seconds(2); + +// The macOS display path caps thumbnails so a 6K display does not blow up the +// picker; mirror the cap so consumers get comparable bytes on both platforms. +constexpr UINT kDisplayThumbnailWidth = 480; + +std::optional GetInt64(const EncodableMap &map, const char *key) { + auto it = map.find(EncodableValue(std::string(key))); + if (it == map.end()) return std::nullopt; + if (std::holds_alternative(it->second)) { + return static_cast(std::get(it->second)); + } + if (std::holds_alternative(it->second)) { + return std::get(it->second); + } + return std::nullopt; +} + +// Everything a one-shot grab needs, so the worker can be a plain function +// rather than a lambda nested inside another lambda's captures. +struct OneShotRequest { + std::shared_ptr d3d; + PlatformTaskRunner *task_runner; + std::shared_ptr> result; + HWND hwnd; + HMONITOR monitor; + // Bounds the encoded image; 0 encodes at source size. + UINT cap_width; +}; + +void PostError(const OneShotRequest &request, std::string code, + std::string message) { + auto result = request.result; + request.task_runner->Post( + [result, code = std::move(code), message = std::move(message)]() { + result->Error(code, message); + }); +} + +struct MonitorSearch { + HMONITOR wanted; + bool found; +}; + +BOOL CALLBACK FindMonitorProc(HMONITOR monitor, HDC, LPRECT, LPARAM lparam) { + auto *search = reinterpret_cast(lparam); + if (monitor == search->wanted) { + search->found = true; + return FALSE; + } + return TRUE; +} + +// Grabs the first frame of a source and posts it back as PNG bytes. Runs on +// its own thread: waiting up to kOneShotTimeout on the platform thread would +// freeze the UI. +void RunOneShot(const OneShotRequest &request) { + D3DContext *d3d = request.d3d.get(); + + std::string create_error; + std::unique_ptr session = + request.monitor + ? WgcSession::CreateForMonitor(request.monitor, request.d3d, + &create_error) + : WgcSession::CreateForWindow(request.hwnd, request.d3d, + &create_error); + if (!session) { + PostError(request, request.monitor ? "display_not_found" : "window_not_found", + create_error); + return; + } + + // The first frame is copied to a staging texture on the capture thread; + // mapping and encoding happen here once capture has stopped, so the two can + // never touch the staging texture at the same time. + std::mutex frame_mutex; + std::condition_variable frame_ready; + Microsoft::WRL::ComPtr staging; + UINT frame_width = 0; + UINT frame_height = 0; + bool have_frame = false; + + auto on_frame = [&](ID3D11Texture2D *texture, UINT width, UINT height) { + std::lock_guard lock(frame_mutex); + if (have_frame) return; + auto target = d3d->CreateStagingTexture(width, height); + auto context = d3d->context(); + if (!target || !context) return; + + D3D11_TEXTURE2D_DESC source_desc{}; + texture->GetDesc(&source_desc); + D3D11_BOX box{}; + box.right = std::min(width, source_desc.Width); + box.bottom = std::min(height, source_desc.Height); + box.back = 1; + if (box.right == 0 || box.bottom == 0) return; + + context->CopySubresourceRegion(target.Get(), 0, 0, 0, 0, texture, 0, &box); + context->Flush(); + staging = target; + frame_width = box.right; + frame_height = box.bottom; + have_frame = true; + frame_ready.notify_one(); + }; + + std::string start_error; + if (!session->Start(on_frame, [](const std::string &) {}, &start_error)) { + PostError(request, "capture_failed", start_error); + return; + } + + { + std::unique_lock lock(frame_mutex); + frame_ready.wait_for(lock, kOneShotTimeout, [&] { return have_frame; }); + } + // Stops capture and waits out any in-flight callback, so |staging| is ours + // alone from here. + session->Stop(); + + if (!have_frame) { + PostError(request, "capture_failed", "one-shot capture timed out"); + return; + } + + auto context = d3d->context(); + D3D11_MAPPED_SUBRESOURCE mapped{}; + if (!context || + FAILED(context->Map(staging.Get(), 0, D3D11_MAP_READ, 0, &mapped))) { + PostError(request, "capture_failed", "could not read the captured frame"); + return; + } + + UINT target_width = 0; + UINT target_height = 0; + if (request.cap_width != 0 && frame_width > request.cap_width) { + target_width = request.cap_width; + target_height = std::max( + 1, static_cast(static_cast(frame_height) * + request.cap_width / frame_width)); + } + std::vector png; + const bool encoded = + EncodeBgraToPng(static_cast(mapped.pData), frame_width, + frame_height, mapped.RowPitch, target_width, + target_height, &png); + context->Unmap(staging.Get(), 0); + + if (!encoded) { + PostError(request, "encode_failed", "PNG encode failed"); + return; + } + auto result = request.result; + request.task_runner->Post([result, png = std::move(png)]() { + result->Success(EncodableValue(png)); + }); +} + +// Resolves the display id Dart sent. Windows has no CGDirectDisplayID +// equivalent, so the id is the HMONITOR itself; 0 means "the primary display", +// which is what a consumer with no display list can ask for. +HMONITOR ResolveMonitor(int64_t display_id) { + if (display_id == 0) { + const POINT origin{0, 0}; + return MonitorFromPoint(origin, MONITOR_DEFAULTTOPRIMARY); + } + MonitorSearch search{reinterpret_cast( + static_cast(display_id)), + false}; + EnumDisplayMonitors(nullptr, nullptr, FindMonitorProc, + reinterpret_cast(&search)); + return search.found ? search.wanted : nullptr; +} + +} // namespace + +// ── CaptureStream ─────────────────────────────────────────────────────────── + +CaptureStream::CaptureStream(std::unique_ptr session, + std::unique_ptr sink) + : session_(std::move(session)), sink_(std::move(sink)) {} + +CaptureStream::~CaptureStream() { + // Shutdown() is the normal path and has already run by the time the last + // reference drops. This covers the one case where it has not — Begin() + // failing before the stream was ever published — and cannot call Shutdown() + // itself, because shared_from_this() is unavailable during destruction. + std::lock_guard lock(mutex_); + if (shut_down_) return; + shut_down_ = true; + session_->Stop(); + sink_->Unregister([]() {}); +} + +bool CaptureStream::Begin(std::string *error) { + // Weak, not shared: the session lives inside this object, and a shared_ptr + // in its own callback would keep the stream alive forever. + std::weak_ptr weak = shared_from_this(); + return session_->Start( + [weak](ID3D11Texture2D *texture, UINT width, UINT height) { + if (auto self = weak.lock()) self->sink_->OnFrame(texture, width, height); + }, + [weak](const std::string &reason) { + if (auto self = weak.lock()) self->MarkStopped(reason); + }, + error); +} + +int64_t CaptureStream::texture_id() const { return sink_->texture_id(); } + +bool CaptureStream::stopped() const { + std::lock_guard lock(mutex_); + return stopped_; +} + +std::string CaptureStream::stop_reason() const { + std::lock_guard lock(mutex_); + return stop_reason_; +} + +uint64_t CaptureStream::frames_published() const { + return sink_->frames_published(); +} + +void CaptureStream::SourceSize(UINT *width, UINT *height) const { + session_->ItemSize(width, height); +} + +void CaptureStream::PublishedSize(UINT *width, UINT *height) const { + sink_->PublishedSize(width, height); +} + +void CaptureStream::MarkStopped(const std::string &reason) { + std::lock_guard lock(mutex_); + if (stopped_) return; + stopped_ = true; + stop_reason_ = reason; +} + +void CaptureStream::Shutdown() { + { + std::lock_guard lock(mutex_); + if (shut_down_) return; + shut_down_ = true; + if (!stopped_) { + stopped_ = true; + stop_reason_ = "stopped"; + } + } + // Stop first: this waits out an in-flight frame callback, so nothing can + // write into the sink after its textures are released. + session_->Stop(); + // The engine may still hold a descriptor; keep this object alive until it + // says otherwise. + sink_->Unregister([keepalive = shared_from_this()]() {}); +} + +// ── CaptureController ─────────────────────────────────────────────────────── + +CaptureController::CaptureController(flutter::TextureRegistrar *registrar, + PlatformTaskRunner *task_runner) + : registrar_(registrar), + task_runner_(task_runner), + d3d_(std::make_shared()) {} + +CaptureController::~CaptureController() { + { + std::lock_guard lock(streams_mutex_); + for (auto &entry : streams_) entry.second->Shutdown(); + streams_.clear(); + } + // Workers borrow the D3D device and the task runner; both outlive them only + // if we wait here. Each is bounded by kOneShotTimeout. + std::unique_lock lock(oneshot_mutex_); + oneshot_done_.wait(lock, [this] { return oneshots_in_flight_ == 0; }); +} + +std::shared_ptr CaptureController::FindStream( + int64_t texture_id) const { + std::lock_guard lock(streams_mutex_); + auto it = streams_.find(texture_id); + return it == streams_.end() ? nullptr : it->second; +} + +bool CaptureController::Handles(const std::string &method) { + return method == "startStream" || method == "stopStream" || + method == "reconfigureStream" || method == "streamStatus" || + method == "captureSingleFrame" || + method == "captureSingleDisplayFrame"; +} + +void CaptureController::HandleMethodCall(const std::string &method, + const EncodableMap &args, + MethodResultPtr result) { + if (method == "startStream") { + auto window_id = GetInt64(args, "windowId"); + if (!window_id) { + result->Error("bad_args", "startStream needs {windowId: Int}"); + return; + } + StartStream(reinterpret_cast(static_cast(*window_id)), + std::move(result)); + return; + } + if (method == "stopStream") { + auto texture_id = GetInt64(args, "textureId"); + if (!texture_id) { + result->Error("bad_args", "stopStream needs {textureId: Int}"); + return; + } + StopStream(*texture_id, std::move(result)); + return; + } + if (method == "reconfigureStream") { + auto texture_id = GetInt64(args, "textureId"); + if (!texture_id) { + result->Error("bad_args", "reconfigureStream needs {textureId: Int}"); + return; + } + ReconfigureStream(*texture_id, std::move(result)); + return; + } + if (method == "streamStatus") { + auto texture_id = GetInt64(args, "textureId"); + if (!texture_id) { + result->Error("bad_args", "streamStatus needs {textureId: Int}"); + return; + } + StreamStatus(*texture_id, std::move(result)); + return; + } + if (method == "captureSingleFrame") { + auto window_id = GetInt64(args, "windowId"); + if (!window_id) { + result->Error("bad_args", "captureSingleFrame needs {windowId: Int}"); + return; + } + CaptureOneShot(reinterpret_cast(static_cast(*window_id)), + nullptr, 0, std::move(result)); + return; + } + if (method == "captureSingleDisplayFrame") { + auto display_id = GetInt64(args, "displayId"); + if (!display_id) { + result->Error("bad_args", + "captureSingleDisplayFrame needs {displayId: Int}"); + return; + } + HMONITOR monitor = ResolveMonitor(*display_id); + if (!monitor) { + result->Error("display_not_found", + "no display with id " + std::to_string(*display_id)); + return; + } + CaptureOneShot(nullptr, monitor, kDisplayThumbnailWidth, std::move(result)); + return; + } +} + +void CaptureController::StartStream(HWND hwnd, MethodResultPtr result) { + if (!WgcSession::IsSupported()) { + result->Error("capture_unsupported", + "Windows.Graphics.Capture is unavailable on this system " + "(requires Windows 10 1803 or newer)."); + return; + } + if (!registrar_) { + result->Error("no_texture_registry", + "Flutter texture registrar not available"); + return; + } + std::string error; + if (!d3d_->Ensure(&error)) { + result->Error("stream_failed", error); + return; + } + + auto session = WgcSession::CreateForWindow(hwnd, d3d_, &error); + if (!session) { + result->Error("window_not_found", error); + return; + } + + auto sink = std::make_unique(registrar_, d3d_); + const int64_t texture_id = sink->Register(); + if (texture_id == 0) { + result->Error("stream_failed", "could not register a Flutter texture"); + return; + } + + auto stream = + std::make_shared(std::move(session), std::move(sink)); + if (!stream->Begin(&error)) { + // Begin() failed after the texture was registered; unwind it here or the + // texture leaks for the life of the engine. + stream->Shutdown(); + result->Error("stream_failed", error); + return; + } + + { + std::lock_guard lock(streams_mutex_); + streams_[texture_id] = stream; + } + result->Success(EncodableValue(texture_id)); +} + +void CaptureController::StopStream(int64_t texture_id, MethodResultPtr result) { + std::shared_ptr stream; + { + std::lock_guard lock(streams_mutex_); + auto it = streams_.find(texture_id); + if (it != streams_.end()) { + stream = it->second; + streams_.erase(it); + } + } + // Unknown ids are a no-op, matching the macOS backend's idempotent stop. + if (stream) stream->Shutdown(); + result->Success(EncodableValue()); +} + +void CaptureController::ReconfigureStream(int64_t texture_id, + MethodResultPtr result) { + auto stream = FindStream(texture_id); + if (!stream) { + result->Error("unknown_texture", + "no active stream for textureId " + std::to_string(texture_id)); + return; + } + + // The capture pool re-sizes itself when the source reflows (see + // WgcSession::OnFrameArrived), so there is nothing to apply here — this + // reports whether the source has outgrown the frames published so far, which + // is the signal the consumer's poll acts on. + UINT source_width = 0; + UINT source_height = 0; + stream->SourceSize(&source_width, &source_height); + UINT published_width = 0; + UINT published_height = 0; + stream->PublishedSize(&published_width, &published_height); + + if (source_width == 0 || source_height == 0) { + result->Success(EncodableValue(EncodableMap{ + {EncodableValue("changed"), EncodableValue(false)}, + {EncodableValue("reason"), EncodableValue("window_gone")}, + {EncodableValue("width"), EncodableValue(static_cast(published_width))}, + {EncodableValue("height"), EncodableValue(static_cast(published_height))}, + })); + return; + } + + const bool changed = + source_width != published_width || source_height != published_height; + result->Success(EncodableValue(EncodableMap{ + {EncodableValue("changed"), EncodableValue(changed)}, + {EncodableValue("width"), EncodableValue(static_cast(source_width))}, + {EncodableValue("height"), EncodableValue(static_cast(source_height))}, + })); +} + +void CaptureController::StreamStatus(int64_t texture_id, + MethodResultPtr result) { + auto stream = FindStream(texture_id); + if (!stream) { + // Unknown texture → already gone → report stopped so the consumer's poll + // tears down, exactly as the macOS backend does. + result->Success(EncodableValue(EncodableMap{ + {EncodableValue("stopped"), EncodableValue(true)}, + {EncodableValue("reason"), EncodableValue("unknown_texture")}, + })); + return; + } + // A lost D3D device kills every stream at once and no per-session callback + // fires, so fold it in here. + const bool device_lost = d3d_->IsDeviceLost(); + const bool stopped = stream->stopped() || device_lost; + std::string reason = stream->stop_reason(); + if (device_lost && reason.empty()) reason = "device_lost"; + + EncodableMap status{ + {EncodableValue("stopped"), EncodableValue(stopped)}, + {EncodableValue("frames"), + EncodableValue(static_cast(stream->frames_published()))}, + }; + if (!reason.empty()) { + status[EncodableValue("reason")] = EncodableValue(reason); + } + result->Success(EncodableValue(status)); +} + +void CaptureController::CaptureOneShot(HWND hwnd, HMONITOR monitor, + UINT cap_width, MethodResultPtr result) { + if (!WgcSession::IsSupported()) { + result->Error("capture_unsupported", + "Windows.Graphics.Capture is unavailable on this system " + "(requires Windows 10 1803 or newer)."); + return; + } + std::string error; + if (!d3d_->Ensure(&error)) { + result->Error("capture_failed", error); + return; + } + + { + std::lock_guard lock(oneshot_mutex_); + ++oneshots_in_flight_; + } + + OneShotRequest request{}; + request.d3d = d3d_; + request.task_runner = task_runner_; + request.result.reset(result.release()); + request.hwnd = hwnd; + request.monitor = monitor; + request.cap_width = cap_width; + + std::thread([this, request]() { + // WIC and the WinRT activation factories both need COM on this thread. + const HRESULT com = CoInitializeEx(nullptr, COINIT_MULTITHREADED); + RunOneShot(request); + if (SUCCEEDED(com)) CoUninitialize(); + + std::lock_guard lock(oneshot_mutex_); + --oneshots_in_flight_; + oneshot_done_.notify_all(); + }).detach(); +} + +} // namespace flutter_window_portal_windows diff --git a/packages/flutter_window_portal_windows/windows/capture/capture_controller.h b/packages/flutter_window_portal_windows/windows/capture/capture_controller.h new file mode 100644 index 0000000..0be0bbd --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/capture/capture_controller.h @@ -0,0 +1,117 @@ +#ifndef FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_CAPTURE_CONTROLLER_H_ +#define FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_CAPTURE_CONTROLLER_H_ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +namespace flutter_window_portal_windows { + +class D3DContext; +class PlatformTaskRunner; +class TextureSink; +class WgcSession; + +// One live capture: a Windows.Graphics.Capture session feeding a Flutter +// texture. Held by shared_ptr because texture teardown is asynchronous — the +// engine may still be reading the surface after the consumer calls stopStream. +class CaptureStream : public std::enable_shared_from_this { + public: + CaptureStream(std::unique_ptr session, + std::unique_ptr sink); + ~CaptureStream(); + + CaptureStream(const CaptureStream &) = delete; + CaptureStream &operator=(const CaptureStream &) = delete; + + // Wires frames through to the texture. Split from the constructor because + // the frame callback captures a shared_ptr to this. + bool Begin(std::string *error); + + int64_t texture_id() const; + bool stopped() const; + std::string stop_reason() const; + uint64_t frames_published() const; + + // Current source size and last published frame size, both in physical + // pixels — what `reconfigureStream` compares. + void SourceSize(UINT *width, UINT *height) const; + void PublishedSize(UINT *width, UINT *height) const; + + // Stops capture and releases the Flutter texture. Safe to call twice. + void Shutdown(); + + private: + void MarkStopped(const std::string &reason); + + std::unique_ptr session_; + std::unique_ptr sink_; + + mutable std::mutex mutex_; + bool stopped_ = false; + bool shut_down_ = false; + std::string stop_reason_; +}; + +// Serves the `flutter_window_portal/capture` methods that need pixels: +// startStream / stopStream / reconfigureStream / streamStatus and the two +// one-shot PNG grabs. Window enumeration stays in the plugin, which owns the +// Win32 helpers. +class CaptureController { + public: + CaptureController(flutter::TextureRegistrar *registrar, + PlatformTaskRunner *task_runner); + ~CaptureController(); + + CaptureController(const CaptureController &) = delete; + CaptureController &operator=(const CaptureController &) = delete; + + // Whether |method| is one this class serves. Checked before dispatch so the + // caller keeps ownership of the result for methods it still handles itself. + static bool Handles(const std::string &method); + + void HandleMethodCall( + const std::string &method, const flutter::EncodableMap &args, + std::unique_ptr> result); + + private: + using MethodResultPtr = + std::unique_ptr>; + + void StartStream(HWND hwnd, MethodResultPtr result); + void StopStream(int64_t texture_id, MethodResultPtr result); + void ReconfigureStream(int64_t texture_id, MethodResultPtr result); + void StreamStatus(int64_t texture_id, MethodResultPtr result); + + // |monitor| is null for a window grab. |cap_width| bounds the encoded image + // (0 = source size), matching the macOS display-thumbnail cap. + void CaptureOneShot(HWND hwnd, HMONITOR monitor, UINT cap_width, + MethodResultPtr result); + + std::shared_ptr FindStream(int64_t texture_id) const; + + flutter::TextureRegistrar *registrar_; + PlatformTaskRunner *task_runner_; + std::shared_ptr d3d_; + + mutable std::mutex streams_mutex_; + std::map> streams_; + + // One-shot grabs run on their own thread; the destructor waits them out so + // a worker can never outlive the D3D device it borrows. + std::mutex oneshot_mutex_; + std::condition_variable oneshot_done_; + int oneshots_in_flight_ = 0; +}; + +} // namespace flutter_window_portal_windows + +#endif // FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_CAPTURE_CONTROLLER_H_ diff --git a/packages/flutter_window_portal_windows/windows/capture/d3d_context.cpp b/packages/flutter_window_portal_windows/windows/capture/d3d_context.cpp new file mode 100644 index 0000000..9f3bc23 --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/capture/d3d_context.cpp @@ -0,0 +1,148 @@ +#include "d3d_context.h" + +#include +#include + +namespace flutter_window_portal_windows { + +using Microsoft::WRL::ComPtr; + +bool D3DContext::Ensure(std::string *error) { + std::lock_guard lock(mutex_); + if (device_) { + if (device_->GetDeviceRemovedReason() == S_OK) return true; + // Dropped by the driver. Everything built on it is dead; rebuild so the + // next startStream succeeds instead of failing forever. + device_.Reset(); + context_.Reset(); + winrt_device_ = nullptr; + } + + // BGRA support is required both by Windows.Graphics.Capture's frame pool and + // by the BGRA shared textures the Flutter engine opens. + UINT flags = D3D11_CREATE_DEVICE_BGRA_SUPPORT; + const D3D_FEATURE_LEVEL levels[] = { + D3D_FEATURE_LEVEL_11_1, D3D_FEATURE_LEVEL_11_0, D3D_FEATURE_LEVEL_10_1, + D3D_FEATURE_LEVEL_10_0, + }; + + ComPtr device; + ComPtr context; + HRESULT hr = D3D11CreateDevice( + nullptr, D3D_DRIVER_TYPE_HARDWARE, nullptr, flags, levels, + ARRAYSIZE(levels), D3D11_SDK_VERSION, &device, nullptr, &context); + if (hr == DXGI_ERROR_UNSUPPORTED || FAILED(hr)) { + // Software rasterizer fallback keeps capture working in VMs and over + // sessions without a hardware adapter. + hr = D3D11CreateDevice(nullptr, D3D_DRIVER_TYPE_WARP, nullptr, flags, + levels, ARRAYSIZE(levels), D3D11_SDK_VERSION, + &device, nullptr, &context); + } + if (FAILED(hr)) { + if (error) *error = "D3D11CreateDevice failed (hr=" + std::to_string(hr) + ")"; + return false; + } + + // Frames arrive on the capture pool's threadpool threads while the raster + // thread reads; without this the shared immediate context would race. + ComPtr multithread; + if (SUCCEEDED(context.As(&multithread))) { + multithread->SetMultithreadProtected(TRUE); + } + + ComPtr dxgi_device; + hr = device.As(&dxgi_device); + if (FAILED(hr)) { + if (error) *error = "ID3D11Device does not expose IDXGIDevice"; + return false; + } + + // Windows.Graphics.Capture takes the WinRT projection of the device, not the + // raw D3D one. + winrt::com_ptr<::IInspectable> inspectable; + hr = CreateDirect3D11DeviceFromDXGIDevice(dxgi_device.Get(), + inspectable.put()); + if (FAILED(hr)) { + if (error) { + *error = "CreateDirect3D11DeviceFromDXGIDevice failed (hr=" + + std::to_string(hr) + ")"; + } + return false; + } + + device_ = device; + context_ = context; + winrt_device_ = inspectable.as< + winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice>(); + return true; +} + +bool D3DContext::IsDeviceLost() const { + std::lock_guard lock(mutex_); + return !device_ || device_->GetDeviceRemovedReason() != S_OK; +} + +ComPtr D3DContext::CreateSharedTexture(UINT width, + UINT height, + HANDLE *out_handle) { + ComPtr device; + { + std::lock_guard lock(mutex_); + device = device_; + } + if (!device || width == 0 || height == 0) return nullptr; + + D3D11_TEXTURE2D_DESC desc{}; + desc.Width = width; + desc.Height = height; + desc.MipLevels = 1; + desc.ArraySize = 1; + desc.Format = DXGI_FORMAT_B8G8R8A8_UNORM; + desc.SampleDesc.Count = 1; + desc.Usage = D3D11_USAGE_DEFAULT; + desc.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET; + // Legacy (non-NT) shared handle: ANGLE opens these via + // EGL_D3D_TEXTURE_2D_SHARE_HANDLE_ANGLE, and it rejects NT handles. + desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED; + + ComPtr texture; + if (FAILED(device->CreateTexture2D(&desc, nullptr, &texture))) { + return nullptr; + } + + ComPtr resource; + if (FAILED(texture.As(&resource))) return nullptr; + HANDLE handle = nullptr; + if (FAILED(resource->GetSharedHandle(&handle)) || !handle) return nullptr; + + if (out_handle) *out_handle = handle; + return texture; +} + +ComPtr D3DContext::CreateStagingTexture(UINT width, + UINT height) { + ComPtr device; + { + std::lock_guard lock(mutex_); + device = device_; + } + if (!device || width == 0 || height == 0) return nullptr; + + D3D11_TEXTURE2D_DESC desc{}; + desc.Width = width; + desc.Height = height; + desc.MipLevels = 1; + desc.ArraySize = 1; + desc.Format = DXGI_FORMAT_B8G8R8A8_UNORM; + desc.SampleDesc.Count = 1; + desc.Usage = D3D11_USAGE_STAGING; + desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ; + + ComPtr texture; + if (FAILED(device->CreateTexture2D(&desc, nullptr, &texture))) { + return nullptr; + } + return texture; +} + +} // namespace flutter_window_portal_windows diff --git a/packages/flutter_window_portal_windows/windows/capture/d3d_context.h b/packages/flutter_window_portal_windows/windows/capture/d3d_context.h new file mode 100644 index 0000000..b0c892d --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/capture/d3d_context.h @@ -0,0 +1,72 @@ +#ifndef FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_D3D_CONTEXT_H_ +#define FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_D3D_CONTEXT_H_ + +#include +#include +#include +#include + +#include +#include + +namespace flutter_window_portal_windows { + +// The one Direct3D 11 device every capture session and texture sink in this +// plugin shares. +// +// Windows.Graphics.Capture hands frames back as textures owned by the device +// the frame pool was created on, so sessions and the sinks that copy out of +// them must agree on a device. The immediate context is made thread-safe via +// ID3D10Multithread because frames arrive concurrently on the capture pool's +// free-threaded callbacks while the Flutter raster thread reads descriptors. +class D3DContext { + public: + D3DContext() = default; + ~D3DContext() = default; + + D3DContext(const D3DContext &) = delete; + D3DContext &operator=(const D3DContext &) = delete; + + // Creates the device, or recreates it when the previous one was removed + // (TDR, driver update, GPU hot-swap). Returns false with |error| set when + // D3D11 is unavailable — the caller surfaces that to Dart rather than + // pretending capture is merely empty. + bool Ensure(std::string *error); + + Microsoft::WRL::ComPtr device() const { return device_; } + Microsoft::WRL::ComPtr context() const { + return context_; + } + winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice winrt_device() + const { + return winrt_device_; + } + + // True once the driver has dropped the device; every live stream is dead at + // that point and must be torn down by the consumer's liveness poll. + bool IsDeviceLost() const; + + // A BGRA texture the Flutter engine's own D3D device can open by shared + // handle (kFlutterDesktopGpuSurfaceTypeDxgiSharedHandle). Uses the legacy + // D3D11_RESOURCE_MISC_SHARED handle rather than an NT handle because that is + // what ANGLE's EGL_D3D_TEXTURE_2D_SHARE_HANDLE_ANGLE accepts. + // + // |out_handle| is owned by the returned texture and must not be closed. + Microsoft::WRL::ComPtr CreateSharedTexture( + UINT width, UINT height, HANDLE *out_handle); + + // A CPU-readable copy target, for the one-shot PNG paths. + Microsoft::WRL::ComPtr CreateStagingTexture(UINT width, + UINT height); + + private: + mutable std::mutex mutex_; + Microsoft::WRL::ComPtr device_; + Microsoft::WRL::ComPtr context_; + winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice winrt_device_{ + nullptr}; +}; + +} // namespace flutter_window_portal_windows + +#endif // FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_D3D_CONTEXT_H_ diff --git a/packages/flutter_window_portal_windows/windows/capture/png_encoder.cpp b/packages/flutter_window_portal_windows/windows/capture/png_encoder.cpp new file mode 100644 index 0000000..e72c1e7 --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/capture/png_encoder.cpp @@ -0,0 +1,89 @@ +#include "png_encoder.h" + +#include +#include +#include + +namespace flutter_window_portal_windows { + +namespace { + +using Microsoft::WRL::ComPtr; + +// Reads the whole of an IStream the encoder wrote into, from the start. +bool ReadStream(IStream *stream, std::vector *out) { + STATSTG stat{}; + if (FAILED(stream->Stat(&stat, STATFLAG_NONAME))) return false; + const ULONG size = static_cast(stat.cbSize.QuadPart); + if (size == 0) return false; + + LARGE_INTEGER zero{}; + if (FAILED(stream->Seek(zero, STREAM_SEEK_SET, nullptr))) return false; + + out->resize(size); + ULONG read = 0; + if (FAILED(stream->Read(out->data(), size, &read)) || read != size) { + out->clear(); + return false; + } + return true; +} + +} // namespace + +bool EncodeBgraToPng(const uint8_t *data, UINT width, UINT height, UINT stride, + UINT target_width, UINT target_height, + std::vector *out) { + if (!data || !out || width == 0 || height == 0) return false; + + ComPtr factory; + if (FAILED(CoCreateInstance(CLSID_WICImagingFactory, nullptr, + CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&factory)))) { + return false; + } + + // GUID_WICPixelFormat32bppBGRA matches DXGI_FORMAT_B8G8R8A8_UNORM, so the + // staging-texture bytes go in without a channel shuffle. + ComPtr bitmap; + if (FAILED(factory->CreateBitmapFromMemory( + width, height, GUID_WICPixelFormat32bppBGRA, stride, stride * height, + const_cast(data), &bitmap))) { + return false; + } + + ComPtr source = bitmap; + if (target_width != 0 && target_height != 0 && + (target_width != width || target_height != height)) { + ComPtr scaler; + if (FAILED(factory->CreateBitmapScaler(&scaler))) return false; + if (FAILED(scaler->Initialize(bitmap.Get(), target_width, target_height, + WICBitmapInterpolationModeFant))) { + return false; + } + source = scaler; + } + + ComPtr stream; + stream.Attach(SHCreateMemStream(nullptr, 0)); + if (!stream) return false; + + ComPtr encoder; + if (FAILED(factory->CreateEncoder(GUID_ContainerFormatPng, nullptr, + &encoder))) { + return false; + } + if (FAILED(encoder->Initialize(stream.Get(), WICBitmapEncoderNoCache))) { + return false; + } + + ComPtr frame; + if (FAILED(encoder->CreateNewFrame(&frame, nullptr))) return false; + if (FAILED(frame->Initialize(nullptr))) return false; + if (FAILED(frame->WriteSource(source.Get(), nullptr))) return false; + if (FAILED(frame->Commit())) return false; + if (FAILED(encoder->Commit())) return false; + + return ReadStream(stream.Get(), out); +} + +} // namespace flutter_window_portal_windows diff --git a/packages/flutter_window_portal_windows/windows/capture/png_encoder.h b/packages/flutter_window_portal_windows/windows/capture/png_encoder.h new file mode 100644 index 0000000..959303e --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/capture/png_encoder.h @@ -0,0 +1,24 @@ +#ifndef FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_PNG_ENCODER_H_ +#define FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_PNG_ENCODER_H_ + +#include + +#include +#include + +namespace flutter_window_portal_windows { + +// Encodes a top-down BGRA buffer as PNG via WIC, optionally rescaling to +// |target_width| x |target_height| first (the display-thumbnail path caps its +// output the way the macOS backend caps SCStreamConfiguration). +// +// Pass 0 for either target dimension to encode at source size. Returns false +// when WIC rejects the buffer or the encode fails; |out| is only written on +// success. COM must already be initialised on the calling thread. +bool EncodeBgraToPng(const uint8_t *data, UINT width, UINT height, UINT stride, + UINT target_width, UINT target_height, + std::vector *out); + +} // namespace flutter_window_portal_windows + +#endif // FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_PNG_ENCODER_H_ diff --git a/packages/flutter_window_portal_windows/windows/capture/texture_sink.cpp b/packages/flutter_window_portal_windows/windows/capture/texture_sink.cpp new file mode 100644 index 0000000..52860be --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/capture/texture_sink.cpp @@ -0,0 +1,123 @@ +#include "texture_sink.h" + +#include +#include + +#include "d3d_context.h" + +namespace flutter_window_portal_windows { + +TextureSink::TextureSink(flutter::TextureRegistrar *registrar, + std::shared_ptr d3d) + : registrar_(registrar), d3d_(std::move(d3d)) {} + +TextureSink::~TextureSink() = default; + +int64_t TextureSink::Register() { + if (!registrar_) return 0; + texture_ = std::make_unique(flutter::GpuSurfaceTexture( + kFlutterDesktopGpuSurfaceTypeDxgiSharedHandle, + [this](size_t, size_t) { return ObtainDescriptor(); })); + texture_id_ = registrar_->RegisterTexture(texture_.get()); + return texture_id_; +} + +const FlutterDesktopGpuSurfaceDescriptor *TextureSink::ObtainDescriptor() { + std::lock_guard lock(mutex_); + if (published_ < 0) return nullptr; + const Buffer &buffer = buffers_[published_]; + if (!buffer.shared_handle) return nullptr; + + descriptor_.struct_size = sizeof(FlutterDesktopGpuSurfaceDescriptor); + descriptor_.handle = buffer.shared_handle; + descriptor_.width = buffer.width; + descriptor_.height = buffer.height; + descriptor_.visible_width = buffer.width; + descriptor_.visible_height = buffer.height; + descriptor_.format = kFlutterDesktopPixelFormatBGRA8888; + // The sink owns both textures for its whole lifetime, so there is nothing + // for the engine to release back. + descriptor_.release_callback = nullptr; + descriptor_.release_context = nullptr; + return &descriptor_; +} + +void TextureSink::OnFrame(ID3D11Texture2D *texture, UINT width, UINT height) { + if (!texture || width == 0 || height == 0) return; + auto context = d3d_ ? d3d_->context() : nullptr; + if (!context) return; + + { + std::lock_guard lock(mutex_); + if (unregistered_) return; + + // Never write the surface the engine was last pointed at. + const int write = (published_ == 0) ? 1 : 0; + Buffer &buffer = buffers_[write]; + if (!buffer.texture || buffer.width != width || buffer.height != height) { + HANDLE handle = nullptr; + auto created = d3d_->CreateSharedTexture(width, height, &handle); + if (!created) return; + buffer.texture = created; + buffer.shared_handle = handle; + buffer.width = width; + buffer.height = height; + } + + // The frame-pool surface can be larger than the content; copy only the + // region the OS says is live. + D3D11_TEXTURE2D_DESC source_desc{}; + texture->GetDesc(&source_desc); + D3D11_BOX box{}; + box.left = 0; + box.top = 0; + box.front = 0; + box.right = std::min(width, source_desc.Width); + box.bottom = std::min(height, source_desc.Height); + box.back = 1; + if (box.right == 0 || box.bottom == 0) return; + + context->CopySubresourceRegion(buffer.texture.Get(), 0, 0, 0, 0, texture, 0, + &box); + // The engine reads this texture from a different D3D device; without a + // flush the copy can still be queued on ours when it does. + context->Flush(); + + published_ = write; + ++frames_published_; + } + + // Outside the lock: the engine may synchronously call back for a descriptor. + if (registrar_) registrar_->MarkTextureFrameAvailable(texture_id_); +} + +void TextureSink::PublishedSize(UINT *width, UINT *height) const { + std::lock_guard lock(mutex_); + const UINT w = published_ >= 0 ? buffers_[published_].width : 0; + const UINT h = published_ >= 0 ? buffers_[published_].height : 0; + if (width) *width = w; + if (height) *height = h; +} + +uint64_t TextureSink::frames_published() const { + std::lock_guard lock(mutex_); + return frames_published_; +} + +void TextureSink::Unregister(std::function on_released) { + { + std::lock_guard lock(mutex_); + if (unregistered_) return; + unregistered_ = true; + // Stop handing out descriptors immediately; the engine may still have one + // in flight, which is why the caller waits for |on_released|. + published_ = -1; + } + if (!registrar_ || texture_id_ == 0) { + if (on_released) on_released(); + return; + } + registrar_->UnregisterTexture(texture_id_, std::move(on_released)); +} + +} // namespace flutter_window_portal_windows diff --git a/packages/flutter_window_portal_windows/windows/capture/texture_sink.h b/packages/flutter_window_portal_windows/windows/capture/texture_sink.h new file mode 100644 index 0000000..07530d2 --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/capture/texture_sink.h @@ -0,0 +1,87 @@ +#ifndef FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_TEXTURE_SINK_H_ +#define FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_TEXTURE_SINK_H_ + +#include +#include +#include +#include + +#include +#include +#include +#include + +namespace flutter_window_portal_windows { + +class D3DContext; + +// Publishes captured frames into a Flutter `Texture` widget. +// +// Frames are handed to the engine as DXGI shared handles, so the pixels never +// leave the GPU — the engine's own device opens the same texture. This is the +// Windows counterpart of the macOS sink's CVPixelBuffer hand-off. +// +// Two shared textures rotate: the capture thread always writes the buffer that +// was NOT last published, so the engine is never reading the surface being +// overwritten. Frames arrive off the platform thread and descriptors are +// requested on the raster thread, so all shared state is behind |mutex_|. +class TextureSink { + public: + TextureSink(flutter::TextureRegistrar *registrar, + std::shared_ptr d3d); + ~TextureSink(); + + TextureSink(const TextureSink &) = delete; + TextureSink &operator=(const TextureSink &) = delete; + + // Registers with the engine and returns the texture id Dart hands to + // `Texture(textureId:)`. Returns 0 on failure. + int64_t Register(); + + int64_t texture_id() const { return texture_id_; } + + // Copies a captured frame into the next shared texture and tells the engine + // a new frame is ready. Called on the capture pool's thread. + void OnFrame(ID3D11Texture2D *texture, UINT width, UINT height); + + // Size of the most recently published frame, in physical pixels. + void PublishedSize(UINT *width, UINT *height) const; + + // How many frames have reached the engine. Reported by `streamStatus` so a + // consumer (or a test) can tell "capturing but idle" from "never started". + uint64_t frames_published() const; + + // Unregisters the texture. |on_released| runs once the engine has finished + // with it, which is when the owner may safely drop this sink. + void Unregister(std::function on_released); + + private: + const FlutterDesktopGpuSurfaceDescriptor *ObtainDescriptor(); + + struct Buffer { + Microsoft::WRL::ComPtr texture; + HANDLE shared_handle = nullptr; + UINT width = 0; + UINT height = 0; + }; + + flutter::TextureRegistrar *registrar_; + std::shared_ptr d3d_; + std::unique_ptr texture_; + int64_t texture_id_ = 0; + + mutable std::mutex mutex_; + Buffer buffers_[2]; + // Index of the buffer the engine may currently read; -1 before the first + // frame. + int published_ = -1; + uint64_t frames_published_ = 0; + bool unregistered_ = false; + + // Handed back to the engine by pointer, so it lives as long as the sink. + FlutterDesktopGpuSurfaceDescriptor descriptor_{}; +}; + +} // namespace flutter_window_portal_windows + +#endif // FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_TEXTURE_SINK_H_ diff --git a/packages/flutter_window_portal_windows/windows/capture/wgc_session.cpp b/packages/flutter_window_portal_windows/windows/capture/wgc_session.cpp new file mode 100644 index 0000000..9dacd53 --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/capture/wgc_session.cpp @@ -0,0 +1,249 @@ +#include "wgc_session.h" + +#include +#include +#include +#include +#include + +#include + +#include "d3d_context.h" + +namespace flutter_window_portal_windows { + +namespace { + +namespace wgc = winrt::Windows::Graphics::Capture; +using winrt::Windows::Graphics::DirectX::DirectXPixelFormat; +using winrt::Windows::Graphics::SizeInt32; + +// Two buffers is the documented minimum that lets the OS keep producing while +// we copy the previous frame out. +constexpr int32_t kFramePoolBuffers = 2; + +} // namespace + +WgcSession::WgcSession(wgc::GraphicsCaptureItem item, + std::shared_ptr d3d) + : d3d_(std::move(d3d)), item_(std::move(item)) {} + +WgcSession::~WgcSession() { Stop(); } + +bool WgcSession::IsSupported() { + // Both the class and the per-machine policy can say no (capture is disabled + // in some managed environments). + return wgc::GraphicsCaptureSession::IsSupported(); +} + +std::unique_ptr WgcSession::CreateForWindow( + HWND hwnd, std::shared_ptr d3d, std::string *error) { + if (!IsWindow(hwnd)) { + if (error) *error = "no window with that id"; + return nullptr; + } + try { + auto interop = winrt::get_activation_factory(); + wgc::GraphicsCaptureItem item{nullptr}; + winrt::check_hresult(interop->CreateForWindow( + hwnd, winrt::guid_of(), + winrt::put_abi(item))); + return std::unique_ptr( + new WgcSession(std::move(item), std::move(d3d))); + } catch (const winrt::hresult_error &e) { + if (error) *error = winrt::to_string(e.message()); + return nullptr; + } +} + +std::unique_ptr WgcSession::CreateForMonitor( + HMONITOR monitor, std::shared_ptr d3d, std::string *error) { + if (!monitor) { + if (error) *error = "no display with that id"; + return nullptr; + } + try { + auto interop = winrt::get_activation_factory(); + wgc::GraphicsCaptureItem item{nullptr}; + winrt::check_hresult(interop->CreateForMonitor( + monitor, winrt::guid_of(), + winrt::put_abi(item))); + return std::unique_ptr( + new WgcSession(std::move(item), std::move(d3d))); + } catch (const winrt::hresult_error &e) { + if (error) *error = winrt::to_string(e.message()); + return nullptr; + } +} + +void WgcSession::ItemSize(UINT *width, UINT *height) const { + UINT w = 0; + UINT h = 0; + try { + if (item_) { + const SizeInt32 size = item_.Size(); + w = static_cast(size.Width > 0 ? size.Width : 0); + h = static_cast(size.Height > 0 ? size.Height : 0); + } + } catch (const winrt::hresult_error &) { + // The source went away between calls; report zero and let the caller's + // liveness path handle it. + } + if (width) *width = w; + if (height) *height = h; +} + +bool WgcSession::Start(FrameHandler on_frame, ClosedHandler on_closed, + std::string *error) { + if (!item_ || !d3d_) { + if (error) *error = "capture item unavailable"; + return false; + } + on_frame_ = std::move(on_frame); + on_closed_ = std::move(on_closed); + + try { + const SizeInt32 size = item_.Size(); + if (size.Width <= 0 || size.Height <= 0) { + if (error) *error = "source has no capturable area"; + return false; + } + pool_width_ = static_cast(size.Width); + pool_height_ = static_cast(size.Height); + + // Free-threaded: frames land on the thread pool, so the plugin does not + // need a DispatcherQueue on Flutter's platform thread. + pool_ = wgc::Direct3D11CaptureFramePool::CreateFreeThreaded( + d3d_->winrt_device(), DirectXPixelFormat::B8G8R8A8UIntNormalized, + kFramePoolBuffers, size); + session_ = pool_.CreateCaptureSession(item_); + + // Both of these are PROPERTIES on later revisions of the session + // interface, so query the interface rather than ApiInformation — an + // IsMethodPresent check silently reports false for a property and leaves + // the setting at its default. + // + // Match the macOS backend's `cfg.showsCursor = false`. Win10 2004+. + if (auto session2 = session_.try_as()) { + session2.IsCursorCaptureEnabled(false); + } + // Windows 11 draws a yellow "being captured" border by default; the tile + // is meant to look like the window itself, so drop it where allowed. Some + // policies refuse, which is not fatal. + if (auto session3 = session_.try_as()) { + try { + session3.IsBorderRequired(false); + } catch (const winrt::hresult_error &) { + } + } + + frame_token_ = pool_.FrameArrived( + [this](const wgc::Direct3D11CaptureFramePool &, const auto &) { + OnFrameArrived(); + }); + closed_token_ = item_.Closed( + [this](const wgc::GraphicsCaptureItem &, const auto &) { + ClosedHandler handler; + { + std::lock_guard lock(frame_mutex_); + handler = on_closed_; + } + if (handler && !stopped_.load()) handler("source_closed"); + }); + + session_.StartCapture(); + return true; + } catch (const winrt::hresult_error &e) { + if (error) *error = winrt::to_string(e.message()); + return false; + } +} + +void WgcSession::OnFrameArrived() { + if (stopped_.load()) return; + std::lock_guard lock(frame_mutex_); + if (stopped_.load()) return; + + try { + auto frame = pool_.TryGetNextFrame(); + if (!frame) return; + + const SizeInt32 content = frame.ContentSize(); + const UINT width = static_cast(content.Width > 0 ? content.Width : 0); + const UINT height = + static_cast(content.Height > 0 ? content.Height : 0); + + if (width != 0 && height != 0) { + // The frame surface belongs to the pool and is recycled the moment the + // frame is released, so the handler copies out of it synchronously. + auto access = frame.Surface() + .as<::Windows::Graphics::DirectX::Direct3D11:: + IDirect3DDxgiInterfaceAccess>(); + Microsoft::WRL::ComPtr texture; + if (SUCCEEDED(access->GetInterface(IID_PPV_ARGS(&texture))) && + on_frame_) { + on_frame_(texture.Get(), width, height); + } + } + + // Release before resizing the pool: Recreate invalidates outstanding + // frames. + frame.Close(); + + if ((width != pool_width_ || height != pool_height_) && width != 0 && + height != 0) { + // The source reflowed. Resizing the pool is what keeps a live stream + // sharp after the user drags the window's edge. + pool_.Recreate(d3d_->winrt_device(), + DirectXPixelFormat::B8G8R8A8UIntNormalized, + kFramePoolBuffers, {static_cast(width), + static_cast(height)}); + pool_width_ = width; + pool_height_ = height; + } + } catch (const winrt::hresult_error &e) { + // A dead source or a lost device surfaces here; report it once and stop + // so the consumer's poll tears the tile down instead of freezing. + ClosedHandler handler = on_closed_; + stopped_.store(true); + if (handler) handler(winrt::to_string(e.message())); + } +} + +void WgcSession::Stop() { + if (stopped_.exchange(true)) return; + + // Revoke first, with no lock held: revocation can block on an in-flight + // handler, and that handler wants |frame_mutex_|. + try { + if (pool_ && frame_token_) pool_.FrameArrived(frame_token_); + if (item_ && closed_token_) item_.Closed(closed_token_); + } catch (const winrt::hresult_error &) { + } + frame_token_ = {}; + closed_token_ = {}; + + // Now drain: taking the lock waits out a handler that had already started. + { + std::lock_guard lock(frame_mutex_); + on_frame_ = nullptr; + on_closed_ = nullptr; + } + + try { + if (session_) { + session_.Close(); + session_ = nullptr; + } + if (pool_) { + pool_.Close(); + pool_ = nullptr; + } + } catch (const winrt::hresult_error &) { + } + item_ = nullptr; +} + +} // namespace flutter_window_portal_windows diff --git a/packages/flutter_window_portal_windows/windows/capture/wgc_session.h b/packages/flutter_window_portal_windows/windows/capture/wgc_session.h new file mode 100644 index 0000000..f2f38bd --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/capture/wgc_session.h @@ -0,0 +1,93 @@ +#ifndef FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_WGC_SESSION_H_ +#define FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_WGC_SESSION_H_ + +#include +#include +#include + +#include +#include +#include +#include +#include + +namespace flutter_window_portal_windows { + +class D3DContext; + +// One live Windows.Graphics.Capture session over a window or a monitor. +// +// Frames arrive on the capture pool's free-threaded callbacks, so |FrameHandler| +// runs off the platform thread and must not touch Flutter directly. The texture +// it receives is only valid for the duration of the call — copy out of it before +// returning. +// +// The session tracks source resizes itself: when a frame's content size stops +// matching the pool, the pool is recreated at the new size and subsequent frames +// arrive at the new dimensions. +class WgcSession { + public: + using FrameHandler = + std::function; + // Fires when the OS ends the capture — the source window closed, or the + // session was revoked. Mirrors SCStream's didStopWithError on macOS. + using ClosedHandler = std::function; + + ~WgcSession(); + + WgcSession(const WgcSession &) = delete; + WgcSession &operator=(const WgcSession &) = delete; + + // True when the OS supports Windows.Graphics.Capture at all (Windows 10 + // 1803+). Callers surface a typed error rather than an empty capture. + static bool IsSupported(); + + // |error| is set and nullptr returned when the source cannot be captured — + // an unknown window, or a source the OS refuses to share. + static std::unique_ptr CreateForWindow( + HWND hwnd, std::shared_ptr d3d, std::string *error); + static std::unique_ptr CreateForMonitor( + HMONITOR monitor, std::shared_ptr d3d, std::string *error); + + // Begins delivering frames. Safe to call once; |error| is set on failure. + bool Start(FrameHandler on_frame, ClosedHandler on_closed, + std::string *error); + + // Tears the session down and blocks until any in-flight frame callback has + // finished, so the caller can then destroy whatever the handler wrote into. + // Idempotent. + void Stop(); + + // The source's current size in physical pixels, as the OS reports it. + void ItemSize(UINT *width, UINT *height) const; + + private: + WgcSession(winrt::Windows::Graphics::Capture::GraphicsCaptureItem item, + std::shared_ptr d3d); + + void OnFrameArrived(); + + std::shared_ptr d3d_; + winrt::Windows::Graphics::Capture::GraphicsCaptureItem item_{nullptr}; + winrt::Windows::Graphics::Capture::Direct3D11CaptureFramePool pool_{nullptr}; + winrt::Windows::Graphics::Capture::GraphicsCaptureSession session_{nullptr}; + winrt::event_token frame_token_{}; + winrt::event_token closed_token_{}; + + // Guards the whole frame-callback body, so Stop() can wait one out before + // dropping the resources the handler writes into. + std::mutex frame_mutex_; + std::atomic stopped_{false}; + + FrameHandler on_frame_; + ClosedHandler on_closed_; + + // Pool dimensions, re-read when the source reflows. Only touched under + // |frame_mutex_|. + UINT pool_width_ = 0; + UINT pool_height_ = 0; +}; + +} // namespace flutter_window_portal_windows + +#endif // FLUTTER_WINDOW_PORTAL_WINDOWS_CAPTURE_WGC_SESSION_H_ diff --git a/packages/flutter_window_portal_windows/windows/engagement_watcher.cpp b/packages/flutter_window_portal_windows/windows/engagement_watcher.cpp new file mode 100644 index 0000000..0bc9cf1 --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/engagement_watcher.cpp @@ -0,0 +1,205 @@ +#include "engagement_watcher.h" + +#include +#include +#include + +namespace flutter_window_portal_windows { + +namespace { + +// Every live watcher, so the C-callback can route an event without a +// per-hook lookup. Only ever touched on the platform thread: watchers are +// constructed there and WINEVENT_OUTOFCONTEXT events are delivered there. +std::vector &LiveWatchers() { + static std::vector watchers; + return watchers; +} + +// A window can settle a pixel or two off what we asked for (DPI rounding, or +// an app enforcing its own minimum), which is not the user dragging it away. +constexpr LONG kAlignmentTolerance = 2; + +bool FramesDiffer(const RECT &a, const RECT &b) { + return std::abs(a.left - b.left) > kAlignmentTolerance || + std::abs(a.top - b.top) > kAlignmentTolerance || + std::abs(a.right - b.right) > kAlignmentTolerance || + std::abs(a.bottom - b.bottom) > kAlignmentTolerance; +} + +} // namespace + +EngagementWatcher::EngagementWatcher( + AlignmentBrokenCallback on_alignment_broken, + DisengagedCallback on_disengaged) + : on_alignment_broken_(std::move(on_alignment_broken)), + on_disengaged_(std::move(on_disengaged)) { + LiveWatchers().push_back(this); +} + +EngagementWatcher::~EngagementWatcher() { + for (auto &entry : process_hooks_) { + if (entry.second.location) UnhookWinEvent(entry.second.location); + if (entry.second.destroy) UnhookWinEvent(entry.second.destroy); + } + process_hooks_.clear(); + watched_.clear(); + if (foreground_hook_) { + UnhookWinEvent(foreground_hook_); + foreground_hook_ = nullptr; + } + auto &live = LiveWatchers(); + live.erase(std::remove(live.begin(), live.end(), this), live.end()); +} + +void EngagementWatcher::Watch(HWND hwnd, const RECT &aligned_frame) { + if (!hwnd || !IsWindow(hwnd)) return; + + auto it = watched_.find(hwnd); + if (it != watched_.end()) { + // Re-engaging the same window: re-baseline rather than double-hook. + it->second.aligned_frame = aligned_frame; + it->second.reported_broken = false; + return; + } + + DWORD pid = 0; + GetWindowThreadProcessId(hwnd, &pid); + watched_[hwnd] = Watched{aligned_frame, pid, false}; + AddProcessHooks(pid); + UpdateForegroundHook(); +} + +void EngagementWatcher::Unwatch(HWND hwnd) { + auto it = watched_.find(hwnd); + if (it == watched_.end()) return; + const DWORD pid = it->second.pid; + watched_.erase(it); + ReleaseProcessHooks(pid); + UpdateForegroundHook(); +} + +bool EngagementWatcher::IsWatching(HWND hwnd) const { + return watched_.find(hwnd) != watched_.end(); +} + +void EngagementWatcher::SuppressGeometryEvents(DWORD duration_ms) { + suppress_until_ = GetTickCount() + duration_ms; +} + +void EngagementWatcher::AddProcessHooks(DWORD pid) { + auto it = process_hooks_.find(pid); + if (it != process_hooks_.end()) { + ++it->second.refs; + return; + } + ProcessHooks hooks{}; + hooks.refs = 1; + hooks.location = SetWinEventHook( + EVENT_OBJECT_LOCATIONCHANGE, EVENT_OBJECT_LOCATIONCHANGE, nullptr, + &EngagementWatcher::OnWinEvent, pid, 0, + WINEVENT_OUTOFCONTEXT | WINEVENT_SKIPOWNPROCESS); + hooks.destroy = SetWinEventHook( + EVENT_OBJECT_DESTROY, EVENT_OBJECT_DESTROY, nullptr, + &EngagementWatcher::OnWinEvent, pid, 0, + WINEVENT_OUTOFCONTEXT | WINEVENT_SKIPOWNPROCESS); + process_hooks_[pid] = hooks; +} + +void EngagementWatcher::ReleaseProcessHooks(DWORD pid) { + auto it = process_hooks_.find(pid); + if (it == process_hooks_.end()) return; + if (--it->second.refs > 0) return; + if (it->second.location) UnhookWinEvent(it->second.location); + if (it->second.destroy) UnhookWinEvent(it->second.destroy); + process_hooks_.erase(it); +} + +void EngagementWatcher::UpdateForegroundHook() { + // Focus can land on any process, so this one cannot be pid-scoped. It is + // only armed while something is engaged. + const bool want = !watched_.empty(); + if (want && !foreground_hook_) { + foreground_hook_ = SetWinEventHook( + EVENT_SYSTEM_FOREGROUND, EVENT_SYSTEM_FOREGROUND, nullptr, + &EngagementWatcher::OnWinEvent, 0, 0, WINEVENT_OUTOFCONTEXT); + } else if (!want && foreground_hook_) { + UnhookWinEvent(foreground_hook_); + foreground_hook_ = nullptr; + } +} + +void CALLBACK EngagementWatcher::OnWinEvent(HWINEVENTHOOK, DWORD event, + HWND hwnd, LONG id_object, + LONG id_child, DWORD, DWORD) { + // Only top-level window events matter; the same hooks also report every + // caret, cursor and child control in the process. + if (id_object != OBJID_WINDOW || id_child != CHILDID_SELF || !hwnd) return; + + // Copy first: a callback may Unwatch, which would invalidate the iteration. + const std::vector watchers = LiveWatchers(); + for (auto *watcher : watchers) { + switch (event) { + case EVENT_OBJECT_LOCATIONCHANGE: + watcher->HandleLocationChange(hwnd); + break; + case EVENT_OBJECT_DESTROY: + watcher->HandleDestroy(hwnd); + break; + case EVENT_SYSTEM_FOREGROUND: + watcher->HandleForeground(hwnd); + break; + default: + break; + } + } +} + +void EngagementWatcher::HandleLocationChange(HWND hwnd) { + auto it = watched_.find(hwnd); + if (it == watched_.end() || it->second.reported_broken) return; + // Our own engage/conform moves fire this too. + if (GetTickCount() < suppress_until_) return; + + RECT current{}; + if (!GetWindowRect(hwnd, ¤t)) return; + if (!FramesDiffer(current, it->second.aligned_frame)) return; + + it->second.reported_broken = true; + if (on_alignment_broken_) on_alignment_broken_(hwnd, current); +} + +void EngagementWatcher::HandleDestroy(HWND hwnd) { + if (watched_.find(hwnd) == watched_.end()) return; + Unwatch(hwnd); + if (on_disengaged_) on_disengaged_(hwnd, "windowClosed"); +} + +void EngagementWatcher::HandleForeground(HWND foreground) { + if (watched_.empty()) return; + + // The engaged source itself coming forward is the normal engaged state. + if (IsWatching(foreground)) return; + + // A dialog or second window of the SAME app counts as still being in the + // source — opening a Save sheet must not tear the engagement down. + DWORD foreground_pid = 0; + GetWindowThreadProcessId(foreground, &foreground_pid); + for (const auto &entry : watched_) { + if (entry.second.pid == foreground_pid) return; + } + + const bool host_came_forward = + host_ && GetAncestor(foreground, GA_ROOT) == host_; + const std::string reason = host_came_forward ? "hostActivated" : "thirdApp"; + + std::vector ending; + ending.reserve(watched_.size()); + for (const auto &entry : watched_) ending.push_back(entry.first); + for (HWND hwnd : ending) { + Unwatch(hwnd); + if (on_disengaged_) on_disengaged_(hwnd, reason); + } +} + +} // namespace flutter_window_portal_windows diff --git a/packages/flutter_window_portal_windows/windows/engagement_watcher.h b/packages/flutter_window_portal_windows/windows/engagement_watcher.h new file mode 100644 index 0000000..e05f2d0 --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/engagement_watcher.h @@ -0,0 +1,97 @@ +#ifndef FLUTTER_WINDOW_PORTAL_WINDOWS_ENGAGEMENT_WATCHER_H_ +#define FLUTTER_WINDOW_PORTAL_WINDOWS_ENGAGEMENT_WATCHER_H_ + +#include + +#include +#include +#include + +namespace flutter_window_portal_windows { + +// Watches engaged windows so engagement is maintained rather than assumed. +// +// This is the Win32 counterpart of the macOS backend's AXObserver plus +// NSWorkspace activation observer, and it pushes the same three things: +// +// * the source moved or resized behind our back → OnAlignmentBroken +// * the source was destroyed → OnDisengaged("windowClosed") +// * focus went elsewhere → OnDisengaged("thirdApp" | +// "hostActivated") +// +// Hooks are installed WINEVENT_OUTOFCONTEXT from the platform thread, so the +// callbacks are delivered on that thread through its message loop and may talk +// to Flutter directly. Location hooks are scoped to the source's process: +// EVENT_OBJECT_LOCATIONCHANGE is extremely chatty system-wide. +class EngagementWatcher { + public: + using AlignmentBrokenCallback = + std::function; + using DisengagedCallback = + std::function; + + EngagementWatcher(AlignmentBrokenCallback on_alignment_broken, + DisengagedCallback on_disengaged); + ~EngagementWatcher(); + + EngagementWatcher(const EngagementWatcher &) = delete; + EngagementWatcher &operator=(const EngagementWatcher &) = delete; + + // The Flutter host window, so a focus change back to our own app is reported + // as "hostActivated" rather than "thirdApp". + void SetHostWindow(HWND host) { host_ = host; } + + // Starts watching |hwnd|, treating |aligned_frame| as where engage left it. + // Re-watching an already-watched window just re-baselines the frame. + void Watch(HWND hwnd, const RECT &aligned_frame); + void Unwatch(HWND hwnd); + bool IsWatching(HWND hwnd) const; + + // Ignores geometry events for |duration_ms|. Engage and conform move the + // window themselves and would otherwise trip their own alignment check — + // the same masking the macOS backend does with selfMutationDeadline. + void SuppressGeometryEvents(DWORD duration_ms); + + private: + struct Watched { + RECT aligned_frame; + DWORD pid; + // Alignment breaks once per engagement: LOCATIONCHANGE fires continuously + // while a window is dragged, and the consumer's session ends on the first + // report anyway. + bool reported_broken; + }; + + // Per-process hooks, reference counted so two engaged windows of the same + // app share one pair. + struct ProcessHooks { + HWINEVENTHOOK location; + HWINEVENTHOOK destroy; + int refs; + }; + + static void CALLBACK OnWinEvent(HWINEVENTHOOK hook, DWORD event, HWND hwnd, + LONG id_object, LONG id_child, + DWORD event_thread, DWORD event_time); + + void HandleLocationChange(HWND hwnd); + void HandleDestroy(HWND hwnd); + void HandleForeground(HWND hwnd); + + void AddProcessHooks(DWORD pid); + void ReleaseProcessHooks(DWORD pid); + void UpdateForegroundHook(); + + AlignmentBrokenCallback on_alignment_broken_; + DisengagedCallback on_disengaged_; + + HWND host_ = nullptr; + std::map watched_; + std::map process_hooks_; + HWINEVENTHOOK foreground_hook_ = nullptr; + DWORD suppress_until_ = 0; +}; + +} // namespace flutter_window_portal_windows + +#endif // FLUTTER_WINDOW_PORTAL_WINDOWS_ENGAGEMENT_WATCHER_H_ diff --git a/packages/flutter_window_portal_windows/windows/flutter_window_portal_windows_plugin.cpp b/packages/flutter_window_portal_windows/windows/flutter_window_portal_windows_plugin.cpp index 37c46e3..bff20e1 100644 --- a/packages/flutter_window_portal_windows/windows/flutter_window_portal_windows_plugin.cpp +++ b/packages/flutter_window_portal_windows/windows/flutter_window_portal_windows_plugin.cpp @@ -7,6 +7,11 @@ #include #include +#include + +#include "capture/capture_controller.h" +#include "engagement_watcher.h" +#include "platform_task_runner.h" namespace flutter_window_portal_windows { @@ -128,6 +133,36 @@ bool IsShareableTopLevel(HWND hwnd) { return (r.right - r.left) > 0 && (r.bottom - r.top) > 0; } +// Collects top-level windows belonging to the source's process that overlap +// it — see the `detectSeparateWindows` handler. +struct SeparateWindowScan { + HWND source; + DWORD pid; + RECT source_rect; + EncodableList found; +}; + +BOOL CALLBACK SeparateWindowProc(HWND hwnd, LPARAM lparam) { + auto *scan = reinterpret_cast(lparam); + if (hwnd == scan->source) return TRUE; + if (!IsWindowVisible(hwnd) || IsIconic(hwnd)) return TRUE; + if (PidForWindow(hwnd) != scan->pid) return TRUE; + if (GetWindowLongW(hwnd, GWL_EXSTYLE) & WS_EX_TOOLWINDOW) return TRUE; + if (GetWindowTextLengthW(hwnd) == 0) return TRUE; + + RECT rect; + if (!GetWindowRect(hwnd, &rect)) return TRUE; + RECT intersection; + if (!IntersectRect(&intersection, &rect, &scan->source_rect)) return TRUE; + + scan->found.push_back(EncodableValue(EncodableMap{ + {EncodableValue("id"), + EncodableValue(static_cast(reinterpret_cast(hwnd)))}, + {EncodableValue("title"), EncodableValue(WindowTitle(hwnd))}, + })); + return TRUE; +} + BOOL CALLBACK EnumProc(HWND hwnd, LPARAM lparam) { auto *out = reinterpret_cast(lparam); if (!IsShareableTopLevel(hwnd)) return TRUE; @@ -180,6 +215,33 @@ FlutterWindowPortalWindowsPlugin::FlutterWindowPortalWindowsPlugin( [this](const auto &call, auto result) { HandleCaptureMethod(call, std::move(result)); }); + + task_runner_ = std::make_unique(); + capture_controller_ = std::make_unique( + registrar->texture_registrar(), task_runner_.get()); + + watcher_ = std::make_unique( + [this](HWND hwnd, const RECT &frame) { + // Report the frame the way every other geometry reply on this channel + // does: logical screen coordinates. + const double scale = ScaleForWindow(hwnd); + EmitEngagement( + "alignmentBroken", hwnd, + EncodableMap{ + {EncodableValue("x"), EncodableValue(frame.left / scale)}, + {EncodableValue("y"), EncodableValue(frame.top / scale)}, + {EncodableValue("width"), + EncodableValue((frame.right - frame.left) / scale)}, + {EncodableValue("height"), + EncodableValue((frame.bottom - frame.top) / scale)}, + }); + }, + [this](HWND hwnd, const std::string &reason) { + EmitEngagement("disengaged", hwnd, + EncodableMap{{EncodableValue("reason"), + EncodableValue(reason)}}); + }); + watcher_->SetHostWindow(HostWindow()); } FlutterWindowPortalWindowsPlugin::~FlutterWindowPortalWindowsPlugin() = default; @@ -241,7 +303,7 @@ void FlutterWindowPortalWindowsPlugin::HandleControlMethod( double scale = ScaleForWindow(hwnd); if (method == "isEngaged") { - result->Success(EncodableValue(engaged_.count(hwnd) != 0)); + result->Success(EncodableValue(watcher_->IsWatching(hwnd))); return; } if (method == "isWindowMinimized") { @@ -284,14 +346,26 @@ void FlutterWindowPortalWindowsPlugin::HandleControlMethod( return; } if (method == "detectSeparateWindows") { - // Staged with capture: report none for now. - result->Success(EncodableValue(EncodableMap{ - {EncodableValue("windows"), EncodableValue(EncodableList{})}})); + // Other top-level windows of the same app sitting over the source — a + // modal dialog, detached DevTools. A single-window capture does not + // include them, so the consumer needs to know they are there. + RECT source_rect; + if (!GetWindowRect(hwnd, &source_rect)) { + result->Success(EncodableValue(EncodableMap{ + {EncodableValue("windows"), EncodableValue(EncodableList{})}})); + return; + } + SeparateWindowScan scan{hwnd, PidForWindow(hwnd), source_rect, {}}; + EnumWindows(SeparateWindowProc, reinterpret_cast(&scan)); + result->Success(EncodableValue( + EncodableMap{{EncodableValue("windows"), EncodableValue(scan.found)}})); return; } if (method == "conformWindowToSize") { auto w = GetDouble(a, "width"); auto h = GetDouble(a, "height"); + // Resizing is our own mutation too — see the note in focusAndWarpCursor. + watcher_->SuppressGeometryEvents(1500); if (w && h) { SetWindowPos(hwnd, nullptr, 0, 0, static_cast(*w * scale), static_cast(*h * scale), @@ -308,7 +382,8 @@ void FlutterWindowPortalWindowsPlugin::HandleControlMethod( return; } if (method == "releaseEngagement") { - if (engaged_.erase(hwnd)) { + if (watcher_->IsWatching(hwnd)) { + watcher_->Unwatch(hwnd); EmitEngagement("disengaged", hwnd, EncodableMap{{EncodableValue("reason"), EncodableValue("released")}}); @@ -326,6 +401,11 @@ void FlutterWindowPortalWindowsPlugin::HandleControlMethod( // the window's own scale, then raises. The exact multi-monitor/DPI // conversion is a live-tuning follow-up (parallels the macOS settle-loop // work); a wrong conversion misplaces the window but does not crash. + // + // The move below fires the same LOCATIONCHANGE events the watcher treats + // as the user dragging the window away; mask them for the engage sequence, + // as the macOS backend does with selfMutationDeadline. + watcher_->SuppressGeometryEvents(1500); if (const auto *rectVal = Find(a, "tileRect")) { if (const auto *rect = std::get_if(rectVal)) { auto x = GetDouble(*rect, "x"); @@ -361,12 +441,16 @@ void FlutterWindowPortalWindowsPlugin::HandleControlMethod( r.top + static_cast(yf * (r.bottom - r.top))); } - engaged_.insert(hwnd); + // Watch where the window actually landed, not where we asked it to go — + // an app that clamps to its own minimum size settles elsewhere, and + // baselining on the request would report alignment as broken immediately. + HWND host = HostWindow(); + watcher_->SetHostWindow(host); + watcher_->Watch(hwnd, r); EmitEngagement("engaged", hwnd, EncodableMap{}); // Return the settled frame in host-view-local logical coords so the // consumer can camera-match. Convert screen physical → host-view logical. - HWND host = HostWindow(); POINT origin{0, 0}; if (host) ClientToScreen(host, &origin); double hostScale = host ? ScaleForWindow(host) : scale; @@ -400,17 +484,11 @@ void FlutterWindowPortalWindowsPlugin::HandleCaptureMethod( return; } - // Windows.Graphics.Capture → Flutter texture is the remaining slice. - // Everything below needs a Direct3D11CaptureFramePool feeding a - // GpuSurfaceTexture/PixelBufferTexture via the registrar's texture - // registrar. Staged; report clearly rather than silently returning null. - if (method == "startStream" || method == "stopStream" || - method == "reconfigureStream" || method == "streamStatus" || - method == "captureSingleFrame" || method == "captureSingleDisplayFrame") { - result->Error( - "unimplemented", - "Windows capture (Windows.Graphics.Capture) is not implemented yet; " - "window control + enumeration are. See the plugin README roadmap."); + if (CaptureController::Handles(method)) { + const auto *args = std::get_if(call.arguments()); + EncodableMap empty; + capture_controller_->HandleMethodCall(method, args ? *args : empty, + std::move(result)); return; } diff --git a/packages/flutter_window_portal_windows/windows/flutter_window_portal_windows_plugin.h b/packages/flutter_window_portal_windows/windows/flutter_window_portal_windows_plugin.h index 8ee9e92..f5df2f1 100644 --- a/packages/flutter_window_portal_windows/windows/flutter_window_portal_windows_plugin.h +++ b/packages/flutter_window_portal_windows/windows/flutter_window_portal_windows_plugin.h @@ -7,19 +7,22 @@ #include #include -#include #include namespace flutter_window_portal_windows { +class CaptureController; +class EngagementWatcher; +class PlatformTaskRunner; + // Windows backend for flutter_window_portal. Registers the two method // channels the shared Dart method-channel implementation speaks // (`flutter_window_portal/control` + `/capture`) and drives Win32 window -// control + Windows.Graphics.Capture. +// control plus Windows.Graphics.Capture. // -// Status: window CONTROL + enumeration implemented; per-window CAPTURE -// (Windows.Graphics.Capture → Flutter texture) is staged (see the capture -// handlers — they return an "unimplemented" error for now). +// Capture delivers frames straight into a Flutter texture as DXGI shared +// handles (see capture/texture_sink.h), so a live window stream never round +// trips its pixels through the CPU. class FlutterWindowPortalWindowsPlugin : public flutter::Plugin { public: static void RegisterWithRegistrar(flutter::PluginRegistrarWindows *registrar); @@ -45,16 +48,23 @@ class FlutterWindowPortalWindowsPlugin : public flutter::Plugin { void EmitEngagement(const std::string &method, HWND hwnd, const flutter::EncodableMap &extra); - // The top-level NSWindow-equivalent HWND hosting the Flutter view, for - // converting Flutter-view-local rects to screen coordinates during engage. + // The top-level HWND hosting the Flutter view, for converting + // Flutter-view-local rects to screen coordinates during engage. HWND HostWindow() const; flutter::PluginRegistrarWindows *registrar_; std::unique_ptr> control_; std::unique_ptr> capture_; - // HWNDs currently engaged (raised over their tile). - std::set engaged_; + // Declared before the members that post to it: destruction runs in reverse, + // so the runner outlives the capture workers that hand results back through + // it. + std::unique_ptr task_runner_; + std::unique_ptr capture_controller_; + + // Owns the engaged-window set: the OS hooks it installs are what let a + // disengage be reported rather than inferred. + std::unique_ptr watcher_; }; } // namespace flutter_window_portal_windows diff --git a/packages/flutter_window_portal_windows/windows/platform_task_runner.cpp b/packages/flutter_window_portal_windows/windows/platform_task_runner.cpp new file mode 100644 index 0000000..8b09927 --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/platform_task_runner.cpp @@ -0,0 +1,83 @@ +#include "platform_task_runner.h" + +#include + +namespace flutter_window_portal_windows { + +namespace { + +constexpr wchar_t kWindowClass[] = L"FlutterWindowPortalTaskRunner"; +constexpr UINT kRunTasksMessage = WM_APP + 0x51; + +} // namespace + +PlatformTaskRunner::PlatformTaskRunner() { + HINSTANCE instance = GetModuleHandleW(nullptr); + WNDCLASSEXW wc{}; + wc.cbSize = sizeof(wc); + wc.lpfnWndProc = &PlatformTaskRunner::WndProc; + wc.hInstance = instance; + wc.lpszClassName = kWindowClass; + // The class is process-wide: a second engine in the same process registers + // the same name and gets ERROR_CLASS_ALREADY_EXISTS, which is fine. + RegisterClassExW(&wc); + + window_ = CreateWindowExW(0, kWindowClass, L"", 0, 0, 0, 0, 0, HWND_MESSAGE, + nullptr, instance, this); +} + +PlatformTaskRunner::~PlatformTaskRunner() { + { + std::lock_guard lock(mutex_); + shutting_down_ = true; + } + if (window_) { + DestroyWindow(window_); + window_ = nullptr; + } + // Anything still queued is dropped deliberately — see Post(). + std::vector> dropped; + { + std::lock_guard lock(mutex_); + dropped.swap(tasks_); + } +} + +void PlatformTaskRunner::Post(std::function task) { + { + std::lock_guard lock(mutex_); + if (shutting_down_ || !window_) return; + tasks_.push_back(std::move(task)); + } + PostMessageW(window_, kRunTasksMessage, 0, 0); +} + +void PlatformTaskRunner::Drain() { + // Swap the queue out before running: a task may Post() another one, and + // holding the lock across arbitrary user code would deadlock. + std::vector> pending; + { + std::lock_guard lock(mutex_); + pending.swap(tasks_); + } + for (auto &task : pending) { + task(); + } +} + +LRESULT CALLBACK PlatformTaskRunner::WndProc(HWND hwnd, UINT message, + WPARAM wparam, LPARAM lparam) { + if (message == WM_NCCREATE) { + auto *create = reinterpret_cast(lparam); + SetWindowLongPtrW(hwnd, GWLP_USERDATA, + reinterpret_cast(create->lpCreateParams)); + } else if (message == kRunTasksMessage) { + auto *self = reinterpret_cast( + GetWindowLongPtrW(hwnd, GWLP_USERDATA)); + if (self) self->Drain(); + return 0; + } + return DefWindowProcW(hwnd, message, wparam, lparam); +} + +} // namespace flutter_window_portal_windows diff --git a/packages/flutter_window_portal_windows/windows/platform_task_runner.h b/packages/flutter_window_portal_windows/windows/platform_task_runner.h new file mode 100644 index 0000000..000c29c --- /dev/null +++ b/packages/flutter_window_portal_windows/windows/platform_task_runner.h @@ -0,0 +1,48 @@ +#ifndef FLUTTER_WINDOW_PORTAL_WINDOWS_PLATFORM_TASK_RUNNER_H_ +#define FLUTTER_WINDOW_PORTAL_WINDOWS_PLATFORM_TASK_RUNNER_H_ + +#include + +#include +#include +#include + +namespace flutter_window_portal_windows { + +// Hops work back onto the Flutter platform thread. +// +// Capture frames arrive on Windows.Graphics.Capture's free-threaded pool and +// the one-shot grabs run on their own worker, but neither a MethodResult nor +// MethodChannel::InvokeMethod may be touched off the platform thread. Anything +// crossing that boundary goes through Post(). +// +// Constructed on the platform thread; owns a message-only window whose WndProc +// therefore runs there too. Post() is safe to call from any thread. +class PlatformTaskRunner { + public: + PlatformTaskRunner(); + ~PlatformTaskRunner(); + + PlatformTaskRunner(const PlatformTaskRunner &) = delete; + PlatformTaskRunner &operator=(const PlatformTaskRunner &) = delete; + + // Queues |task| to run on the platform thread. Tasks queued after the runner + // begins shutting down are dropped rather than run on the wrong thread — + // callers must treat delivery as best-effort (every caller here is either a + // Dart notification or a MethodResult whose engine is going away anyway). + void Post(std::function task); + + private: + static LRESULT CALLBACK WndProc(HWND hwnd, UINT message, WPARAM wparam, + LPARAM lparam); + void Drain(); + + HWND window_ = nullptr; + std::mutex mutex_; + std::vector> tasks_; + bool shutting_down_ = false; +}; + +} // namespace flutter_window_portal_windows + +#endif // FLUTTER_WINDOW_PORTAL_WINDOWS_PLATFORM_TASK_RUNNER_H_ diff --git a/pubspec.yaml b/pubspec.yaml index 5234b15..f84bf84 100644 --- a/pubspec.yaml +++ b/pubspec.yaml @@ -1,5 +1,5 @@ name: flutter_window_portal -description: "Capture a native OS window into a Flutter texture and control its geometry, raise, and engagement lifecycle. Federated: macOS today (ScreenCaptureKit + Accessibility), Windows planned (Windows.Graphics.Capture + Win32/UIA)." +description: "Capture a native OS window into a Flutter texture and control its geometry, raise, and engagement lifecycle. Federated: macOS (ScreenCaptureKit + Accessibility) and Windows (Windows.Graphics.Capture + Win32)." version: 0.1.0 homepage: https://github.com/FlutterFlow/flutter_window_portal repository: https://github.com/FlutterFlow/flutter_window_portal