-
feat(x): assistant screen pointer + click/type control during screen share
frostbyte_neo 发布于
2026-08-10 16:14:48 +00:00 On calls with a live screen share, the assistant can now point at the
user's real screen and — with permission — click and type on it.- screen-pointer builtin (permission: none): animated laser-dot overlay
with ping rings and optional label at fractional frame coordinates, on
a transparent click-through screen-saver-level panel over the shared
(primary) display. Auto-hides; torn down the instant the share ends. - screen-control builtin (permission: prompt): real CGEvents posted via
osascript JXA + ObjC bridge (no native modules) — cursor move + click
with proper double-click clickState, unicode typing with Return
handling. Deliberately NOT System Events "click at", which returns
success without clicking on modern macOS. Accessibility is the single
permission; clicks are telegraphed by the pointer ~600ms before landing. - Both tools ride the app-navigation skill; video-mode prompt teaches
point-while-explaining vs act-only-on-request, one action per step
verified against the next frame, never type secrets. - Auto-permission: voice/PTT submissions no longer fall back to manual
mode (composer default is auto), and the classifier prompt knows
screen-control — explicit asks approve silently, unrequested/secret/
destructive actions still card. - Settings → Permissions panel: every OS permission grouped by feature
with honest statuses (unknowns stay unknown), grant buttons where a
native prompt exists, System Settings deep links elsewhere (incl.
Accessibility/Automation/Notifications panes), focus-triggered refresh. - Companion pill/tucked mascot: pulsing green listening halo + loud
status chip while the mic gate is open (PTT feedback was too subtle). - Injection failures carry the raw macOS error and map to the exact
grant to fix, including the stale-grant case of ad-hoc signed builds.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
下载附件
- screen-pointer builtin (permission: none): animated laser-dot overlay