Your AI assistant can now see your screen. Here’s how to use it well

Until recently, you had to describe a problem to an AI assistant in words. Now you can show it. Tap the share-screen button, pick a window, and ask about what you see: “why won’t this form submit?” or “what does this setting do?”

Think of it as a friend looking over your shoulder. It only sees what you choose to share, and only while sharing is on. It works because today’s assistants are multimodal.

One rule before you start: close anything private, like bank details, health records or other people’s messages.

The major assistants now accept a live view of your screen. Share a window or your phone screen, then talk or type about what’s on it, and the assistant answers with that view in front of it.

Where it earns its keep: spreadsheets (“why doesn’t column C add up?”), forms and settings you don’t understand, and unfamiliar software, where it can walk you through each click. It’s quicker than screenshots because you keep working while you ask.

The catch is privacy. Anything visible while you share can be processed, and some services keep recordings or transcripts. Check the retention settings, share a single window rather than the whole screen, and close tabs with personal data first.

Screen sharing turns a live view of a window into model context. Frames are sampled and sent alongside your speech or text, so the model can ground its answer in the interface itself rather than in your description of it. That is what multimodal models make possible.

The useful patterns for power users: debugging (share a terminal or stack trace and ask for the likely cause), data work (have it read a dashboard and question the numbers) and walkthroughs of software you rarely use. Expect it to miss small text and fast-changing views: zoom in and pause.

Two checks. Retention: some services store session recordings or transcripts, and business plans may handle this differently from personal ones. Data boundaries: a shared window can expose credentials, customer data or regulated information, so treat every share like a screenshot sent to a vendor. Developers can build similar features with vision-capable model APIs by sending screenshots, with the user’s consent.