AI, Closer to the Work

Google's AI Edge Foresight is a useful place to look at what is changing in local inference. Announced in October 2026, the experimental Mac app uses Gemma 4 and EmbeddingGemma 2 to help turn shorthand into fuller meeting notes and retrieve information from conversations and personal files. Google describes the AI processing as fully local.

What interests me is the relationship between those capabilities. The conversation provides material for a note, while the user's shorthand indicates what mattered to them. Retrieval brings relevant information into the task, and generation helps put it together. The inference happens on the machine where the work is taking place.

An announcement does not tell us how consistently that works across difficult meetings, unfamiliar terminology, or different hardware. Foresight is still an experiment. But it makes a particular product direction tangible: more of the interpretation and generation around everyday work can happen locally.

That's more interesting than the familiar question of whether a small model can beat a large one.

A wider range of work can happen locally

There is a long history behind this. In The Computer for the 21st Century (1991), Mark Weiser wrote:

“The most profound technologies are those that disappear.

He was describing computing that becomes part of ordinary activity, allowing people to concentrate on what they are doing. Today's local language models do not fulfil that entire vision. Still, it is a useful way to think about a capability becoming available inside an existing activity, without demanding a separate destination for every use.

On-device machine learning itself is well established. Google described local transcription, sound classification, and search in Pixel's Recorder app in 2019. In December 2023, it demonstrated Gemini Nano summarizing Recorder content on Pixel 8 Pro without a network connection.

The change is therefore an expansion of what can happen on the device. Recognition and classification are being joined by more flexible language generation, extraction, and retrieval. A transcript can become material for another task, rather than simply an alternative representation of the recording.

There is also a change in how developers receive these capabilities. Apple's 2025 Foundation Models framework exposed its on-device language model to apps, with structured generation and tool calling. In that year's research overview, Apple described the roughly three-billion-parameter model as suitable for tasks including summarization, extraction, and text refinement, while stating: “It is not designed to be a chatbot for general world knowledge.

That limitation is useful context. A product does not always need a model capable of answering questions about everything. It may need to interpret the information already inside a task and produce a modest, useful result.

Proximity changes the interaction

There are two kinds of proximity here, and they are easy to confuse. One is computational: inference happens on the device. The other is contextual: the application has access to the information needed for the task. A local model does not automatically know what is in a person's files, and a remote model can be given excellent context.

The interesting combination is a useful model, relevant information available with permission, and an interaction that keeps the person close to their original work.

AI is becoming available for smaller pieces of work: interpreting a phrase, finding a passage, expanding a note, or extracting a few fields. Those operations can sit inside familiar application behavior. Their usefulness may be apparent in the completed work, without the user having to manage a separate conversation with an assistant.

The computation still has a cost

My interest in AI economics makes the phrase “free inference” difficult to leave unexamined. Local execution can remove a metered cloud inference charge from a feature. The computation still consumes resources, and those resources now sit on the customer's device.

That changes whose constraints become visible. Memory is shared with other applications. Battery capacity matters away from a charger. Sustained workloads can behave differently from a short demonstration. A feature that is inexpensive for its vendor to serve can still be expensive in the attention or device capacity it demands from its user.

The MELTing Point research, published in 2024, examined language-model execution across mobile and edge devices. Its measurements exposed substantial variation across hardware and frameworks, along with memory, energy, and thermal constraints. In their accompanying account, the researchers put the distinction plainly: “Tractability of the LLM inference workload does not imply deployability.”

Those measurements describe the models and devices they tested in 2024, rather than the limits of every current device. The distinction remains useful: demonstrating that inference runs establishes feasibility. It leaves open how comfortably the capability coexists with the rest of someone's work.

This makes local inference an interesting economic shift, with costs distributed differently across the system. A vendor can reduce dependence on per-request cloud spending while taking on compatibility, model delivery, and support work. The customer supplies hardware and energy. Whether the whole arrangement is more efficient depends on the workload and how often the capability is useful.

Local capability and shared knowledge

There is another older idea worth returning to. In their 2019 essay Local-first Software, Martin Kleppmann and colleagues argued for software that combines local ownership and offline work with collaboration. Their proposal treated the local copy as primary, with servers assisting synchronization, rather than making every interaction subordinate to a remote service.

Local inference adds another possible layer to that relationship. Having a document available locally can support reading and editing it. Having a suitable model available locally can also support operations such as extraction or summarization. Access to both the material and the capability becomes part of what the person has available to work with.

But local inference alone does not make an application local-first, guarantee ownership, or establish that all its data remains on the device. Storage, synchronization, telemetry, and optional integrations are separate parts of the system. The location of a model answers one architectural question.

The same distinction matters for organizational knowledge. A personal meeting note and an authoritative record of a team decision have different responsibilities. A local assistant can help someone prepare the note; deciding what becomes shared, who can change it, and which version is authoritative requires a wider system.

These developments give us a more flexible division of work between devices and shared services. The latter remain useful for coordination, current information, and capabilities beyond a particular machine. What is becoming more concrete is the possibility of completing meaningful parts of the work without making every language operation a remote request.

My product taste is drawn to that possibility. I like software that keeps the material I am working on available, helps within the activity, and makes its limits understandable. Local inference can contribute to that experience when its capabilities fit the task. It does not establish the fit by itself.

For my own work, I would value an assistant that helps me recover the relevant passage, make sense of an unfinished note, and continue with what I was doing. The shift becomes meaningful to me when those small acts of assistance feel like dependable parts of the software I already use, with enough visibility to understand and correct what they contribute.