Hardware for a Local Coding Agent: What Needs to Be Always On?

The confusing part of a “local coding agent” build is the word local. It can mean the agent interface runs on your desktop, the model runs on a GPU in another room, or both the code and inference stay on hardware you control. Those are different setups with different hardware needs.

Real user questions reflect that tangle: people ask which agent works with their local model server, why a setup fails despite answering chat prompts, and whether an 8 GB GPU can handle multi-step coding work. A separate thread asks what model, backend, and context fit an 8 GB card for real project work. Those are useful problem statements, not controlled hardware tests or evidence that one product is best (agent + local setup question, hardware and model-fit question).

The buying decision is therefore not “which agent needs a GPU?” It is: which parts must stay available, where will inference run, and what task quality and delay are acceptable? For general runtime tradeoffs, see Ollama vs llama.cpp vs vLLM; for a hardware-first model-fit framework, see The Local-AI Hardware Buying Framework.

Split the always-on part from the model

A coding-agent setup has at least two jobs:

1. Orchestration: the IDE or terminal integration, task queue, gateway, repository checkout, and tool permissions.
2. Inference: loading the language model and processing prompts, code, tool results, and conversation context.

They can share one machine, but they do not have to. A small, low-power host can keep a gateway or automation service reachable while sending model requests to a workstation, a model server that wakes when needed, or a cloud API. Continue, for example, documents configuring its Ollama provider against a remote machine’s API address. That is a software integration option, not a claim that every remote setup is secure or reliable by default.

This separation matters for “always-on.” If you want to message an agent at any hour, some orchestration endpoint must be reachable. It does not follow that a large GPU must sit at full power around the clock. If you require an immediate local answer with no wake-up delay and no external service, then the inference machine must also be available. If a few seconds to wake a desktop is acceptable, a scheduled or on-demand inference host may suit better. Measure actual idle draw and wake time for the hardware you own; don’t infer an electricity bill from the GPU model name alone.

Size the inference machine for the model and context

There is no reliable one-line VRAM minimum for “agentic coding.” A model’s weights are only part of the memory budget. The configured context, runtime overhead, operating system, IDE, and other loaded applications also need room. A longer context generally consumes more memory; reducing the context can resolve some out-of-memory errors, but it may also leave the agent with less repository history or fewer tool results in view.

The current software guidance illustrates why a universal number misleads. Ollama’s coding-tool launch guide recommends a context of at least 64,000 tokens for its listed workflows and gives one example—glm-4.7-flash—at about 23 GB of VRAM for that context. That is a vendor-published requirement for a named model/context combination, not a blanket requirement for every coding agent and not a LocalRig benchmark. Continue’s Ollama guide likewise says to adjust context to available memory and notes that some models claiming tool support may still fail in Agent mode. Check the precise model tag, runtime, quantization, context setting, and tool-call support before buying hardware.

Practical implications:

• Small GPU or CPU-only machine: treat it as a trial platform for small models, autocomplete, or a remote-model gateway—not as proof that a demanding, long-context coding loop will be comfortable. If inference is slow or tools fail, first separate model limitations from a broken integration.
• Midrange GPU: choose a specific model and target context, then verify its memory footprint with the runtime’s own status tools. Leave room for the desktop and other workloads. Do not assume the model file’s disk size equals peak VRAM use.
• Larger VRAM GPU or high-memory unified-memory system: this opens more model/context choices, but does not guarantee good code edits, dependable tool calls, or fast end-to-end task completion. Test the exact model-agent pairing on representative repository work.
• Separate always-on gateway and inference workstation: useful when availability matters but keeping the larger system awake does not. The tradeoff is another network/service boundary and possible wake latency; secure the endpoint and avoid exposing an unauthenticated model API to the public internet.

Test the agent, not just chat

A model that answers “write a function” in a chat window has not yet passed an agent test. An agent must follow the harness’s tool format, inspect files, choose actions, handle tool output, and recover from errors over multiple turns. Continue’s documentation explicitly makes tool use a requirement for Agent mode and describes cases where advertised support does not work as expected. A failure can come from the model, quantization, context configuration, runtime, or adapter—not only from too little VRAM.

Before upgrading, test three tasks in a disposable copy of a real repository: a small bug fix, a multi-file change with tests, and a task that requires the agent to inspect unfamiliar code. For each setup, record whether it completed, whether its tool calls parsed, how much human correction was needed, end-to-end time, and peak memory. Review every diff; do not grant an unattended agent write access to valuable repositories just because the model runs locally. If the small task fails at tool-call formatting, buying a larger GPU may not fix it.

When local inference makes sense

Local inference is a strong fit when keeping prompts and code on your own machine is a priority, you need offline operation, you already own hardware that can run the desired model, or you have steady enough usage to justify maintaining the host. “Local” still requires checking the agent’s telemetry and connected services: a local model does not by itself prove that every part of the workflow stays on-device.

The cost comparison should include more than the model API bill. For local use, include the purchase or opportunity cost of hardware, power while loaded and idle, cooling, storage, setup, maintenance, and the value of your time. For cloud use, check current pricing, usage limits, retention terms, network dependence, and the cost of repeated agent calls. A local machine you already own and use often has a different break-even point from a new GPU bought for occasional experiments. There is no honest universal “local is cheaper” threshold without usage and local electricity assumptions. Use the rent-vs-buy framework for the broader ownership calculation; this guide deliberately does not quote a fixed cloud price.

When cloud—or a hybrid—is the better fit

Use cloud inference when your local model cannot complete the work reliably, when you need a larger context or stronger capability than your machine can support, or when occasional usage does not justify buying and maintaining hardware. Cloud also removes the need to keep a suitable inference host available, though the orchestration environment and credentials still need care.

A hybrid arrangement is often the least-regret starting point: use a local model for bounded, private, or routine tasks, then route difficult work to a cloud model after checking what code and prompts are sent and under what terms. Conversely, a local workstation can serve several clients on a trusted home network while a small gateway remains available. In either case, start with narrow permissions, inspect proposed changes, and keep secrets out of prompts and test repositories.

A decision rule

Your priority	Start here	Hardware decision
Keep an agent endpoint reachable, with occasional coding	Separate gateway from inference	Keep the gateway modest; wake or remotely call the model host when needed
Private/offline coding on hardware already owned	Local model + a tool-compatible agent	Test one model, quant, runtime, and context before buying anything
Best results on difficult code tasks, used occasionally	Cloud model or local/cloud hybrid	Compare current service terms against the cost of a dedicated inference box
Several concurrent users or jobs	Treat it as a serving workload, not a personal agent	Benchmark concurrency and memory under the intended load before choosing a server

The least-regret purchase is usually the one you can postpone until a representative task fails for a known reason. First confirm that the agent, model, and runtime can exchange tool calls. Then measure memory and task completion at the context you actually need. Buy for that measured constraint—not for the phrase “AI agent” on a product page.

Who this is NOT for

• People who need a model recommendation without specifying a task, repository size, context, privacy requirement, and current hardware.
• Buyers expecting an 8 GB GPU—or any fixed VRAM tier—to guarantee dependable autonomous coding.
• Teams looking for a production multi-user inference-server capacity plan. That needs a defined concurrency target and load test, not this single-user buying framework.
• Anyone who wants unattended agents to modify important code without review, backups, or scoped permissions.

Sources

The vendor documentation above supports the integration and configuration statements; it does not establish independent coding quality. The Reddit links document the questions people are asking, not representative market demand or reproducible benchmark results. Hardware behavior varies by model revision, quantization, runtime, context, and host configuration. Recheck each vendor’s current instructions before purchase or deployment.