Can I Run Recursive Flint Locally? 24 GB Holds One Q4 File, Not the Other

Direct answer

Recursive Flint is a 39.8B MoE coding model, about 2.5B parameters active per token, with a 262,144-token context and an Apache 2.0 license. On 24 GB, Unsloth’s 18.49 GB 3-bit file is the short-context fit. ggml-org’s Q4_K_M is 23.39 GB and bartowski’s is 25.35 GB, so only the smaller Q4 file fits that card. Filled 262K context is unmeasured here.

Flint is a coding-agent model. The hardware question for an always-on agent box is the separate page, Hardware for a local coding agent. This page answers whether a named file fits a memory class.

What are the specs on the model card?

39.8B total, about 2.5B active, 262,144 tokens, Apache 2.0. Those figures are from the RecursiveSI/flint card, created 2026-10-21 and read 2026-10-22. The card’s footnote says the active-parameter count includes the output head and excludes the full input embedding. The active body alone is about 2.2B. “40B-A2.5B” is the card’s rounded name for the same model.

Property	Card value
Developer	Recursive
Architecture	Decoder-only mixture-of-experts Transformer
Total parameters	39.8B
Active parameters per token	About 2.5B, including the output head
Active body, excluding that head	About 2.2B
Layers	49: 2 dense, then 47 MoE
Experts	256 routed, 6 selected per token, plus one always-on shared expert
Attention	35 sliding-window layers, 14 full-attention layers
Sliding window	513 tokens (the current token and 512 before it)
Heads	32 query, 4 key/value, head dimension 128
Context	262,144 tokens, prompt and output together
Vocabulary	150,144
Knowledge cutoff stated on the card	June 2026
License	Apache 2.0
Inputs	Text

The same card’s release evaluations, at temperature 1.0, top-p 0.95, and top-k 20, are 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2, and 74.2% on SWE-bench Verified. The card says Terminal-Bench counts solutions that used web retrieval as unsuccessful, and that agentic scores are average pass@1 over at least five attempts. Those are Recursive’s scores. They are not tokens per second, and they are not a LocalRig run.

The official BF16 checkpoint is the safetensors repo. The card says hf download RecursiveSI/flint is about 80 GB. The tree API on 2026-10-22 sums the fifteen safetensors shards to 79,575,138,056 bytes (79.58 GB). Unsloth’s three BF16 GGUF shards sum to 79,576,479,200 bytes, which also rounds to 79.58 GB and is a different byte count. ggml-org’s single-file BF16 GGUF is 79,625,761,760 bytes (79.63 GB). The serving kit on that card is SGLang on one B300 or one H200. Transformers modeling code is not in the repo yet. That serving path is a datacenter GPU, not a 24 GB desktop.

Which Q4_K_M file is which?

Three files share the name Q4_K_M and three byte sizes. The Hugging Face tree API on 2026-10-22:

Repo and file	Bytes	Decimal GB
ggml-org/Flint-GGUF · Flint-Q4_K_M.gguf	23,387,911,136	23.39
lmstudio-community/flint-GGUF · flint-Q4_K_M.gguf	24,166,841,184	24.17
bartowski/RecursiveSI_flint-GGUF · RecursiveSI_flint-Q4_K_M.gguf	25,346,651,360	25.35

Decimal GB is bytes ÷ 1,000,000,000, rounded to two decimals. The ggml-org and bartowski sizes match the 23.39 GB and 25.35 GB figures that were in circulation. They are different files. The gap is 1,958,740,224 bytes (1.96 GB).

ggml-org’s card says that in its Q4_K_M the small non-expert tensors (embeddings, attention, shared expert) stay at Q8_0. Keeping tensors at Q8_0 uses more bytes, and ggml-org’s file is the smaller of the two, so that sentence does not explain the gap. Bartowski’s card, read the same day, does not state a cause either. This page leaves the cause unverified.

Unsloth’s flint-UD-Q4_K_XL.gguf is a fourth 4-bit file, 25,054,688,704 bytes (25.05 GB). The name is UD-Q4_K_XL, not Q4_K_M. Unsloth’s guide prints 25.05 GB, which matches the byte size. Their 4-bit memory band is 26 GB of RAM plus VRAM, or unified memory. That band is a vendor total-memory figure. It is wider than the file.

Which memory class holds the file?

Compare the file with the class. Then stop. A file smaller than the class is a weights fit. KV cache, the desktop, and a filled 262,144-token window are extra. This page does not have a measured KV byte count at 262,144. Where Recursive states a higher minimum for a full-context command, that minimum is quoted as their recipe.

Unsloth’s total-memory bands, copied 2026-10-22 from their Flint guide (RAM + VRAM, or unified memory):

Band	Unsloth total memory
2-bit	13–15 GB
3-bit	16–20 GB
4-bit	26 GB
6-bit	38 GB
8-bit	44 GB

Their guide says the 3-bit UD-Q3_K_XL file is 18.50 GB and fits 24 GB at short context. The tree API size of that file is 18,494,977,472 bytes, which rounds to 18.49 GB. Both figures are the same file. The llama.cpp example on that page uses --ctx-size 8192.

Discrete GPUs

Class	File fits as weights	File does not fit	What a source adds
16 GB	Unsloth UD-IQ3_XXS 14.54 GB, UD-Q2_K_XL 13.56 GB, UD-IQ2_XXS 11.86 GB. Bartowski IQ2_XXS 12.03 GB	Unsloth UD-Q3_K_XL 18.49 GB. Bartowski IQ3_XXS 17.49 GB. Every Q4_K_M above. Official int4 GGUF 25.99 GB. NVFP4 22.86 GB	One speed run on an RTX A4000 16 GB exists. See the speed section. The quant repo is not named there
24 GB	UD-Q3_K_XL 18.49 GB. ggml-org Q4_K_M 23.39 GB, with little spare. lmstudio-community Q4_K_M 24.17 GB, with less spare. NVFP4 file 22.86 GB. Bartowski Q3_K_M 19.53 GB	Bartowski Q4_K_M 25.35 GB. Unsloth UD-Q4_K_XL 25.05 GB. Official int4 GGUF 25.99 GB	Unsloth’s 4-bit band is 26 GB. Their 3-bit guide is the 24 GB short-context path
32 GB	All three Q4_K_M files. UD-Q4_K_XL 25.05 GB. Official int4 file 25.99 GB. NVFP4 file. Bartowski Q5_K_M 30.81 GB, tight. Unsloth UD-Q5_K_XL 29.92 GB	Bartowski Q6_K 35.20 GB. ggml-org Q8_0 42.35 GB. Unsloth’s 6-bit band is 38 GB and the 8-bit band is 44 GB	32 GB is the first class that holds every published Q4_K_M as a file
48 GB	Q8_0 at 42.35 GB (ggml-org and bartowski; Unsloth’s Q8_0 is 42.30 GB). Q6_K. Unsloth’s 8-bit band of 44 GB	BF16 GGUF. ggml-org’s single-file BF16 is 79.63 GB. Unsloth’s three BF16 shards sum to 79.58 GB	Recursive’s int4 GGUF card asks for at least 48 GB of unified memory for its Metal command at 262,144 context with F16 KV. The NVFP4 card asks for at least 48 GB of GPU memory, or unified memory on Spark, for its CUDA command

The 24 GB row is the buying fork. A used RTX 3090 is the usual 24 GB card, and it matches Unsloth’s short-context 3-bit path. It also matches ggml-org’s 23.39 GB Q4_K_M only as a weights file. It does not match bartowski’s 25.35 GB Q4_K_M. Download the file whose bytes you measured, not the quant name alone.

Browse used RTX 3090 24GB on eBay → for that 24 GB class. The checklist is the used RTX 3090 buying guide.

Check RTX 4060 Ti 16GB on Amazon → only if the file is one of the sub-16 GB quants in the table. The 18.49 GB 3-bit file is already over that card.

A 32 GB card is the class that holds both disputed Q4_K_M files and Unsloth’s 25.05 GB 4-bit file. Renting that class before buying is the cheaper experiment: RunPod, Vast.ai, GPUMart.

If the job is a dense daily driver on one used 24 GB card, the current LocalRig default remains Qwen 3.8 27B. Flint is the MoE coding model with a much smaller active count and a file size that still depends on which Q4 you grabbed. GLM-4.7-Flash is the other 24 GB coding-sized MoE already on this site (about 18 GB at 4-bit in Unsloth’s older table). Gemma 4 26B-A4B is a different MoE whose 4-bit band is 16–18 GB. gpt-oss-20b is the easier 16 GB guest. None of those file sizes are Flint’s.

Macs and Spark

Unified memory is shared with the operating system. The file test is the same. The spare room is smaller, and this page does not have a measured macOS overhead for Flint.

Pool	What the files say	What a source measured or required
24 GB Mac	mlx-community flint-mixed-3-4bit sums to 17.84 GB. The card prints 17.8 GB. Official MLX int4 sums to 22.57 GB. The card prints about 22.6 GB. UD-Q3_K_XL is 18.49 GB	The mlx-community card says peak use was about 19.5 GB, measured on a 64 GB Mac, and that a real 24 GB Mac was not tested. It also says macOS gives the GPU about 16 GB on a 24 GB Mac until iogpu.wired_limit_mb is raised
32 GB Mac	All three Q4_K_M files fit as files. UD-Q4_K_XL 25.05 GB fits. Official int4 GGUF 25.99 GB fits as a file	The same mlx-community card says the wired-limit step is for 24 GB Macs
48 GB Mac	Q8_0 at 42.35 GB fits as a file. Official int4 GGUF is the file Recursive’s Metal command targets	Recursive’s int4 GGUF card says the Mac needs at least 48 GB of unified memory for that command at 262,144 context
64 GB Mac	Q8_0 fits. BF16 at 79.63 GB does not	The mlx-community speed table is an M5 Max with 64 GB. See below. It is one source
128 GB Mac	ggml-org BF16 at 79.63 GB is under 128 GB as a file	Operating-system overhead at that size is not measured here
DGX Spark, 128 GB, as TandemLLM names it	BF16 at 79.63 GB is under 128 GB as a file. EXL3 6.0 safetensors sum to 30.54 GB. The card prints 30.5 GB. EXL3 4.0 sums to 20.63 GB. The 4.0 card prints 20.6 GB	TandemLLM’s 6.0 card says the weights occupy 28.5 GiB in memory on one DGX Spark (GB10, 128 GB). The speed rows are in the next section
64 GB NVIDIA RTX Spark	Q8_0 at 42.35 GB fits as a file. BF16 at 79.63 GB does not	Unsloth’s 150+ tok/s sentence names this machine. It does not name a quant. It is a vendor claim, and the product name is RTX Spark, not the DGX Spark sentence above

The two Spark names are doing different jobs. DGX Spark 64GB vs 128GB is the memory split for the Grace Blackwell box. RTX Spark laptops is the laptop line. Unsloth’s speed sentence says “64GB NVIDIA RTX Spark.” TandemLLM’s speed table says “One DGX Spark (GB10, 128 GB).” This page does not merge those machines.

How fast is it, and who measured it?

Three single-source measurements, plus two vendor claims that omit the quant. Nothing in this section is a LocalRig run. Community-cited figures stay labeled.

RTX A4000 16 GB, one llm-speed run

llm-speed run r_e8r7fxw1ke9, measured 2026-10-22. Model label on the page: Flint-40B-A2.5B-UD-ctx8k IQ3_XXS. Runtime string on the page: [email protected] (build 11803, commit a37cc4610). That commit is the object GitHub returns for tag b11803. Hardware: RTX A4000 16 GB.

Workload, as the page names it	Decode tok/s	Prefill tok/s	TTFT
chat-short	92.89	331.7	295 ms
chat-long	86.67	1,458.7	2,066 ms
concurrent-decode	90.24	not printed	not printed
agent-trace	84.71	3,147.3	645 ms

The 92.9 tok/s figure in circulation is the chat-short cell rounded from 92.89. The agent-trace cell, which is the closer label to a coding agent, is 84.71 tok/s. The page does not print the prompt length, the batch, the driver, the OS, or the Hugging Face repo. The “UD” in the model label matches Unsloth’s naming, and Unsloth’s UD-IQ3_XXS file is 14.54 GB, which is under 16 GB. Bartowski’s IQ3_XXS is 17.49 GB, which is over 16 GB. The speed page does not say which file ran. Single source.

M5 Max 64 GB, mlx-community’s own table

The mlx-community/flint-mixed-3-4bit card, read 2026-10-22, reports speed on an M5 Max with 64 GB. The engine for the MLX rows is mlx-vlm 0.7.10. Output length changes the rate: 119 tok/s at 14.0K output tokens, 91 and 76 tok/s at 16.7K and 22.2K in the server row, and 76 tok/s at 26.6K with mlx-lm from a pull request the card says is not released. A separate row uses Recursive’s rec 0.1.0 on the official INT4 at 117 tok/s for 8.8K output tokens. The card says the 24 GB fit was inferred from the 64 GB machine and was not run on a 24 GB Mac. Prompt length and batch are not in that speed table. Single source, and it is the quantizer’s own card.

DGX Spark 128 GB, TandemLLM EXL3 6.0

The EXL3 6.0 card, read 2026-10-22, describes one DGX Spark (GB10, 128 GB), TandemLLM 0.14.0, one request at a time, 512 greedy tokens, thinking off, a fresh server, and one run per cell. Plain decode on that card: 98.4 tok/s prose, 97.2 code, 97.1 chat, 94.3 edit, 94.3 copy, 95.0 on a game-file prompt of 2,048 tokens. A 4,096-token prompt reads at 5,045 tok/s. Weights in memory: 28.5 GiB. Load time: about 40 seconds.

The drafted “copy” cell is 353.4 tok/s. The card says copy and edit are asks whose answer repeats much of the prompt. That cell is a repetition task. It is not a coding-agent decode, and it is not the 150 tok/s vendor line.

The same card, for TandemLLM 0.14.1, reports plain decode of 64 greedy tokens at 52.0 tok/s with a 64k context and 35.4 tok/s at 128k, one run each, exact cache. An FP8 cache mode is faster on that card (66.1 and 50.1) and the card says the output is then no longer the exact model. Single source.

The 6.0 card says the 4.0 pack decodes about 38% faster. The 4.0 card does not include this speed table. This page does not turn 38% into a tok/s number.

Vendor claims

Richard Socher, 21 October 2026: “125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark.” The post does not name the Mac chip, the Spark memory, the quant, the runtime, the context, or the batch. Vendor claim.

Unsloth’s Flint guide, read 2026-10-22: “125+ tokens/sec on an M4 Pro MacBook Pro and 150+ on a 64GB NVIDIA RTX Spark.” That sentence is more specific about the machines and still omits the quant, the runtime version, the context, and the batch. Vendor claim. It is not the same wording as the post, and this page does not splice them into one configuration.

How do the runtimes load it?

Four loaders have public files. They are not interchangeable builds. Sampling on the model card, and on the quant cards that repeat it, is temperature 1.0, top-p 0.95, top-k 20. Preserve reasoning_content when a client replays history. The commands below are the ones the cards print. LocalRig has not run them.

llama.cpp

Tag b11803 was published 2026-10-21. GitHub marks the release as a pre-release. The tag object is commit a37cc4610, message “models : add Flint model (FlintForCausalLM) (#30928)”. ggml-org’s GGUF card points at that pull request and shows:

llama serve -hf ggml-org/Flint-GGUF

Recursive’s int4 GGUF is flint-int4-q4_1.gguf, 25,992,966,976 bytes (25.99 GB). The card prints “26.0 GB.” It is aimed at Apple Silicon Metal. The card says the Mac needs at least 48 GB of unified memory, then serves the full 262,144 context with F16 KV:

llama-server -m flint-int4-gguf/flint-int4-q4_1.gguf \
  --alias flint -ngl all -c 262144 -np 1 --cache-ram 0 \
  -b 256 -ub 128 -t 8 -fa on -ctk f16 -ctv f16 --jinja \
  --temp 1 --top-p 0.95 --top-k 20 --min-p 0 \
  --host 127.0.0.1 --port 8080

The NVFP4 file is flint-nvfp4.gguf, 22,863,084,160 bytes (22.86 GB). The card prints “22.9 GB.” It says the runtime was tested on an NVIDIA B300, an RTX PRO 6000 Blackwell, an RTX Spark, and a DGX Spark, and it asks for at least 48 GB of GPU memory, or unified memory on Spark. The command uses -c 262144 and leaves --swa-full unset.

Unsloth’s guide does not build that tag. It clones unslothai/llama.cpp on branch iq1-narrow, downloads flint-UD-Q4_K_XL.gguf, and runs llama-cli at --ctx-size 8192. Stock b11803 and that fork branch are two builds. Use the build the file’s card names.

The runtime comparison for a different model is Ollama vs llama.cpp vs vLLM. There is no Flint vLLM packet on this page.

Ollama

v0.40.8, published 2026-10-21, says ollama run flint and says Flint is supported on macOS, with other GPU support in a subsequent release. The release text calls it a 40B mixture-of-experts with about 2.5B active, a 262K context, and Apache 2.0.

The library tags page, read 2026-10-22, lists flint:40b and flint:latest as the same digest 2a087bdb088a, 23 GB, 256K context. ollama run flint:40b is the command on the flint:40b page. Other tags on that list: flint:40b-a2b (same digest, 23 GB), flint:40b-a2b-gguf-nvfp4 (23 GB), flint:40b-a2b-q4_1 (26 GB), flint:40b-a2b-mlx-bf16 (80 GB), and flint:40b-a2b-qat-int4 (23 GB). Each tag is labeled 256K. The model card writes 262,144. 256 × 1,024 = 262,144, so the labels can be the same window. This page uses 262,144 when a source writes the integer.

The library page also says quantized versions are available for Apple Silicon and other GPUs. That sentence and the v0.40.8 macOS sentence are both dated sources, and they do not match. This page did not hash Ollama’s blobs against the Hugging Face files. The 26 GB and 80 GB tags line up with the rounded sizes Recursive prints for the int4 GGUF and the BF16 download. A matching round number is not a checksum.

The tag string 40b-a2b is Ollama’s. The card’s rounded name is 40B-A2.5B, and the library text still says 39.8 billion parameters. This page does not read “a2b” as a second parameter count.

MLX

Official MLX weights are RecursiveSI/flint-int4. The card says projections and experts are group-64 affine INT4, embeddings are INT8, and the weights occupy about 22.6 GB. The five safetensors sum to 22.57 GB. The card’s launcher is rec launch pi from the Rec install guide. The GGUF twin is the Metal command above.

mlx-community/flint-mixed-3-4bit is an unofficial build. The card says so. It says mlx-vlm 0.7.10 and later can run Flint, prints a 17.8 GB size (the shards sum to 17.84 GB), and gives this generate line:

mlx_vlm.generate --model mlx-community/flint-mixed-3-4bit --temperature 1.0 --top-p 0.95 --top-k 20 \
  --max-tokens 32768 --prompt "Write a Python function that merges overlapping intervals."

The same card says Flint support in mlx-lm was still pull request 1998 and not a release, as of that README. It also reports a tool-call failure mode in mlx-vlm 0.7.10 (41 of 85 calls in their OpenCode test used a wrong argument name). That is their test, on the 64 GB machine, not a 24 GB Mac result.

EXL3

There is no TandemLLM organization on the Hub in the search run on 2026-10-22. The packs are 0xBakeer/TandemLLM-Flint-EXL3-4.0 and 0xBakeer/TandemLLM-Flint-EXL3-6.0. The 6.0 card says TandemLLM runs the pack from release 0.14.0, on a DGX Spark, with:

~/TandemLLM/run/ops/use-profile.sh flint-exl3-6.0

It also says stock exllamav3 does not contain the Flint architecture. A loader needs the patch in that repo, or its own Flint code, and it has to honor expert_interm_div of 128. The card says a loader that ignores the divisor returns expert outputs 128 times too small. The conversion source revision printed on the card is 2a247be0, which is the RecursiveSI/flint revision the Hub API returned on 2026-10-22.

Who this is NOT for

• People who treat 39.8B total parameters as a dense 40B download. The active count is about 2.5B. The file is still 11.86 GB to 79.63 GB depending on the quant.
• A 24 GB shopper who downloaded bartowski’s 25.35 GB Q4_K_M, or Unsloth’s 25.05 GB UD-Q4_K_XL, and expects the whole file on the GPU.
• A 16 GB shopper who downloaded bartowski’s 17.49 GB IQ3_XXS. Unsloth’s 14.54 GB UD-IQ3_XXS is the IQ3-class file under that card, and the speed page does not prove which one it ran.
• Anyone who quotes 125+ or 150+ tok/s as a measured LocalRig result. Both sentences are vendor claims, and both omit the quant.
• Anyone who needs a filled 262,144-token context on 24 GB. Recursive’s own full-context commands name 48 GB.
• Anyone buying a Spark from a speed sentence without checking which Spark. Unsloth names a 64 GB RTX Spark. TandemLLM names a 128 GB DGX Spark.

What this page could not verify

• The cause of the 1.96 GB gap between the ggml-org and bartowski Q4_K_M files.
• The Hugging Face repo, prompt length, batch, driver, OS, and GPU residency behind llm-speed run r_e8r7fxw1ke9.
• A checksum match between any Ollama tag and a Hugging Face file.
• Which of the two Ollama sentences about non-Mac GPUs is current. The release note and the library page disagree, and this page has no NVIDIA Ollama log.
• A filled 262,144-token KV size on any GPU.
• A 24 GB Mac run. The mlx-community card says that test was not done.
• Absolute tok/s for the EXL3 4.0 pack. The 6.0 card says it is about 38% faster and does not print the 4.0 table.
• A LocalRig first-party row. There is none.

Methodology

• Identity and context: RecursiveSI/flint model card, created 2026-10-21, read 2026-10-22. Active parameters are the card’s “about 2.5B including the output head” and “about 2.2B” for the body.
• File sizes: Hugging Face tree API, 2026-10-22. Decimal GB = bytes ÷ 1e9, two decimals. Shard totals are sums of the weight files only, not tokenizer sidecars.
• Speed: each row names its page. llm-speed is one run on 2026-10-22. mlx-community and TandemLLM figures are the publishers’ own cards. The 125+ and 150+ lines are vendor claims.
• Runtime versions: llama.cpp tag b11803, commit a37cc4610, published 2026-10-21, marked pre-release. Ollama v0.40.8, published 2026-10-21. Library tags read 2026-10-22.
• LocalRig first-party: none.

Sources

• RecursiveSI/flint model card, created 2026-10-21, accessed 2026-10-22. Specs, license, release scores, SGLang note.
• ggml-org/Flint-GGUF, tree API 2026-10-22. Q4_K_M 23,387,911,136 bytes. Q8_0 42,350,359,520 bytes. BF16 79,625,761,760 bytes.
• bartowski/RecursiveSI_flint-GGUF, tree API 2026-10-22. Q4_K_M 25,346,651,360 bytes.
• lmstudio-community/flint-GGUF, tree API 2026-10-22. Q4_K_M 24,166,841,184 bytes.
• unsloth/flint-GGUF and Unsloth’s Flint guide, accessed 2026-10-22.
• RecursiveSI/flint-int4-gguf and RecursiveSI/flint-nvfp4-gguf, accessed 2026-10-22.
• RecursiveSI/flint-int4 and mlx-community/flint-mixed-3-4bit, accessed 2026-10-22.
• 0xBakeer/TandemLLM-Flint-EXL3-6.0 and EXL3 4.0, accessed 2026-10-22.
• llama.cpp b11803, published 2026-10-21. Tag commit a37cc4610.
• Ollama v0.40.8, published 2026-10-21. Library tags, accessed 2026-10-22.
• Richard Socher, 2026-10-21. Vendor speed wording.
• llm-speed r_e8r7fxw1ke9, measured 2026-10-22. Single source.
• Hardware for a local coding agent.