Blog · 2026-08-29 · By Dhrumil Barot, Founder

Lemmalog’s 38× context cut: the extraction-cost break-even

Lemmalog cut LongMemEval answer context 38.52× while F1 rose from 0.222 to 0.463. We price the reader and keep extraction cost honest.

If Lemmalog extraction costs $10 once, a LongMemEval-shaped workload breaks even after 50 Sonnet 5 questions or 494 gpt-5.6-luna questions. A LoCoMo-shaped workload needs 323 or 3,226. Those are scenarios, not measured Lemmalog bills. The extraction token count is unpublished, so “38× cheaper overall” is not supported.

Prices verified 2026-08-29.

TL;DR

• LongMemEval answer context fell from about 104,000 to 2,700 input tokens per question. That is 38.52× smaller, or 97.4038% less. Published F1 moved from 0.222 for full context to 0.463 ± 0.010 for Lemmalog.
• LoCoMo is the counterexample. Context fell from about 18,900 to 3,400 tokens, or 82.0106% less. F1 moved from 0.542 to 0.533 ± 0.001.
• Repricing a 200-output-token scenario on Sonnet 5 cuts the LongMemEval reader from $210.00 to $7.40 per 1,000 questions. LoCoMo falls from $39.80 to $8.80.
• Extraction uses Claude Sonnet 4.6 once per conversation. Its token count is not public. Break-even is one-time extraction dollars / reader dollars saved per question.

Verdict table

The dollar table uses 200 output tokens per question. That is an editable scenario because the benchmark write-up reports answer context, not answer output. Output cost is equal on both sides and cancels from the savings calculation.

What we computed and why

Lemmalog converts a conversation into structured facts. Datalog rules maintain derived facts, provenance, retractions, and temporal state. A reader then receives the relevant maintained state instead of the full transcript.

This creates two separate large language model bills.

1. Extraction reads the conversation and produces facts. The author used Claude Sonnet 4.6, chunked and file-cached. It runs once per conversation.
2. Answering reads either the full history or Lemmalog’s selected state. This repeats for every question.

The public benchmark gives enough data to price the second line. It does not give extraction input tokens, output tokens, or dollars. We keep that line symbolic instead of inventing a receipt.

Our reader calculation is:

reader_cost = (input_tokens × input_price + output_tokens × output_price) / 1,000,000

The saving per question is simpler:

reader_saving = (full_input_tokens - Lemmalog_input_tokens) × input_price / 1,000,000

The assumed output count disappears because both paths produce the same answer shape. This is why the break-even can remain exact even when the output scenario is editable.

Assumptions table

These are reader-cost scenarios across current models. They do not reproduce the benchmark’s standardized reader model or transfer its F1 to another model.

LongMemEval: 101,300 input tokens saved per question

The input reduction is 104,000 - 2,700 = 101,300 tokens. Full context is 104,000 / 2,700 = 38.5185× larger. Lemmalog removes 101,300 / 104,000 = 97.4038% of reader input.

The quality receipt moves in the same direction. F1 gains 0.463 - 0.222 = 0.241. Relative to 0.222, that is a 108.5586% increase. The author ran Lemmalog three times and reports a ±0.010 spread.

That does not make Lemmalog the best memory system in the comparison. PropMem and SimpleMem scored higher on LongMemEval. The relevant finding is narrower: Lemmalog beat the published full-context F1 while sending far less context to the answering model.

The 200 output tokens add $2.00 per 1,000 Sonnet 5 answers to both sides. They do not create the $202.60 saving. The 101.3 million avoided input tokens do.

Put your actual answer shape into the LLM cost calculator. The missing input is the one-time extraction bill, which the calculator cannot infer from the benchmark.

LoCoMo: cheaper reader, slightly lower F1

LoCoMo saves 18,900 - 3,400 = 15,500 input tokens per question. Full context is 5.5588× larger. The input reduction is 82.0106%.

Quality moves the other way. Lemmalog reports 0.533 ± 0.001 F1 versus 0.542 for full context. That is a 0.009 F1 loss, or 1.6605% relative to the full-context result.

The percentage is lower than the 82.0106% input cut because the unchanged 200-token output bill remains. More importantly, this benchmark does not support a free quality claim. A workload owner must decide whether 0.009 F1 is worth the reader saving.

The extraction-cost break-even

Let E be the one-time extraction bill for one conversation. Let S be the reader saving per future question.

break_even_questions = E / S

For a $10 extraction scenario:

Round up in production. A 49.36-question threshold means the 50th question clears the scenario.

The often-quoted token form is:

queries_to_break_even = extraction_tokens / (full_context_tokens - memory_context_tokens)

That form is valid only when “extraction tokens” are converted into reader-input-token equivalents at the same rate. Actual extraction used Sonnet 4.6. Its input and output carry different prices. The dollar equation is the safe one.

If you later measure extraction as X Sonnet 4.6 input tokens and Y output tokens, then:

E = (3X + 15Y) / 1,000,000

Insert that dollar value into the table. Do not multiply the context ratio by the whole bill.

What this means for routing

Route after splitting ingestion from answering. They have different economics.

• Extraction is a one-time quality-sensitive job. A stronger model can make sense if bad facts poison many later answers.
• Answering repeats. That makes its model routing decision more sensitive to context size and query volume.
• Stable instructions may qualify for prompt caching. Retrieved facts change per question, so do not assume the entire 2,700-token prompt hits cache.
• Full-context cost grows with history. Lemmalog’s selected state is intended to stay bounded, but the public figures are benchmark averages, not a guaranteed context window ceiling.

Our zero-token agent-memory audit separates memory operations from final QA. The prompt-caching break-even guide handles repeated prefixes. Lemmalog adds a third choice: pay once to extract maintained state, then send less evidence to every reader call.

What to meter before building it

Measure the extraction path before choosing an architecture.

1. Record extraction input, cached input, and output tokens by conversation.
2. Count successful questions served from that extracted state.
3. Keep reader tokens separate from extraction tokens.
4. Track accepted answers, not only attempted calls.
5. Compare quality by question category. LongMemEval and LoCoMo disagree on the direction.

This turns an unknown E into a bill. It also exposes whether your conversation survives long enough to cross the threshold.

Honest tradeoff: extraction can lose twice

Do not build this pipeline for short-lived chats with one or two follow-up questions. The full-context path may end before extraction repays itself. Engineering, indexing, and fact-schema maintenance also cost money outside input/output token pricing.

Extraction errors are more serious than ordinary retrieval misses. A missing fact can make every later answer wrong. The author found that some benchmark failures came from facts that were never extracted. Deterministic derivation preserves provenance after extraction; it does not make the extractor deterministic.

Lemmalog also trails full context by 0.009 F1 on LoCoMo. If that gap maps to costly failures in your workload, the cheaper reader is not the cheaper outcome.

Caveats

The token counts are approximate. The F1 values come from the author’s public benchmark harness, not an independent Vynaris rerun. LongMemEval has 102 questions, while LoCoMo has 1,986. Neither is a production vulnerability-research trace.

Current model price rows reprice the published token shapes. They do not claim that deepseek-v4-flash, gpt-5.6-luna, gemini-3.5-flash-lite, Haiku 4.5, or Sonnet 5 would reproduce Lemmalog’s reported F1.

DeepSeek’s table uses peak cache-miss pricing. Its off-peak rates are half. OpenAI’s row uses short-context Standard pricing because the 104,000-token input fits that rate band. No cache discount is assumed anywhere.

FAQ

Is Lemmalog 38× cheaper than full context?

No. LongMemEval answer context is 38.52× smaller. The whole system also pays one-time extraction, storage, indexing, and answer output. Extraction tokens are not public.

How many questions does extraction need to repay?

Divide one-time extraction dollars by reader dollars saved per question. At a $10 extraction scenario, LongMemEval-shaped traffic breaks even after 50 Sonnet 5 questions or 494 gpt-5.6-luna questions.

Did quality improve on both benchmarks?

No. LongMemEval F1 rose from 0.222 to 0.463 ± 0.010. LoCoMo fell from 0.542 to 0.533 ± 0.001.

Why assume 200 output tokens?

The source publishes answer context, not answer output. We need an output scenario to show total reader bills. It cancels from the savings and break-even equations.

When should we keep full-context prompting?

Keep it for short conversations, low query reuse, or workloads where extraction misses are unacceptable. A simpler system can be cheaper when it ends before the break-even.

Sources

• Jordy Zomer’s Lemmalog benchmark write-up, published 2026-08-28 and verified 2026-08-29: benchmark sizes, token counts, F1, three-run method, extraction model, and the missing extraction bill.
• OpenAI API pricing, verified 2026-08-29: gpt-5.6-luna Standard short-context $0.20 input / $1.20 output per 1M tokens.
• Anthropic pricing, verified 2026-08-29: Sonnet 5 $2/$10, Haiku 4.5 $1/$5, and Sonnet 4.6 $3/$15 per 1M input/output tokens.
• Gemini API pricing, verified 2026-08-29: gemini-3.5-flash-lite Standard $0.30 input / $2.50 output per 1M tokens.
• DeepSeek pricing, verified 2026-08-29: deepseek-v4-flash peak cache-miss $0.44 input / $1.32 output per 1M tokens; off-peak is half.

All arithmetic is reproducible in artifacts/lemmalog-38x-context-cut-extraction-cost-break-even-math.py. No Vynaris traffic, customer, telemetry, or routing data is used.