How to Cut Token Costs for AI-Powered Log Analysis

Feed an LLM every log line your systems produce and you'll get a bill that looks like a denial-of-service attack on your own wallet. Most SOC teams bolt an LLM onto their log pipeline, point it at everything, and only notice the token spend when finance does. This is how you get the same triage quality for a fraction of the tokens, by the end you'll have a pipeline that samples, dedupes, caches and budgets before a single log line reaches the model.

Step 1: Baseline where your tokens actually go

You can't cut what you haven't measured. Before touching the pipeline, instrument it so every call to the model logs its input size, output size and which stage of the pipeline sent it.

import time, json

def log_llm_call(stage, input_tokens, output_tokens, cost_per_1k=0.003):
    record = {
        "ts": time.time(),
        "stage": stage,
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "est_cost_usd": (input_tokens + output_tokens) / 1000 * cost_per_1k,
    }
    with open("llm_token_ledger.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

Run this for a week across every stage (alert triage, root-cause summarization, report generation) and you'll almost always find one stage burning 70% of the budget on repetitive, low-value calls.

Step 2: Sample before you summarize

Most log analysis tasks don't need every event, they need a representative slice plus every anomaly. Push a sampling layer in front of the LLM: keep 100% of high-severity and unmatched-pattern logs, but sample routine noise (health checks, repeated auth successes, known-benign scans).

import random

def should_send_to_llm(log, sample_rate=0.05):
    if log["severity"] in ("critical", "high"):
        return True
    if log.get("is_known_pattern", False):
        return random.random() < sample_rate
    return True  # unknown pattern, always send

This alone typically cuts input volume by 60-90% on noisy pipelines like firewall or auth logs, without dropping the events that matter.

Step 3: Cluster repeated patterns before you send them

SOC logs are repetitive by nature: the same failed-login pattern fires a thousand times an hour. Cluster near-duplicate log lines by a normalized template (strip IPs, timestamps, user IDs) and send the LLM one representative per cluster plus a count, not all thousand.

import re, hashlib
from collections import defaultdict

def template_hash(line):
    normalized = re.sub(r"\d+\.\d+\.\d+\.\d+", "<ip>", line)
    normalized = re.sub(r"\d{2,}", "<num>", normalized)
    return hashlib.sha1(normalized.encode()).hexdigest()

clusters = defaultdict(list)
for line in raw_logs:
    clusters[template_hash(line)].append(line)

# send one sample + count per cluster instead of every line
llm_input = [
    {"sample": lines[0], "count": len(lines)}
    for lines in clusters.values()
]

Step 4: Cache LLM queries on the log pattern, not the raw text

If you've seen this template before and already asked the model "is this malicious," don't ask again. Cache the response keyed on the template hash, not the literal log line, so caching survives across IPs, timestamps and user IDs.

import shelve

def cached_analysis(template_hash_key, log_sample, llm_call_fn, ttl_hours=24):
    with shelve.open("llm_cache.db") as cache:
        entry = cache.get(template_hash_key)
        if entry and (time.time() - entry["ts"]) < ttl_hours * 3600:
            return entry["result"]
        result = llm_call_fn(log_sample)
        cache[template_hash_key] = {"ts": time.time(), "result": result}
        return result

A cache hit rate of even 40% on repetitive alert types is a 40% token cut with zero loss in coverage, since you're not caching the raw incident, you're caching the pattern's verdict.

Step 5: Batch into time windows instead of per-event calls

Calling the model once per event wastes tokens on repeated system prompts and context. Buffer events into short windows (say, 60 seconds or 500 events, whichever comes first) and send one batched request per window.

# pipeline-config.yaml
batching:
  window_seconds: 60
  max_events_per_batch: 500
  flush_on_critical: true   # critical events skip the buffer

Batching amortizes your system prompt and few-shot examples across hundreds of events instead of paying for them on every call.

Step 6: Set a per-pipeline token budget with alerts

Give each pipeline (triage, summarization, reporting) a daily token ceiling, and alert before it's blown, not after the invoice arrives.

# token-budgets.yaml
pipelines:
  alert-triage:
    daily_token_limit: 2000000
    alert_at_percent: 80
  incident-summary:
    daily_token_limit: 500000
    alert_at_percent: 80

Wire this into whatever alerting you already use for on-call, so a runaway pipeline pages someone the same way a runaway EC2 bill would.

Gotchas

• Aggressive sampling can bury a rare incident inside the discarded 95%. Always route unknown or unmatched patterns through unsampled.
• Caching by template hash can go stale if attacker behavior evolves inside a pattern you thought was benign. Keep TTLs short for anything security-relevant.
• Clustering hides count, not context. Make sure your prompt includes the count so the model doesn't treat a thousand-event spike as a single log line.
• Batching delays detection by your window size. Never buffer critical-severity events, flush those immediately.

Cut tokens with sampling, clustering and caching before the model ever sees a log line, and your SOC's AI bill stops scaling with your log volume and starts scaling with your actual incident rate, which is the number that should have driven it all along.