Rule engines are back: combining predictable logic with probabilistic LLMs

If you are asking how to combine a rules engine with an LLM, the short answer is this: let the model turn messy input into typed facts, and let rules turn typed facts into decisions. The model reads, extracts, classifies, and drafts. The rules decide, compute, enforce, and route. Put a strict schema between the two, and never let a decision that matters be made inside a prompt.

I spent 28 years building low-latency trading systems at major financial institutions, and I hold US Patent 8,301,798 for a configuration-driven rule and integration engine. I have worked in AI/ML since 2018. So I have watched rule engines go out of fashion, and I am now watching LLMs bring them back, for a reason that has nothing to do with nostalgia. The two technologies fail in opposite places, which makes them unusually good partners.

Why rule engines fell out of fashion

The business-rules era of the 2000s made a big promise: take policy out of code, put it in a rules repository, and let business users maintain it. The products were serious pieces of engineering. The outcome was often disappointing, for three reasons.

Rule bases sprawled. Thousands of rules with implicit interactions, where firing order mattered and nobody could say with confidence what the system would do for a given input without running it.

The tooling was heavy. Proprietary authoring environments, specialist skills, long release cycles. The "business user edits the rule" story frequently ended with a developer editing the rule after a ticket.

Rules need structured input, and the world does not supply it. This is the one that mattered most. A rule like "if the invoice exceeds the PO amount by more than 5%, hold it" is trivial to write. Getting the invoice amount, the PO number, and the vendor identity out of a scanned PDF was the expensive part. Rule engines sat behind a wall of data entry, custom parsers, and brittle templates.

Then machine learning arrived and absorbed the perception-style problems that people had been attacking with hand-written heuristics. Rules came to look like the thing you wrote before you had a model.

Why probabilistic models bring them back

LLMs removed the wall. A model can read the scanned invoice, the email thread, the contract clause, and the call transcript, and hand you structured fields. The unstructured-input problem that limited rule engines for two decades is now largely solved.

But an LLM is a poor final authority. It is probabilistic: the same input can produce different output. Its arithmetic is unreliable. Its reasoning is not inspectable in a form an auditor will accept. And when policy lives in a prompt, a policy change is a prompt change, which alters behaviour in ways you cannot enumerate.

In trading systems, the checks that stand between an order and the market are deterministic. Nobody would accept "the limit check is usually right." Most businesses have an equivalent class of decisions: approvals, eligibility, pricing, tax treatment, routing, anything a regulator or a customer can challenge. Those decisions need the same answer every time, a stated reason, and a change history.

So the division of labour looks like this:

Four architectures

These are not mutually exclusive. Most production systems I would sign off on use two or three of them together.

1. Rules as pre-filter and router

Rules run first, before any model call. They decide whether the model is needed at all, which model or prompt handles the request, and what the model is allowed to see.

Typical uses: reject malformed or out-of-scope requests, send known document types to a cheap template parser and only the unknown ones to the model, strip or mask fields the model must not receive, and enforce per-tenant permissions. This is the cheapest architecture to add and it cuts cost and risk immediately, because the most predictable traffic never touches the probabilistic component. Our piece on when LLMs are the right tool covers the same instinct at the project level.

2. Rules as post-validator and guardrail

The model produces output; rules check it before anything downstream sees it. Does the JSON validate against the schema? Do the line items sum to the stated total? Is the date in a plausible range? Does every extracted value appear in the source document? Does the drafted reply contain a commitment the company does not make?

A failed check has a defined consequence: retry with the error fed back, fall back to a stricter prompt, or send to a human. What it never does is pass through silently.

3. The model as feature extractor feeding a rule engine

This is the core pattern. The model's only job is to populate a typed schema. The rule engine consumes that schema, joins it with reference data from your systems of record, and makes the decision. The schema is a contract: the model cannot add fields, the rules cannot see free text.

The benefit is that the two halves can be tested, versioned, and replaced independently. Swap the model and the rules do not change. Change the policy and the extraction does not change.

4. The model as rule author, with human review and tests

Policy documents are prose. Turning a forty-page policy into a rule set is tedious, and a model is good at producing a first draft: "read this policy, propose rules in our format, cite the paragraph each rule comes from."

The draft is a proposal, not a deployment. A domain expert reviews it, the rules are stored as data under version control, and a test suite of worked cases must pass before the new version goes live. The model saves authoring time; it never gets write access to production policy.

Configuration-driven rules and decision tables

The lesson I took from the first rule-engine era is that rules should be data, not code, and that the format should be simple enough to read without training.

Rules as data means a policy change is a reviewed change to a configuration file or table, with a version number, an author, and an effective date. It does not require a build or a redeploy of the application, and it can be rolled back on its own.

Decision tables are the most durable format I know. Each row is a rule, each column is a condition or an outcome, and a domain expert can check completeness by looking at it. A hit policy states what happens when more than one row matches: first match wins, or collect all matches. State it explicitly; implicit ordering is what made the old rule bases unmanageable.

Keep the operator vocabulary small: equals, comparison, set membership, null checks. The moment rules can call arbitrary code, you have rebuilt the application inside the rule engine, and lost the readability that justified it.

A worked example

Invoice approval. The model extracts; reference data comes from the vendor master and the PO system; rules decide.

The rule set, as versioned configuration:

ruleset: invoice-approval
version: "2027.02.1"
hit_policy: first
rules:
  - id: INV-001
    when:
      - { field: po_number, op: is_null }
      - { field: vendor_requires_po, op: eq, value: true }
    then: { outcome: reject, reason: "Vendor requires a PO and none was found" }

  - id: INV-002
    when:
      - { field: over_po_pct, op: gt, value: 5 }
    then: { outcome: hold, reason: "Invoice exceeds PO amount by more than 5%" }

  - id: INV-003
    when:
      - { field: amount, op: gte, value: 10000 }
    then: { outcome: manager_approval, reason: "Amount at or above approval threshold" }

  - id: INV-004
    when: []
    then: { outcome: auto_approve, reason: "Within PO tolerance and below threshold" }

The extraction contract and the evaluator:

from datetime import date
from decimal import Decimal
from pydantic import BaseModel, Field, ValidationError

class ExtractedInvoice(BaseModel):
    """The only thing the model is allowed to produce."""
    vendor_name: str
    po_number: str | None
    amount: Decimal = Field(gt=0)
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    invoice_date: date

OPS = {
    "eq": lambda a, b: a == b,
    "gt": lambda a, b: a is not None and a > b,
    "gte": lambda a, b: a is not None and a >= b,
    "in": lambda a, b: a in b,
    "is_null": lambda a, _: a is None,
}

def evaluate(facts: dict, ruleset: dict) -> dict:
    for rule in ruleset["rules"]:
        if all(OPS[c["op"]](facts[c["field"]], c.get("value")) for c in rule["when"]):
            return {
                "outcome": rule["then"]["outcome"],
                "reason": rule["then"]["reason"],
                "rule_id": rule["id"],
                "ruleset_version": ruleset["version"],
                "inputs": facts,
            }
    raise RuntimeError("Rule set has no default row")

def decide(model_json: str, source_text: str, ruleset: dict) -> dict:
    try:
        inv = ExtractedInvoice.model_validate_json(model_json)
    except ValidationError as err:
        return {"outcome": "human_review", "reason": f"Schema failure: {err.error_count()} errors"}

    if not grounded(inv, source_text):          # every value must appear in the document
        return {"outcome": "human_review", "reason": "Extracted value not found in source"}

    vendor = vendor_master.lookup(inv.vendor_name)   # system of record, not the model
    po = po_system.get(inv.po_number) if inv.po_number else None

    facts = inv.model_dump() | {
        "vendor_requires_po": vendor.requires_po,
        "over_po_pct": pct_over(inv.amount, po.amount) if po else None,   # computed in code
    }
    return evaluate(facts, ruleset)

Notice what the model does not do. It does not know the approval threshold. It does not compute the percentage over PO. It does not know whether the vendor requires a PO. It reads a document and fills in five fields.

Auditability

Every decision record carries the rule id, the rule set version, and the exact inputs the rule saw. Store alongside it the source document hash, the model identifier, the prompt version, and the raw model output.

With that, the question "why was this invoice held in March?" has a complete answer: rule INV-002, version 2027.02.1, because over_po_pct was 7.3, computed from an extracted amount and a PO record, and here is the document the amount came from. If the extraction was wrong, you can see that it was the extraction. If the policy was wrong, you can see which version and who approved it. Our article on audit trails for LLM systems covers the logging side in more depth.

Try producing that answer from a system where the policy was a paragraph in a prompt.

Testing the two halves differently

Rules get unit tests. They are deterministic, so ordinary table-driven tests work: given these facts, expect this outcome and this rule id. Include boundary cases on every threshold, and a completeness check that every combination of inputs reaches some row. Run them on every change to the rule set, and keep a regression pack of real historical decisions to replay against a proposed version before it goes live.

Extraction gets an evaluation set. A few hundred labelled documents with the correct field values, scored per field, re-run on every model or prompt change. This is a statistical measure, not a pass or fail, and it should be tracked over time as described in the three measurements we insist on.

Keeping the two separate tells you where a regression came from, which a single end-to-end accuracy number never will.

Failure handling

Do not rely on the model's self-reported confidence; it is not calibrated well enough to gate decisions. Derive confidence from checks you control:

• Does the output validate against the schema?
• Does each extracted value appear verbatim in the source?
• Do computed relationships hold, such as line items summing to the total?
• Do two independent extraction passes agree on every decision-relevant field?

Any failure routes the item to a human queue with the document, the extracted fields, and the failed check displayed together. The human's correction becomes a new labelled example for the evaluation set. The rule engine only ever sees facts that passed.

Anti-patterns

Policy in the prompt. "Approve if the amount is under 10,000 unless the vendor is new" is a rule. In a prompt it is unversioned, untested, and applied probabilistically.

Arithmetic in the model. Totals, percentages, date differences, tax, currency conversion. Extract the operands; compute in code.

Letting the model see the threshold. If the model knows the cutoff, it can be nudged toward it, by the document's wording or by a user. Keep decision boundaries on the rules side of the schema.

A free-text field in the contract. A "notes" field that the rules then parse reintroduces everything the schema was meant to exclude.

Rules that call the model. Once a rule condition is "ask the LLM whether this looks suspicious," the rule engine is no longer deterministic. If you need a model judgment, extract it as a typed, enumerated fact upstream, and test it like any other extraction.

No default row. Every rule set needs an explicit outcome for "nothing matched," and it should almost always be human review.

Where to start

Take one decision your system currently makes inside a prompt. Write down the facts the decision depends on, define them as a schema, and move the decision into a small table with a version number. It is usually a few days of work, and it is the point at which the system becomes something you can defend. For the mechanics of making the extraction side repeatable, see making LLM output deterministic enough for finance and tax work.