How to Build and Test an AI Workflow Step by Step

Check AI workflow handoffs with a six-record answer key, a runnable validator and regression tests for dropped, duplicated and mislabeled records.

By DailySignal.app · Published 2026-02-01 · Updated 2026-11-04

Two colleagues arranging illustrated process cards beside a notebook

Before you connect an AI workflow to the customer inbox, give it a few messages whose expected answers you can check by hand. Include an ambiguous message, a blank record and a duplicate. A tidy summary is useful only if the right evidence survives every step.

Start with one small handoff: turn feedback into a draft issue summary for a reviewer. The example below begins with three messages, expands to six test records and shows how to spot a failure before adding more automation.

Follow three messages through the process

In this fictional feedback sample, message A says 'My invoice was charged twice.' Message B says 'I can't find the search box.' Message C says 'The search results don't show older projects.'

A useful first output classifies A as a billing issue and B/C as search issues, while keeping each message attached. The summary says: 'Two of three messages concern search; one concerns billing. The billing message needs individual review. We don't know whether this sample represents all customers.'

That is a completed handoff. It preserves the evidence and separates a count from a claim about the whole customer base. A statement that most customers dislike search would go beyond these inputs.

Define what passes between steps

For each message, keep an ID, original text, category, reason and unresolved question. The summary step should read those records, not an earlier narrative that may have lost a qualification.

Allow an unresolved category. A message saying 'It happened again' doesn't establish billing or search without context. The process should route it for clarification instead of forcing it into the most common category.

Place checks where mistakes would spread

Before summarizing, inspect a few classifications against the original messages. Before reporting, check the counts and keep the sample size visible. Before any customer action, confirm the account and the exact authorized response through your normal process.

A billing classification isn't permission to issue a refund or change a record. Start with a draft report for the responsible reviewer. You can evaluate further automation after the analysis itself is reliable.

Build a small test you can actually grade

Keep the original three messages, then add D: “It happened again,” E: a blank message, and F: a second copy of message B with the same message ID. These are deliberate test cases. Decide the expected behavior before asking the AI to process them.

  • A should be billing, with the original complaint preserved and a human review flag. It should not say that a duplicate charge has been verified.
  • B and C should be search, with their different complaints still visible.
  • D should remain unresolved because it lacks enough context.
  • E should be marked blank and excluded from the issue summary.
  • F should be flagged as a repeated record and excluded from the unique-message count. Two different customers saying the same thing would be a different case; matching wording alone is not a reason to delete feedback.

Under these rules, the summary has four unique nonblank messages: two search, one billing and one unresolved. Search is two of four, or 50%, of that selected sample. The blank and repeated record remain in the processing log. If your report instead uses only the three categorized messages, say so explicitly; two of three answers a different question.

This is why a small test is useful. A polished summary that says “most customers have search problems” fails even if every sentence reads well. It has changed both the population and the meaning of the evidence.

Give the reviewer an answer key

Try this request with the fictional inputs above:

Classify each record as billing, search or unresolved. Preserve its ID and original text. Give a short reason and flag anything needing human review. Treat blank records and repeated IDs separately, keeping an exclusion reason. Summarize unique nonblank messages, include unresolved messages in that total, and show the count behind every percentage. Do not verify a billing event, infer missing context or take customer action.

Review the individual records before reading the summary. Did A become a confirmed payment error? Did D get forced into billing? Did the second B get counted twice? Then recalculate the totals yourself. A table that adds up can still rest on a wrong classification.

A completed review note might read: “All four usable messages retained. A, B and C classified as expected. D incorrectly labeled billing. Blank and duplicate excluded correctly. Summary denominator correct, but billing count wrong. Repair the unresolved rule and rerun all six records.” This is an illustrative result, not a test of a particular AI product.

Keep the input set and scoring rules unchanged when comparing the revision. Otherwise, you will not know whether the workflow improved or the test simply became easier. After repairing it, add a few new examples the prompt was not written around. Passing six familiar records is a starting point, not a measured accuracy rate for your customer inbox.

Use a small answer-key check before reading the prose

For this exact teaching exercise, the Python checker below checks that all six output IDs survive and that their status and category match the answer key. It does not classify messages, verify a billing event or evaluate a real customer inbox. Use it alongside the original-text review, not instead of that review.

Save the code as feedback-answer-check.py and the output you want to check as a JSON list. Each object needs id, status and category. Use keep for A through D, blank for E and duplicate for F. E and F use JSON null for category. F is the separate processing record for the repeated B input, so its output ID stays F.

#!/usr/bin/env python3
"""Grade the article's six synthetic feedback records, not a real inbox."""
import json
import sys

EXPECTED = {
    'A': ('keep', 'billing'),
    'B': ('keep', 'search'),
    'C': ('keep', 'search'),
    'D': ('keep', 'unresolved'),
    'E': ('blank', None),
    'F': ('duplicate', None),
}

def grade(rows):
    if not isinstance(rows, list):
        raise ValueError('Expected a JSON list of six record objects.')
    found = {}
    for row in rows:
        if not isinstance(row, dict) or row.get('id') not in EXPECTED:
            raise ValueError('Unknown or missing record ID.')
        record_id = row['id']
        if record_id in found:
            raise ValueError('Repeated output ID: ' + record_id)
        found[record_id] = (row.get('status'), row.get('category'))
    if set(found) != set(EXPECTED):
        raise ValueError('Missing output IDs: ' + ', '.join(sorted(set(EXPECTED) - set(found))))
    for record_id, expected in EXPECTED.items():
        if found[record_id] != expected:
            raise ValueError('Answer-key mismatch: ' + record_id)
    return {'records': 6, 'unique_nonblank': 4, 'search': 2,
            'billing': 1, 'unresolved': 1, 'search_percent': 50}

if __name__ == '__main__':
    try:
        with open(sys.argv[1], encoding='utf-8') as source:
            result = grade(json.load(source))
        print(json.dumps(result, indent=2))
    except (IndexError, OSError, ValueError) as error:
        print('FAIL: ' + str(error), file=sys.stderr)
        sys.exit(1)

Run python3 feedback-answer-check.py output.json. A matching six-record answer returns four unique nonblank messages and a 50% search share. We ran this checker against synthetic fixtures: the complete answer key passed; dropping C, changing D to billing, counting F as another search message, and repeating an output ID each failed. Those are checks of this small script and its fixtures, not a benchmark of an AI model.

If your real task uses different categories or exclusion rules, write its answer key before adapting the check. A correct category with an invented quotation still needs human correction; this checker does not inspect the source wording.

Repeat the run and check what actually happened

Run the same test more than once and record the tool or model, date and workflow version. Look for cases that change category between runs. A single clean run can hide an unstable decision that sends the next ambiguous message down the wrong path.

For workflows that use tools, inspect the final destination too. “Report saved” in an AI response does not establish that the file exists, opens and contains the approved numbers. Keep this first test to a draft in a designated test location. Check that no customer reply, refund or production record change occurred.

Anthropic's guide to evaluating AI agents distinguishes a test case, repeated attempts and the final state of the environment. The practical application here is simple: keep the expected answer beside each case, repeat the run and inspect the resulting artifact. This six-record exercise is an editorial example, not an Anthropic benchmark or a required sample size.

If a source is unavailable, mark the run incomplete and name the missing source. You can review the supported subset, but do not label it a complete report or quietly substitute an older file.

Compare a revision without moving the goalposts

Suppose the first version sorts the six records correctly except for D, “It happened again.” You add a rule that missing context must stay unresolved. Before celebrating the cleaner answer, put the original and revised requests through the same six records, with the same answer key and tool settings. Save both requests so you can see exactly what changed.

Here is a completed fictional comparison note: “Version 1 incorrectly assigned D to billing in two of three attempts. Version 2 kept D unresolved in all three attempts, but one attempt dropped C from the summary. Both versions excluded the blank and repeated record correctly. Keep Version 1 as the working baseline until Version 2 also preserves every usable message.”

The new rule fixed the problem it was written for, but the missing message creates another problem. A single overall score would hide the choice the reviewer actually has to make. Record classification, retained evidence, exclusions and summary arithmetic separately. Treat a dropped usable message as a failed run even if the remaining percentages add up.

For each attempt, keep the input file, request version, tool or model label, settings you can control, output and review note together. If the tool changes during the comparison, record that change and avoid attributing the difference entirely to your new wording. You can still compare the resulting work, but you have changed more than one ingredient.

Next, try three fresh fictional records: G says “The search box disappears on mobile,” H says “My invoice total is higher than expected,” and I says “Thanks, that fixed it.” Decide their expected treatment before running them. G belongs with search. H needs billing review without claiming a charge error. I needs an unresolved or non-issue treatment defined by your workflow; it should not become another complaint just to fit the categories.

Keep these fresh checks separate from the six familiar records. If the revision fails them, write down the failure and repair the rule before trying another new set. Once a case has shaped the request, it is useful for catching regressions, but it no longer tells you how the request handles an unfamiliar example.

A useful release note would say: “Version 2 retained all usable records across the repeated baseline checks, preserved the exclusions and passed the fresh cases under our answer key. Approved for the next reviewed draft report.” It would not claim that nine fictional inputs establish accuracy for the whole inbox. The next step is a supervised trial on an appropriate real sample, with the same record-level review.

Document one repeatable run

Save the approved inputs, answer key, reusable request, expected fields, reviewer and failure instructions. Have a colleague run a new sample and explain any place they needed help. Measure preparation, review and corrections together before calling the workflow a saving.

Use the decision-brief guide to choose the next expansion. First make one handoff dependable; add another step only when it removes useful work without hiding the evidence.

When a workflow starts using tools, practice defining its limits in the free course on AI agents.