Blog · 2026-09-17 · Vynaris Team

AI Red Team in Your SOC: A Hosted-Model Workflow That Audits Clean

Weekly adversarial testing fails without a designed workflow. Roles, cadence, tickets, prompt libraries, logging, and cost control for AI red-teaming that satisfies auditors.

An AI red team program needs a repeatable soc red team workflow before it needs more prompts or a larger model. Most SOC teams know they should adversary-test detections, but few do it weekly because no one designed the roles, ticket template, prompt library, logging standard, or cost model. This guide turns those pieces into an operating workflow for authorized testing with hosted uncensored models. It is designed to give leadership a concrete program to review and auditors a traceable record to sample.

The operating assumption throughout is authorized testing only, on systems and data you own or are assessed to test, with the standing prohibitions every Vynaris model page states. The workflow adds process controls around that baseline. A September 2026 Chosun report may prompt more questions about safety-filter-removed models in security work, but coverage is not evidence that a model is suitable for your program. Your authorization, controls, and records are.

SOC red team workflow roles for an AI red team

Every exercise week assigns three roles to different people. The operator runs the harness, triages outputs, and files tickets. The reviewer scores borderline outputs, validates severity, and approves what ships to system owners. The authorizer owns the scope, approves target changes, and can halt the exercise. One person should not hold operator and reviewer duties in the same week. That separation is a practical quality control, not a substitute for technical review.

Rotate weekly. Rotation spreads prompt-craft skill across the team, reduces reviewer fatigue from disturbing content, and builds enough bench strength for vacations or incidents. New joiners can shadow for two cycles, then operate with a reviewer double-checking their first solo week. Record the assigned names and the authorization version in the exercise ticket so the separation is visible later.

SOC red team workflow cadence for an AI red team

Monday opens with a 30-minute planning session: pick the week's target from the backlog, confirm that scope is current, and assign roles. Tuesday and Wednesday run the harness: the operator executes prompt suites against pinned model IDs, collects outputs, and files draft tickets. Thursday is review day: the reviewer scores borderline outputs, validates severity against the rubric, and approves tickets for delivery. Friday closes with a 30-minute retro and backlog grooming: what the models found, what the harness missed, which prompts retire, and which attack paths enter the backlog.

The planning example describes roughly 300 to 800 evaluated prompts per week per operator. Treat that as a capacity-planning range, not a performance benchmark. Prompt complexity, review depth, rate limits, and the target environment can change it materially. The durable output is not volume by itself. It is a versioned prompt library and a detection backlog with enough evidence to survive staff turnover.

Tickets are the unit of work

Every finding becomes a ticket with a fixed schema: a title naming the control gap, severity from the published rubric, the exact prompts and model IDs that demonstrated it, full outputs attached, affected systems, recommended fix, owner, due date, and retest status. The ticket template belongs in the tracker, not in anyone's head, so findings survive handoffs.

Severity uses a four-level rubric the team publishes internally: critical for remotely demonstrable compromise paths, high for privilege escalation or data-access gaps with plausible exploit chains, medium for hardening gaps with theoretical paths, and low for hygiene findings. Calibrate the rubric quarterly against external references and past incidents. Treat those references as calibration inputs, not proof that your own severity is correct. A rubric without examples drifts toward alarm fatigue or complacency.

Prompt library design for detection testing

Maintain the prompt library in version control with three directories: standard suites pinned by hash, mission suites per target class, and retired prompts with retirement reasons. Every prompt carries metadata such as author, date added, last run, hit rate, and false-positive notes. Prompts that stop finding anything can move to the archive with a note; keeping dead prompts in every run consumes review time without improving coverage.

Seed the library from the open methodology in red-teaming with abliterated models and the eval script in the leaderboard post, then grow it from your own incidents. A real detection gap can become five regression prompts within a week of the postmortem if the team has enough evidence to vary the path responsibly. Published suite descriptions from commercial hosts, such as the Abliteration.ai refusal specs, can provide context for instrument behavior, but they do not establish coverage of your environment. Set a coverage-growth target for the first two quarters and revise it from observed gaps.

Prompt craft for AI red team detection testing

Detection-testing prompts differ from general red-team prompts in one important respect: every prompt must name the expected detection. A prompt that asks the model to exfiltrate data through DNS tells you little unless you record whether the DNS monitoring rule fired, at what fidelity, and with what latency. Structure every detection prompt in three parts: the attack action described precisely enough to execute within scope, the detection expected to trigger named by rule ID, and the evidence fields the rule should populate. Score ternary outcomes, detected with full context, detected with gaps, or missed, rather than binary pass-fail.

Build prompt variants along four axes: technique substitution, trying the same objective three different ways; obfuscation level, from plain to heavily encoded; timing, from burst to low-and-slow across session boundaries; and privilege level, from unauthenticated to administrative. A detection that catches only the plain variant may be a signature rather than a resilient control. Retire variants when their hit rate reaches 100 percent across two consecutive runs with no rule changes, and promote survivors into the regression set that guards against detection decay after a SIEM content update.

Metrics for an auditable AI red team program

Track five metrics weekly and review them in the Friday retro. First, prompts evaluated, which shows throughput and can reveal harness or capacity problems. Second, findings filed by severity, which shows output volume and can reveal stale prompt libraries. Third, findings closed and retest pass rate, which shows whether work is producing remediation. Fourth, cost per finding closed, computed from the receipt ledger against closed tickets. Fifth, reviewer agreement on borderline samples, which tests the scoring process itself.

Set alert thresholds rather than displaying dashboards without action. A 30 percent week-over-week drop in evaluated prompts can page the operator. Zero critical findings for a month can trigger a suite-refresh sprint, not celebration. Reviewer agreement below 70 percent can trigger rubric calibration. A doubling of cost per finding quarter over quarter can trigger a model-mix review. These are operating thresholds, not universal industry benchmarks, so document why your team chose them and review them with the authorizer.

Quarterly program review for the SOC red team workflow

Once a quarter, report to stakeholders outside the SOC: security leadership, engineering management, and a compliance observer. Keep the main review focused. Show findings closed by severity with retest pass rates; coverage growth on the control map; cost per finding closed with a ledger sample; prompt-library additions, retirements, and hit rates by class; incidents and near misses in the program itself, including reviewer-welfare notes; and the next quarter's target backlog with authorization status.

Prepare two appendices for recurring questions. The first explains model mix: which profiles served which volume, why they were selected, and what refusal and capability baselines support the choice. The second compares alternatives: what the same quarter would have cost as consultant days or platform licenses. The standing pricing and receipt exports can supply inputs, but the report should distinguish measured spend from forecasts.

Incident integration when the AI red team finds a live issue

The harness may find a live issue, or a real incident may land during an exercise. Both cases need a pre-agreed procedure. For a live finding, the operator stops the run, the reviewer validates severity, the authorizer notifies the system owner under the scope document's emergency clause, and the finding jumps the ticket queue with an incident tag. Pause testing on affected targets until the owner confirms containment or grants continued-testing permission in writing.

For a real incident during an exercise window, halt testing to avoid contaminating forensic evidence, preserve harness logs with timestamps for the incident timeline, and offer the prompt library and model access to responders under their direction. Do not let the red team decide unilaterally that its own testing should continue. Document both procedures in the scope appendix and rehearse them in the annual tabletop alongside the methodology guide scenarios.

Logging and cost control for hosted models

Log what an auditor will ask for before they ask: who ran which suite, against which model IDs, at what time, at what cost, with the scope-document version attached. Store prompts, outputs, and scores in the evidence repository with retention matching company policy. Never log raw credentials or customer data that prompts may surface. Redact at capture time with the same discipline as any assessment engagement.

Pinned, named builds with linked lineage, such as DeepSeek V4 Flash uncensored for depth work and Qwen3.6 uncensored for volume probing, make findings easier to attribute to a known artifact. Do not treat a model page as proof of suitability. Record requested and served model IDs, and use per-request receipts from the router to reconcile cost. Standing rates are on pricing.

The planning example uses 500 evaluated prompts averaging 2,000 input and 800 output tokens, or roughly 1M input and 0.4M output tokens. Keep those figures labeled as a planning example, not a benchmark. Do not repeat its fixed $2 to $5 weekly estimate as a universal current rate. Calculate the actual figure from the model, token mix, and receipts used by your program. Set a fixed monthly credit allocation with alerts at 75 percent, then report cost per finding closed rather than cost per prompt. Integration details for the tooling live in the docs.

Frequently asked questions

How is an AI red team workflow different from penetration testing?

Penetration testing is point-in-time, human-led, and deep. This workflow is continuous, model-assisted, and broad: it finds detection gaps and control weaknesses between formal assessments. The two complement each other, and neither replaces the other.

What if reviewers burn out on adversarial content?

Rotate roles weekly, cap review sessions at two hours with breaks, provide an opt-out without career penalty, and keep prohibited categories out of every suite permanently. Reviewer welfare is a program-sustainability requirement. Make it part of the authorizer's review, not an informal favor.

Can small teams run this with one security engineer?

Yes, with modified separation. The engineer can operate while a peer from engineering or an external reviewer scores borderline outputs. The roles matter more than headcount, and even a two-person rotation is stronger than an unreviewed solo practice. Record the exception and its compensating review.

How do we measure whether the program works?

Track findings closed per quarter, retest pass rates, mean time from finding to fix, and detection coverage growth on your control map. Rising prompt volume with flat findings closed means the harness needs new attack paths or better triage, not automatically more runs.

Where do findings go if they affect production?

To system owners first, under the remediation windows in the scope document, with severity driving the timeline. General channels get aggregated statistics after remediation, never live vulnerability detail. Preserve the authorization and evidence trail for the incident record.