How to Run a Security Review for a New AI Vendor

Procurement has a signed quote, the champion has a demo that looks great, and you have until Friday to say yes or no. By the end of this guide you'll have a questionnaire to send the vendor, a scorecard that lives in your repo, a script that totals it, and a short list of red flags that stop approval. This AI vendor security review is small on purpose: it is third-party AI due diligence at the size a one-week deadline allows. It checks the vendor's terms and controls, not your feature design.

The checks follow two public sources. The NIST AI Risk Management Framework has a category, GOVERN 6, for policies on risks from third-party software, data and other supply chain issues. The OWASP Top 10 for LLM Applications supplies the vendor-facing risks: Sensitive Information Disclosure (LLM02), Supply Chain (LLM03) and Excessive Agency (LLM06).

1. Scope the AI vendor security review (day 1)

Write down three things: what data the tool will see, whether it can take actions in your systems, and who will use it. A tool that reads public docs gets a light pass. One that sees customer records gets every check below.

Put the answer in the use_case field of the scorecard so the decision is traceable later.

2. Send the vendor questionnaire and ask for documents (day 1)

The questions live in the scorecard in the next step, so send the question fields as written. Ask for the document, not the answer: the contract clause, the sub-processor list, the audit report, the settings page. A claim on a marketing page scores 0 until it shows up in a contract.

3. Write the scorecard in YAML (days 2 to 3)

Save one file per vendor, such as vendors/acme-summarizer.yaml. Acme is a fictional vendor.

vendor: Acme Summarizer          # fictional vendor for the example
use_case: Summarize support tickets (contains customer PII)
reviewed_on: 2026-10-21
next_review: 2027-04-21
threshold: 75                    # percent of the weighted maximum
# score: 0 = no, unknown, or marketing page only
#        1 = yes, with a gap (opt-out, plan-dependent, partial)
#        2 = yes, in the contract or audit report (cite it in evidence)
checks:
  - id: no-training
    question: Is our data excluded from model training and tuning by default, in the contract?
    ref: OWASP LLM02 Sensitive Information Disclosure
    weight: 5
    blocker: true
    score: 2
    evidence: DPA section 4.2
  - id: retention
    question: How long are prompts, outputs and logs kept, and can we set retention to zero or delete on request and at termination?
    ref: OWASP LLM02 Sensitive Information Disclosure
    weight: 3
    score: 1
    evidence: Admin setting exists, 30 day floor for abuse monitoring
  - id: subprocessors
    question: Who are the sub-processors, including the model provider behind the product, and do we get notice of changes?
    ref: OWASP LLM03 Supply Chain
    weight: 3
    blocker: true
    score: 2
    evidence: Sub-processor list v7, 30 days notice clause
  - id: regions
    question: In which regions are data stored and processed, and can we pin them?
    ref: NIST AI RMF GOVERN 6
    weight: 2
    score: 2
    evidence: Region selector in tenant settings
  - id: access
    question: Do we get SSO, role-based access, audit logs and scoped, revocable API keys?
    ref: NIST AI RMF GOVERN 6
    weight: 3
    score: 2
    evidence: Admin docs, audit log export tested in trial
  - id: incident-notice
    question: What is the committed breach notice window, and what is the process when their AI service fails?
    ref: NIST AI RMF GOVERN 6.2
    weight: 3
    blocker: true
    score: 2
    evidence: DPA section 9, 72 hours
  - id: certifications
    question: Which current audits cover this exact product (SOC 2 Type II, ISO/IEC 27001)?
    ref: NIST AI RMF GOVERN 6
    weight: 2
    score: 2
    evidence: SOC 2 Type II report, period ends 2026-06-30
  - id: agent-permissions
    question: If the tool can take actions in our systems, can we scope its permissions and require approval for risky ones?
    ref: OWASP LLM06 Excessive Agency
    weight: 3
    na: false                    # set true if the tool only reads and answers
    score: 1
    evidence: Per-connector scopes, no approval step yet

Each check has a weight, a score from 0 to 2 and an evidence pointer. blocker: true means a 0 ends the review whatever the total says. The ref field records why the question exists. Data retention and training terms carry the most weight because they decide where your data ends up. The incident-notice check comes from GOVERN 6.2, which asks for contingency processes for failures or incidents in high-risk third-party AI systems.

4. Total the scorecard with a script (day 4)

import sys

import yaml  # pip install pyyaml


MAX_SCORE = 2


def review(path):
    with open(path, encoding='utf-8') as f:
        card = yaml.safe_load(f)

    earned = possible = 0
    blockers, notes = [], []
    for check in card['checks']:
        if check.get('na'):
            continue
        score = check.get('score', 0)  # unanswered counts as no
        if score == MAX_SCORE and not check.get('evidence'):
            score = 1
            notes.append(check['id'] + ': scored 2 with no evidence, counted as 1')
        earned += check['weight'] * score
        possible += check['weight'] * MAX_SCORE
        if check.get('blocker') and score == 0:
            blockers.append(check['id'])

    pct = round(100 * earned / possible)
    print(card['vendor'] + ': ' + str(pct) + '% (threshold ' + str(card['threshold']) + '%)')
    for note in notes:
        print('  note: ' + note)
    if blockers:
        print('  STOP, blocker failed: ' + ', '.join(blockers))
        return 1
    if pct < card['threshold']:
        print('  BELOW THRESHOLD: close the gaps or narrow the use case')
        return 1
    print('  PASS: approve with the gaps recorded as conditions')
    return 0


if __name__ == '__main__':
    sys.exit(review(sys.argv[1]))

Save it as score.py and run it against the vendor file:

pip install pyyaml
python score.py vendors/acme-summarizer.yaml

The example scorecard prints this:

Acme Summarizer: 88% (threshold 75%)
  PASS: approve with the gaps recorded as conditions

The script counts an unanswered check as 0, downgrades a 2 with no evidence to a 1, skips checks marked na: true, and exits non-zero on a failed blocker or a total under the threshold. That exit code lets CI fail a pull request that adds a vendor file. The 75% threshold is a starting point, so tune it after your first few reviews.

5. Know the red flags that stop approval (day 5)

Any of these ends the review until the vendor fixes it in writing:

• Training terms that live only in a FAQ or blog post, not the contract.
• No data processing agreement, or one that won't name sub-processors, including the model provider behind the product.
• No committed breach notice window.
• Retention you can't configure, and no deletion at termination.
• An agent that writes to your systems with all-or-nothing permissions.
• A certification that covers a different product, or one that is in progress with no date.

If the vendor passes, merge the YAML in a pull request. That merge is your decision record. If it fails, narrow the use case (say, no customer PII), rescore, and record the narrower approval.

Gotchas

• Plan tier changes the terms. Training, retention and region options can differ between free, team and enterprise plans, so score the plan you will actually buy.
• The model provider is a sub-processor too. Ask who it is and what terms bind it.
• Audits have scope and dates. Check which product and period a SOC 2 Type II report covers.
• Blockers beat totals. A high score with a failed blocker is still a no.
• Re-run when features change. New agents or connectors change the Excessive Agency answer, so keep next_review honest instead of waiting for an incident.

A scorecard in the repo turns a rushed yes into a recorded decision, which is what you'll want when the vendor's next incident notice lands in your inbox.