P6· L-G· Bands Foundational → Operational

· VEN-01

Vendor Evaluation Operating Methodology

Vendor Evaluation Operating Methodology is the canonical pre-procurement instrument for assessing Legal AI vendors. Anchored to Pillar P6 (Vendor, Procurement & Technology) on the Governance Layer, the Module advances the function from Band 2 Operational to Band 4 Optimised on the Defensibility and Sophistication lenses. Mandatory Pass/Fail gates (data protection, security, regulatory, financial viability) precede five-dimension weighted scoring mapped to Risk Taxonomy 2026. No pilot proceeds without an evidence bundle that withstands regulatory, client, and internal audit scrutiny. Methodology v2026.1.

Foundational

Lift 2 · Guided

Per-engagement

12-week evaluation cycle per vendor engagement, with 2–4 hours per week from core team

Methodology v2026.1·Verified 23 May 2026·Reviewed 23 May 2026

Executive Summary

The Weighted Vendor Evaluation Scorecard (VEN-01) is the firm’s canonical pre-procurement tool for assessing legal AI vendors. It combines mandatory Pass/Fail gates with a five-dimension weighted scoring model mapped to the Risk Taxonomy 2026. The Scorecard ensures that no AI pilot proceeds without cleared data protection, security, regulatory, financial viability, and reference checks. Technical performance, governance and compliance, business viability, integration and usability, and cost and commercial terms are scored using a 1–5 scale with defined decision thresholds. Agentic AI capabilities trigger additional governance and logging requirements. VEN-01 connects directly to USE-01 use case prioritisation, USE-02 pilot design, GOV-03 risk register, DAT-03 DPA checklist, and the AI Bill of Materials (AI BoM). The completed Scorecard, together with supporting artefacts, forms a Defensibility Posture Statement (DPS) evidence bundle demonstrating structured, risk-aware vendor selection that can withstand regulatory, client, and internal audit scrutiny.

Defensibility Evidence Produced

Pass/Fail gate completion and Risk Taxonomy 2026 cross-walked Governance and Compliance scores constitute the vendor selection DPS evidence bundle for the Defensibility lens; mandatory AI BoM registration post-selection provides ongoing Defensibility lens evidence that the tool is approved, documented, and governed

Elements:

Methodology transparencyEvidence framework

Purpose

The Weighted Vendor Evaluation Scorecard (VEN-01) is the canonical pre-procurement assessment instrument for selecting legal AI vendors. It provides a structured, defensible scoring methodology across five weighted dimensions, with mandatory Pass/Fail gates that operate independently of weighted scores. Completion of this Scorecard is a required input to USE-02 pilot authorisation; no pilot proceeds with a vendor that has not cleared Pass/Fail thresholds.

Operating cadence: Per-engagement — completed for each AI vendor under active procurement consideration.

Owner: Legal Operations, STR-07 AI Task Force, Technology & Procurement.

When to Use This Module

Before any AI vendor selection decision, regardless of contract value
When evaluating vendors for a specific use case identified in USE-01
When refreshing an existing vendor relationship (annual re-evaluation)
When a Shadow AI audit (Risk Taxonomy 2026, Class 6) surfaces an unapproved tool requiring retrospective assessment

AI Bill of Materials — Pre-Evaluation Requirement (Metric 0)

Before beginning weighted scoring, confirm the following AI BoM Pre-Check is complete:

| AI BoM Pre-Check | Status |

|—|—|

| Vendor not already registered in AI BoM as declined or decommissioned | Confirm |

| AI BoM slot approved for this use-case category by STR-07 AI Task Force | Confirm |

| Intended use case aligns with a ranked opportunity in USE-01 | Confirm |

If the vendor clears all Pass/Fail criteria and weighted scoring, AI BoM registration is the mandatory post-selection step before any pilot commences. The AI BoM entry must record: vendor name, product version, approved use-case scope, data classification handled, contract DPA reference, and Agentic Tier designation.

Section 1: Evaluation Framework

Dimension Weights and Risk Taxonomy 2026 Mapping

| Dimension | Default Weight | Risk Taxonomy 2026 Class(es) |

|—|—|—|

| 1. Technical Performance | 30% | Class 1: Hallucination and accuracy; Class 3: Bias and fairness; Class 9: Operational resilience |

| 2. Governance and Compliance | 25% | Class 2: Privilege and confidentiality; Class 4: Privacy and data protection; Class 7: Regulatory compliance drift; Class 8: IP and licensing |

| 3. Business Viability | 20% | Class 5: Supply chain and vendor dependency |

| 4. Integration and Usability | 15% | Class 9: Operational resilience; Class 6: Shadow AI (low adoption drives Shadow AI) |

| 5. Cost and Commercial | 10% | Class 5: Supply chain and vendor dependency |

Agentic Tier Supplement

If the vendor’s product includes autonomous AI agents, add the following criteria to Dimensions 1 and 2 before scoring:

| Agentic Tier Criterion | Dimension | Status |

|—|—|—|

| Kill-switch and human override capability | Dimension 2 (Governance) | Mandatory |

| Intervention frequency logging | Dimension 1 (Technical) | Mandatory |

| Autonomous action scope documentation | Dimension 2 (Governance) | Mandatory |

| Audit trail for agentic decisions | Dimension 2 (Governance) | Mandatory |

Dimension 1: Technical Performance (30%)

| Subcriteria | Weight | Risk Taxonomy Class |

|—|—|—|

| Accuracy and Reliability | 35% | Class 1: Hallucination and accuracy |

| RAG and Knowledge Integration | 25% | Class 1 |

| Performance and Scalability | 25% | Class 9: Operational resilience |

| Model Sophistication | 15% | Class 3: Bias and fairness |

Accuracy targets: Citation accuracy >95%; Hallucination rate <1%; Legal reasoning consistency verified through structured testing.

Dimension 2: Governance and Compliance (25%)

Risk Taxonomy 2026 cross-walk for all Governance and Compliance sub-criteria:

| Sub-criterion | Risk Taxonomy 2026 Class | Evaluation Requirement |

|—|—|—|

| Security Certifications (SOC 2 Type II, ISO 27001) | Class 9: Operational resilience | Current certification required; expired = Pass/Fail failure |

| Data Protection — no training on client data | Class 2: Privilege and confidentiality; Class 4: Privacy and data protection | Contractual and technical safeguards; DPA required per DAT-03 |

| Regulatory Alignment (EU AI Act, GDPR, ABA Rules 1.6/1.1/5.3) | Class 7: Regulatory compliance drift | Compliance mapping across all applicable frameworks |

| Auditability and Transparency | Class 1: Hallucination and accuracy | Complete logging of system activities and decisions |

| Bias Detection and Reporting | Class 3: Bias and fairness | Systematic bias monitoring and reporting required |

| IP and Licensing | Class 8: IP and licensing | AI-generated output ownership clearly defined in contract |

GOV-03 Risk Register feed: Governance and Compliance scores below 3.0 in any subcriteria must be logged as GOV-03 Risk Register entries under the corresponding Risk Taxonomy 2026 class before proceeding.

Operational Signals

ven-01.vendor-evaluated

→ Defensibility Posture Statement

Each completed evaluation writes a DE-3 Evidence framework record per vendor.

Per Module run

ven-01.tier-input

→ Vendor Index

Anonymised tier-movement input to Vendor Index quarterly refresh.

Quarterly

ven-01.pass-fail-rate

→ Annual Legal AI OS Index

Pre-procurement Pass/Fail rate — Annual Legal AI OS Index vendor-discipline signal.

Annual

Recommended Stakeholders

Owner

Head of Legal Operations

Approvers

General Counsel
CIO / CISO
Risk & Compliance

Contributors

Procurement
Engineering / IT

Informed

AI Task Force
Finance

Inputs · Outputs

Inputs

· USE-01 ranked use case
· STR-07 AI Task Force AI BoM slot approval
· DAT-03 DPA requirements checklist
· Previous vendor assessment records (if any)

Outputs

· Pass/Fail verification record
· Weighted dimension scores with evidence rationale
· GOV-03 Risk Register entries for scores below 3.0
· AI BoM registration record (post-selection)
· Final vendor selection recommendation for STR-07 approval

Framework Crosswalk

EU AI Act

European Union

Governance and Compliance dimension requires explicit mapping of vendor controls and roadmap to EU AI Act obligations for relevant risk categories.

GDPR

European Union

Data protection and privacy sub-criteria test for lawful processing, data minimisation, purpose limitation, and cross-border transfer safeguards.

SOC 2 Type II

AICPA

Security baseline Pass/Fail gate requires current SOC 2 Type II or equivalent as evidence of operational resilience controls.

ISO/IEC 27001

ISO

Security certification sub-criterion recognises ISO 27001 as equivalent evidence of information security management maturity.

ABA Model Rules 1.1, 1.6, 5.3

American Bar Association

Regulatory alignment checks that vendor use supports lawyer competence, confidentiality, and supervision of non-lawyer assistance in AI contexts.

NIST AI Risk Management Framework

NIST

Technical performance, bias monitoring, and governance controls are aligned to NIST AI RMF functions for mapping, measuring, and managing AI risk.

Operational Artefacts

VEN-01 Weighted Vendor Evaluation Scorecard (working template)
xlsx · v2026.1
Gated
VEN-01 Evaluation Playbook and Instructions
pdf · v2026.1
Gated
VEN-01 Pass/Fail Gate Checklist
checklist · v2026.1

Sequence

Run before

GOV-01
Defensible AI Governance Framework
Establish the governance structure, policy suite, and risk register that make Legal AI defensible to boards and regulators.

Run next

Diagnostic Relevance

Running Vendor Evaluation Operating Methodology strengthens the Defensibility lens — expected Band progression: Operational → Optimised.

Confidence: high

Key Takeaways

Mandatory Pass/Fail gates override all weighted scores — a single FAIL triggers automatic disqualification regardless of technical capabilities
All five evaluation dimensions are cross-mapped to Risk Taxonomy 2026; Governance and Compliance subcriteria below 3.0 generate GOV-03 Risk Register entries
Agentic Tier AI products require four supplementary criteria: kill-switch, intervention logging, scope documentation, and agentic decision audit trail
AI BoM registration is mandatory post-selection before any pilot under USE-02 commences
STR-07 AI Task Force must approve final vendor selection at Stage 5 before contract execution

Run this Module

Operational artefacts available to Practitioner Membership members. Methodology v2026.1.

View Membership

Targeting

Audience

GC / CLOLegal OperationsRisk & Compliance

Strengthens

Defensibility lensAdoption lens

Module Details

Format: Module
Difficulty: Foundational
Pillar: P6
Layer: G · Governance
Owner: Head of Legal Operations
Access: Practitioner Membership
Certification: Practitioner

Maturity Bands

FoundationalOperational

Risk Classes Mitigated

Operated within

Advisory pathway

Legal AI OS for Legal Operations →

In the Ecosystem

P6 · Vendor →Module Library →Find the Right Diagnostic →

Related Modules

View Module Library →

VEN-02

Vendor RFP Methodology

Structured RFP template for procuring legal AI solutions with defensible, risk-aware evaluation.

DAT-03

Vendor Data Protection Obligations

Canonical checklist for reviewing and negotiating AI vendor data protection agreements.

STR-03

Risk Matrix — Use Case × Risk Taxonomy 2026 × Likelihood

Systematic risk-scoring framework for AI legal use cases aligned to Risk Taxonomy 2026.

Referenced in Intelligence

Where this Module is anchored

Canonical Vocabulary

Terms this Module anchors

Vendor Index

Advanta's annual publication ranking legal AI vendors across standardised evaluation dimensions — capability, Defensible AI posture, commercial terms, and ecosystem fit. Vendor Index entries are produced independently; vendors do not pre-review their entries. Editorial independence from vendors is binding. For clarity, 'vendor rankings' and 'vendor scorecard' refer to this same publication.

Defensible AI

The practice of designing, deploying, and governing AI systems that withstand regulatory scrutiny, board challenge, and client examination. Defensible AI requires documented evidence of governance, not stated intent alone. Advanta operationalises defensibility for legal AI: it builds on and extends compliance- and responsible-AI approaches by making governance testable and audit-ready.

AI BoM

The procurement-grade inventory of AI components — models, datasets, tools, APIs, and vendors — used in a legal function's AI stack. The AI BoM (AI Bill of Materials) is the foundation for vendor due diligence, contract governance, and regulatory disclosure. It is the procurement-grade variant of the broader AI Inventory concept. 'AI Bill of Materials' is acceptable on first use only.

Where this Module lives

Vendor Evaluation Operating Methodology produces the Defensibility Element 3 (Evidence framework) per-vendor record that anchors the Defensibility Posture Statement vendor section. Each completed evaluation contributes anonymised inputs to the public Vendor Index without revealing the function’s confidential scorecard. Without this Module, vendor onboarding proceeds without canonical evidence, AI BoM registration lacks a defensible basis, and the DPS sits without DE-3 evidence.

Advisory

When this Module sits inside a Programme.

Modules are operated in-house by GC and Legal Operations teams. When the capability transformation is multi-Pillar — or when the regulator timeline tightens — Advanta operates the canonical Module sequence as a Programme.

View Engagement Models Find the Right Diagnostic

← Back to Module Library