World-Check(and other data suppliers) data: Validation - Understanding the Testing Process

Comsure, as a compliance consulting business, offers a risk data validation service [e.g., WorldCheck, Acuris]. We do this to provide an independent assessment of the screening service being used.

What we do is no secret and can be undertaken by a compliance team as part of their monitoring CMP. I've shared the process below, and I've used WC as my data source for this update.

World-Check[WC]  data: validation - Understanding the Testing Process

• To test whether WC is working correctly —meaning it accurately detects actual hits (true positives), misses nothing critical (low false negatives), and efficiently filters out irrelevant alerts (optimised false positives) —you can perform self-validation using a structured, iterative approach.
• This draws on industry best practices such as risk-based tuning, sandbox testing, and performance metrics.
• The goal is to balance sensitivity (detecting risks) with specificity (avoiding false positives). Average false-positive rates in AML screening hover around 90%, but optimisation can reduce them to 10-30% without increasing false negatives.
• You SHOULD  always document your tests for audit trails, as regulators emphasise explainable processes.

Step-by-Step Guide to Testing Yourself

1. Prepare Your Test Environment and Data

• Use a Sandbox or Staging Setup: World-Check supports isolated testing environments (similar to a "sandbox" in AML tools). If you don't have one, request it from LSEG support or use the "World-Check On Demand" API for batch testing without affecting live data. This lets you run "what-if" scenarios on historical snapshots.
• Build a Test Dataset:
• Positive Test Cases (True Hits): Include 20-50 known risky entities from World-Check's database, like sanctioned individuals (e.g., OFAC-listed names) or PEPs. Vary complexity: exact matches, fuzzy names (e.g., "John Smith" vs. "Jon Smyth"), and edge cases (e.g., common names with different DOBs).
• Negative Test Cases (Clean Entities): Use 100-200 legitimate profiles (e.g., internal customer data or public clean lists). Include common false positive triggers like widespread names (e.g., "Muhammad Ali" – the boxer vs. a sanctioned entity) or corporate suffixes (e.g., "Bank Ltd.").
• Mixed Dataset: Combine both for realism. Ensure data quality: clean, structured inputs (e.g., standardised formats for names and addresses) to avoid garbage-in-garbage-out issues.
• Source Ideas: Pull from OFAC/UN sanctions lists (publicly available) or LSEG's sample datasets. Aim for diversity: high-risk (e.g., international wires) vs. low-risk (e.g., domestic retail).

2. Run Initial Screening Tests

• Screen the Dataset: Input your test cases into World-Check via batch upload, API, or the UI. Apply your current false-positive algorithm settings (e.g., fuzzy matching thresholds, risk scores, and stopwords such as "Inc." or "Corp.").
• Generate Alerts: Note all hits/alerts. Categorise them:
• True Positive (TP): Correctly flagged risk.
• False Positive (FP): Legitimate entity flagged (e.g., due to name similarity).
• True Negative (TN): Clean entity not flagged.
• False Negative (FN): Risky entity missed (critical aim for 0%).
• Manual Review Sample: For 20-30% of alerts, investigate manually (e.g., cross-check with secondary data like addresses). This simulates your team's workflow.

3. Calculate Key Performance Metrics

Track these to quantify efficiency. Use a simple spreadsheet or World-Check's reporting tools for automation.

• Benchmark: If your FP rate >30%, your algorithm needs tuning (e.g., tighten thresholds for low-risk segments). Compare against baselines: pre- vs. post-optimisation runs.

4. Tune and Re-Test Iteratively

• Optimise False Positives: Based on results, adjust in World-Check:
• Risk-Based Thresholds: Segment screening (e.g., stricter fuzzy logic for high-risk geographies, looser for low-risk, like domestic retail).
• Advanced Filters: Enable secondary identifiers, keyword exclusion (e.g., ignore "Junior"), and AI-driven scoring to auto-dismiss low-confidence matches.
• Pattern Recognition: Use machine learning modules (if enabled) for self-optimisation, retrain on your false positive history.
• Data Hygiene: Purge outdated entries and standardise inputs (e.g., transliterate names).
• Run What-If Scenarios: In the sandbox, test tweaks (e.g., raise match score from 80% to 90%) on the same dataset. Re-screen and recalculate metrics.
• Address False Negatives: If any FNs appear, loosen rules selectively and re-test positives only, never compromise recall for precision.
• Iterate 3-5 Times: Each cycle: Test → Analyse → Tune → Document. Aim for 20-50% FP reduction per cycle.

5. Validate in Real-World Conditions

• Backtesting: Apply your tuned setup to 3-6 months of historical live data. Measure end-to-end: screening time, investigation hours per alert, and resolution rate.
• Stress Test: Screen high-volume batches (e.g., 10,000+ cases) to check scalability and speed.
• Human-in-the-Loop: Involve your team for anonymous reviews, track resolution time (<5-10 min/alert ideal), and feedback to refine.
• Monitor Post-Deployment: After going live, track metrics weekly. Use World-Check's analytics for trends (e.g., rising FPs in certain regions?).

Best Practices and Tips

• Start Small: Begin with 200-500 test cases to keep it manageable; scale up.
• Leverage LSEG Resources: Check World-Check's user guides for built-in testing tools (e.g., configurable filters). Contact support for custom tuning sessions; they offer regular "search tuning exercises."
• Tools for Efficiency: Export results to Excel/Pandas for metric calcs; use free ROC curve tools to visualise threshold impacts.
• Common Pitfalls to Avoid:
• Overly broad rules (e.g., exact name matches only) → High FNs.
• Ignoring context (e.g., no DOB checks) → High FPs.
• No documentation → Audit risks.
• Regulatory Alignment: This aligns with FATF/OFAC guidance, which focuses on risk-based approaches. If FPs fall short of targets but FNs rise, reassess risk appetite.
• Expected Outcomes: Well-tuned systems like World-Check can cut FPs by 60-80%, freeing teams for high-value work.

IF YOU NEED ANY ASSISTANCE WITH YOUR WORLD-CHECK OR OTHER SERVICE [E.G. ACURIS, DOW JONES, ETC.] DATA: VALIDATION AND  TESTING PROCESS, PLEASE CALL COMSURE -  DETAILS ARE HERE

Mathew Beale - Chartered FCSI

Principal & Director - Comsure Compliance Limited, Comsure Technology Limited, Comsure Mauritius

(the "Comsure Group of Companies")

mathewbeale@comsuregroup.com

www.comsuregroup.com