Service

Training Data Forensics

Analysis to determine if specific data was used for AI model training.1

Download the CFAA Evidentiary Checklist

Analysis of Training Data

For copyright and privacy cases involving AI, establishing whether a model ingested specific data requires a defensible statistical workflow anchored in peer-reviewed techniques.2910

Training Data Verification

We deploy membership inference, loss landscape analysis, and gradient leakage tests to quantify whether a dataset influenced model parameters.3

Web Provenance Tracing

We reconstruct ingestion pipelines via crawl artifacts, robots.txt differentials, and dataset manifests to establish access pathways.47

Ongoing Model Monitoring

Post-order, we monitor new model releases with change-point detection to flag reuse or novel ingestion of restricted corpora.69b

How We Investigate Suspect Training Data

The workflow implements dataset inference as a formal hypothesis test: we engineer matched cohorts, extract high-dimensional probe signals from the model, learn a distribution-specific decision surface, and quantify attribution strength with controlled error bounds.5

1. Controlled Cohort Design

Input corpora are shingled, deduplicated, and stratified by metadata (timestamp, domain, format) before being partitioned into suspect and validation cohorts A/B. We verify distributional alignment via Jensen–Shannon distance and n-gram frequency profiles, ensuring the inference task is not confounded by temporal or topical drift.8

2. Multi-Signal Probing

We execute a probe suite spanning token-level log-likelihood deltas, Min-k% probability aggregates, perturbation Jacobians, zlib compression ratios, and reference-model KL divergences. Signals are sampled across checkpoints and temperature settings to factor out instruction-tuning artifacts and decoder variance.

3. Signal Calibration

Probe outputs are standardized, then an elastic-net logistic model is trained on the A-split holdout to learn probe weights specific to the distribution. We perform nested cross-validation and permutation importance to isolate signals with non-spurious predictive contribution.

4. Statistical Attribution

Calibrated scores for the B-split cohorts feed a Welch t-test with Benjamini–Hochberg correction across datasets. We return p-values, standardized effect sizes, and bootstrap confidence intervals to quantify attribution strength under explicit Type I error guarantees.

The Deliverable: A Technical Report

Our analysis culminates in a technical report detailing methodology, statistical findings, and data provenance narratives. The report is structured for courtroom use, with reproducible metrics, confidence statements, and expert interpretations.

Review the Litigation Impact Metrics

Source Notes

  1. S-Square Research, "Litigation Impact Metrics 2023–2025." ↩
  2. Maini et al., "LLM Dataset Inference: Did you train on my dataset?" ICLR (2024). Source ↩
  3. Shokri et al., "Membership Inference Attacks Against Machine Learning Models," IEEE S&P (2017). Source ↩
  4. Carlini et al., "Extracting Training Data from Large Language Models," IEEE S&P (2021). Source ↩
  5. S-Square Research, "Discovery Effort Benchmarks," and methodology cross-check with Maini et al. (2024). ↩
  6. National Institute of Standards and Technology, "AI Risk Management Framework." ↩
  7. Google Search Central, "Introduction to robots.txt." ↩
  8. Andrei Z. Broder, "On the resemblance and containment of documents," ACM Digital Library. ↩
  9. Gao et al., "The Pile: An 800GB Dataset of Diverse Text for Language Modeling," arXiv (2021). ↩
  10. Schuhmann et al., "LAION-5B: An open large-scale dataset for training next generation image-text models," arXiv (2022). ↩
  11. National Institute of Standards and Technology, "Generative AI Risk Management Profile." ↩