Methodology: Dataset Inference

In AI litigation, establishing a clear chain of evidence for data usage is paramount. Our dataset inference methodology provides a rigorous, multi-faceted approach to determine if and how specific data was used to train an AI model. This process is critical for asserting data privacy and intellectual property rights.

Our Approach

We employ a combination of advanced statistical analysis, model interrogation techniques, and data provenance tracing to build a comprehensive evidentiary picture. Our process involves several key steps designed to uncover evidence of data usage within an LLM:

• Membership Inference Attacks: We use cutting-edge techniques to test whether specific data points were included in a model's training set, informed by industry research on model privacy leakage. By analyzing the model's responses—such as its confidence scores or subtle statistical biases—we can infer the presence of the data in its training set.
• Data Provenance Tracing: Our team traces the origins of training data, scouring web archives, code repositories, and academic papers to document its path from source to model. This helps establish the likelihood that the defendant had access to the plaintiff's data.
• Model Interrogation: By carefully querying a model, we can often elicit behaviors that strongly indicate it was trained on a specific corpus of data. This can include causing the model to regurgitate content verbatim.
• Statistical Verification: All findings are subjected to rigorous statistical evaluation to quantify the likelihood that the observed model behavior is a direct result of it being trained on the client's data, rather than a coincidental occurrence.

Relevant Research

Our methods are informed by the latest advancements in machine learning transparency and accountability. The following papers provide foundational insights into the techniques we adapt and build upon: