Talking To Machines – Upcoming Synthetic Replication Games

RD
Raymond Duch
Sun, Sep 28, 2025 10:35 AM

In the context of a large replication effort of experimental work using LLMs as synthetic participants, the Talking to Machine team is organizing a series of one-day events dedicated to adapting the original material of experiments published in top-tier journals to a standardised format which allows replicating them with a synthetic sample. The objective of the project is to systematically evaluate whether, and under which conditions, LLMs can be leveraged to augment or replace human samples in experimental work. More details on the project and the first iteration conducted during IMEBESS 2025 can be found herehttps://talkingtomachines.org/projects/imebess-2025-synthetic-replication-games/. We are inviting researchers from diverse backgrounds and at all career stages to participate. During the event, participants will form small teams of 3-4 members, based on their research interests. Each team will be assigned a study from a leading journal.
Participants will be recognized as co-authors on the paper that will present the aggregated results of multiple synthetic replications using a variety of LLMs.

Upcoming Games

  • Friday, 24 Oct 2025 - Hybrid - Nuffield College, University of Oxford - Focused on field experiments
  • Tuesday, 18 Nov 2025 - Virtual - Focused on experiments in behavioural economics
  • Tuesday, 16 Dec 2025 - Virtual - Focused on experiments in political sciences

Games Structure

Researchers participating in the event will join a small team of 3-4 members with shared research interests. Each team will engage in the synthetic replication of one or more experiments published in a top-tier journal. Each synthetic replications will involve:

  1. The computational replication of the original results using the provided analysis script.
  2. Converting the original data collection instrument into a set of standardised experimental prompts.
  3. Introducing – and documenting – the prompt adaptations required to conduct the experiment with a synthetic sample.
  4. Running a pilot synthetic replication and benchmarking the LLM predictions with the results of the original study.

Registration

To register and find more information on each event, please head to the Talking to Machines websitehttps://talkingtomachines.org/projects/imebess-2025-synthetic-replication-games/.

We hope to see many of you!

Best wishes,

Talking to Machines

Raymond Duch
Senior Research Fellow

Nuffield College
OX1 1NF Oxford

raymond.duch@nuffield.ox.ac.ukmailto:raymond.duch@nuffield.ox.ac.uk
www.raymondduch.comhttp://www.raymondduch.com/
mobile: +44 7 760 167 008

In the context of a large replication effort of experimental work using LLMs as synthetic participants, the Talking to Machine team is organizing a series of one-day events dedicated to adapting the original material of experiments published in top-tier journals to a standardised format which allows replicating them with a synthetic sample. The objective of the project is to systematically evaluate whether, and under which conditions, LLMs can be leveraged to augment or replace human samples in experimental work. More details on the project and the first iteration conducted during IMEBESS 2025 can be found here<https://talkingtomachines.org/projects/imebess-2025-synthetic-replication-games/>. We are inviting researchers from diverse backgrounds and at all career stages to participate. During the event, participants will form small teams of 3-4 members, based on their research interests. Each team will be assigned a study from a leading journal. Participants will be recognized as co-authors on the paper that will present the aggregated results of multiple synthetic replications using a variety of LLMs. Upcoming Games * Friday, 24 Oct 2025 - Hybrid - Nuffield College, University of Oxford - Focused on field experiments * Tuesday, 18 Nov 2025 - Virtual - Focused on experiments in behavioural economics * Tuesday, 16 Dec 2025 - Virtual - Focused on experiments in political sciences Games Structure Researchers participating in the event will join a small team of 3-4 members with shared research interests. Each team will engage in the synthetic replication of one or more experiments published in a top-tier journal. Each synthetic replications will involve: 1. The computational replication of the original results using the provided analysis script. 2. Converting the original data collection instrument into a set of standardised experimental prompts. 3. Introducing – and documenting – the prompt adaptations required to conduct the experiment with a synthetic sample. 4. Running a pilot synthetic replication and benchmarking the LLM predictions with the results of the original study. Registration To register and find more information on each event, please head to the Talking to Machines website<https://talkingtomachines.org/projects/imebess-2025-synthetic-replication-games/>. We hope to see many of you! Best wishes, Talking to Machines Raymond Duch Senior Research Fellow Nuffield College OX1 1NF Oxford raymond.duch@nuffield.ox.ac.uk<mailto:raymond.duch@nuffield.ox.ac.uk> www.raymondduch.com<http://www.raymondduch.com/> mobile: +44 7 760 167 008