# Pre-registration: synthetic sample replication and design sensitivity Frozen before any data was collected. Anything that changed after the first run is recorded in the "Deviations" section at the bottom, with a timestamp. **Frozen at:** 2026-09-15 (UTC), before the first report was created. **Platform:** moevox.com REST API (`POST /api/v1/reports`), account credits at freeze time: 7,897. **Prepared for:** the article series at `articles/moevox/`. --- ## 0. What the platform actually does (verified before freezing) Two behaviours were confirmed by inspecting a previously completed report (`articles/citation-survey.json`) rather than by reading docs, because they change what a "replication" can honestly claim: 1. **The submitted `question` is a research goal, not a fielded item.** The platform generates its own questionnaire around it — ten slots in the observed report (`slot_winner`, `slot_driver`, `slot_risk_structured`, `slot_risk_open`, `slot_adoption_intent`, `slot_recommendation_support`, `slot_trust`, `slot_clarity`, `slot_fit`, `slot_switching_barrier`) — and **rewrites the primary question's wording**. Submitted: *"When an AI assistant answers your question, which of these makes you trust its answer most?"* Fielded stem: *"Which of the following factors most significantly increases your trust in the accuracy and reliability of an AI assistant's response?"* 2. **Submitted options are preserved, but an opt-out may be added.** A two-option concept test on the public homepage came back with three entities, the third being *"Neither of these options"*. **Consequence for this study.** Every comparison below is **option-set matched, not wording matched**. The generated stem is published verbatim next to the benchmark stem so a reader can see the drift and judge it. This is a limitation of the study and is reported as one. --- ## 1. Study A — replication against published benchmarks Four reports. Population string is submitted verbatim and recorded; the platform maps it to its own frame, so the audience line is an input, not a controlled variable. ### A1 — Education (mechanical control) - **Goal submitted:** "What is the highest level of education you have completed?" - **Options:** `Less than high school` / `High school graduate or equivalent` / `Some college or associate degree` / `Bachelor's degree` / `Graduate or professional degree` - **Audience:** `US adults aged 25 and older` - **Sample size:** 1,000 - **Benchmark:** ACS 2024 1-year estimates, table S1501, population 25+. Pinned to the exact published percentages at analysis time; the source table is cited in the article. - **Why it is here:** this item is a *positive control*, not a finding. Education is a variable that already exists in the sampling frame (ACS PUMS). If the pipeline cannot reproduce the frame's own marginal on a frame variable, nothing else in this study is interpretable. It is the cheapest available test of "does the machinery work at all". ### A2 — Interpersonal trust (attitudinal, two options) - **Goal submitted:** "Generally speaking, would you say that most people can be trusted or that you can't be too careful in dealing with people?" - **Options:** `Most people can be trusted` / `You can't be too careful in dealing with people` - **Audience:** `US adults aged 18 and older` - **Sample size:** 1,000 - **Benchmark:** GSS 2024, web administration, no volunteered response: **37.2% / 62.5%** (N=2,521). - **Why it is here:** a long-running, non-commercial, methodologically documented item with a published topline and a documented mode effect. ### A3 — The same item with one option added (option-set sensitivity) - **Goal submitted:** identical to A2. - **Options:** A2's two options **plus** `It depends` - **Audience:** `US adults aged 18 and older` - **Sample size:** 500 - **Benchmark:** GSS 2024, web administration, with volunteered response: **19.4% / 43.3% / 37.3% "depends"**. - **Why it is here:** GSS's own documentation shows this single option moving the headline number by **17.8 points**. A2 and A3 test whether our pipeline shows an effect in the same direction and rough magnitude. - **Stated asymmetry:** in GSS the difference between the two published figures is an administration/mode difference, not a pure option-set difference. In our study it is a pure option-set difference, with the same submitted goal. The comparison is **directional, not numerical.** ### A4 — Cigarette smoking (behaviour with a documented social-desirability problem) - **Goal submitted:** "Do you currently smoke cigarettes?" - **Options:** `Yes` / `No` - **Audience:** `US adults aged 18 and older` - **Sample size:** 500 - **Benchmark:** CDC/NCHS, current cigarette smoking among adults: **9.9% (2024)**, **10.8% (2023)** (National Health Interview Survey). - **Why it is here:** a behaviour item where the real-world number is itself depressed by social desirability. It is the honest test of whether synthetic respondents inherit the same reporting bias humans have, or a different one. ### A5 — Naive LLM control (no platform, zero credits) Same four items, asked of a general-purpose model with no census grounding, in the two forms people actually use it: - **Distribution form:** "You are simulating a sample of 1,000 US adults. Give the percentage distribution." Repeated **5 times** per item; the spread across repeats is reported alongside the mean, because instability is a distinct failure from bias. - **Persona form:** generate 200 individual respondents in one batch, aggregate once. Purpose: separates "synthetic respondents are biased" from "the naive way of doing this is biased *and unstable*". Without this control the study would only be able to say the first. --- ## 2. Study C — design sensitivity One neutral marketing-copy item, held fixed except for the variable under test. The item is a two-option choice between two plausible taglines, chosen so no option has an obvious social-desirability pull. - **Goal submitted:** "Which of these two taglines makes you more likely to try a grocery delivery service? A: 'Free delivery on your first order.' B: 'Groceries at your door in 30 minutes.'" - **Options:** the two taglines. - **Audience:** `US adults aged 18 and older` | Run | Variable under test | Sample size | Credits | |---|---|---|---| | C1 | baseline | 50 | 50 | | C2 | baseline | 100 | 100 | | C3 | baseline | 200 | 200 | | C4 | baseline | 500 | 500 | | C5 | baseline | 1,000 | 1,000 | | C6 | option order reversed | 200 | 200 | | C7 | leading stem ("Most people prefer Option A. Which do you prefer?") | 200 | 200 | | C8 | neutral stem, reworded | 200 | 200 | **Caveat stated up front:** because the platform generates the questionnaire, the stem is regenerated per run, so C8 and the baseline differ on more than the variable we intended. The generated stems are published verbatim, and C8 is reported as a **lower bound** on wording sensitivity, not a clean estimate. C4 (n=500) doubles as the baseline against which C6 and C7 are compared. --- ## 3. Registered predictions Written before the runs. Reported whether or not they hold. | # | Prediction | Falsified if | |---|---|---| | P1 | Education falls within 5 points of ACS on every category | any category off by >5 | | P2 | Trust item within 10 points of the GSS topline (both options) | either option off by >10 | | P3 | Adding "It depends" moves our headline trust number by ≥10 points, same direction as GSS's 17.8 | movement <10 or opposite direction | | P4 | Smoking over-reported (synthetic respondents do not inherit humans' under-reporting) | topline ≤ 9.9% | | P5 | Naive-LLM repeat spread ≥5 points on at least 2 of the 4 items, while our platform gives one number per run | naive spread <5 on ≥3 items | **P4 is the one we expect to fail, and that is the interesting outcome.** Stated in advance so that it cannot be re-framed afterwards as a discovery. --- ## 4. Decision rules 1. **One run per configuration.** If a comparison contradicts a prediction, that is the result; no rerun to obtain a friendlier number. 2. **Technical failure only** justifies a rerun (HTTP error, `status: failed`, timeout). A rerun is disclosed with the reason. 3. **No cherry-picking the primary question.** The reported number is always `slot_winner` (the primary decision question in `report_blueprint`), never a support slot that happens to land closer to the benchmark. 4. **Percentages are recomputed from the stored per-respondent rows**, not copied from the report narrative, so every figure in the article can be re-derived from published files. 5. **All raw responses are published**, including the ones that make the platform look worse. 6. **Bootstrap confidence intervals** (10,000 resamples, seeded) are reported for every platform percentage. A deviation smaller than the interval is not called a deviation. --- ## 5. Registered cost | Study | Credits | |---|---| | A1–A4 | 3,000 | | C1–C8 | 2,450 | | A5 (naive LLM control) | 0 | | **Total** | **5,450** | > **Correction (2026-09-16, see D5):** this table originally read 2,650 / 5,650. That was an > addition error. The per-run rows above sum to 2,450. Nothing else in the pre-registration > depended on it, but the figures published in the articles are the measured ones. Balance after freeze: 7,897. Expected remaining: ~2,247. --- ## 6. Deviations Every entry below happened after the freeze above. Nothing before this line was changed. ### D1 — 2026-09-15, during a2: report recovered instead of re-created The polling loop crashed on a transient `fetch failed` after report `req_09b718ba631c17bd` (`a2-trust-two-option`) had already been created and charged (1,000 credits). The report was **re-fetched by request_id**, not re-created, so no extra credits were spent and the data is the original run. Two script defects were fixed as a result: the payload is now written to disk immediately after creation, and transient network errors during polling no longer abort the run. ### D2 — 2026-09-15, after a1 and a2 completed: two diagnostic runs added Reading the first two runs' respondent records showed that the pool carries **zero** respondents whose own ACS record is below high school, and very few aged 65+. Those were observations, not predictions, so two runs were added to test them directly rather than infer the cause: | Run | Purpose | Sample | Credits | |---|---|---|---| | `x-frame-65plus` | ask for 500 → 200 respondents aged 65+ and see what the pool returns | 200 | 200 | | `x-frame-less-than-hs` | ask for 200 respondents aged 25+ below high school | 200 | 200 | These are labelled `study: 'post-freeze'` in the run registry and are reported in the article as a follow-up test, separate from the pre-registered replication table. ### D3 — 2026-09-15: ACS region shares are approximate Census regions were aggregated by this study from ACS 2024 1-year data and rounded to one decimal place. The article cites them as approximate and does not draw conclusions from differences smaller than two points. ### D4 — 2026-09-15: education enum mapping corrected The first version of the self-consistency check mapped only three of the pool's `education_level` values and silently dropped `bachelor` and `master`, which made the agreement rate look like it was computed over 588 respondents instead of 1,000. The mapping was completed before any figure was written into an article. Recorded here because the corrected version changes a reported number. ### D5 — 2026-09-16, after all runs: arithmetic error in the registered cost table The registered cost table in §5 stated 2,650 credits for C1–C8. The per-run figures in the same section sum to **2,450**. The error was in the total, not in the design, and no run was added or removed because of it. Actual expenditure for the whole batch was **5,948 credits** (A 2,999, C 2,449, the 100-credit diagnostic, and the 400-credit post-freeze follow-up), against a balance of 7,897 at freeze time. ### D6 — 2026-09-16, before publication: two reported figures corrected The disagreement counts in the frame audit were first written as 65 below / 44 above the record; the recomputed values are **58 below / 51 above**, and the largest single disagreement is *high school → some college* (28), not *high school → less than high school* (23). The C study was first described as 2,449 respondent records and 2,650 credits; the measured values are **2,450 records, 2,449 usable answers, 2,449 credits**. Both were corrected before any article was published, and both are recorded here because a self-audit that silently edits its own numbers is not one.