BioGrade · expert reasoning data for biopharma agentsEnvironmentsBenchmarkCompare modelsData flywheelThesis
← Systematic Review & Meta-Analysis comparison · model: GPT (frontier)
LLM-judge score: 3.2 / 5 · flawed
screening: 2 effect_measure: 4 pooling_method: 3 numerical: 3 faithfulness: 4
The agent wrongly excluded an eligible RCT (26020745, East Asian DOAC vs warfarin) as a 'subpopulation/duplicate', dropping k from 6 to 5, and only screened 8 records instead of 12, missing the observational, VTE, and mechanical-valve exclusions entirely—so the PRISMA flow is incomplete and screening is materially off. Its included-study effects and CIs are faithful to the tool data and its pooled OR 0.80 (0.72-0.89) and I^2=23% are close to the true 0.782 (0.701-0.873) / I^2=33.6% given the missing study, with coherent log-scale handling and correct favours-DOAC interpretation. However it never names DerSimonian-Laird or reports Cochran's Q, and the deviations from the reference stem from the screening error.
Evidence synthesis / HEOR · Evidence scientist / systematic reviewer

Systematic Review & Meta-Analysis — agent trajectory for expert review

gpt-5.2-chat-latest · 2026-06-06T10:33:25.254401+00:00 · 8 tool calls · 16.8s · 8117 tokens
run id: systematic_review_20260606_031816_systematic-review__gpt-frontier

Query

Conduct a systematic review and meta-analysis: pool the effect of direct oral anticoagulants (DOACs) vs adjusted-dose warfarin on stroke or systemic embolism in adults with non-valvular atrial fibrillation, screening per the registered inclusion criteria. Report the included/excluded studies with reasons, the PRISMA counts, the pooled odds ratio with 95% CI, and the heterogeneity (I^2).

Trajectory

Agent · turn 1
⚙ calls list_review_questions()
↳ list_review_questions result
show data
{
  "questions": [
    {
      "id": "Q-EFFICACY",
      "title": "DOAC vs warfarin for prevention of stroke/systemic embolism in AF",
      "population": "Adults with non-valvular atrial fibrillation",
      "intervention": "Direct oral anticoagulant (DOAC: apixaban, dabigatran, rivaroxaban, edoxaban)",
      "comparator": "Adjusted-dose warfarin (vitamin-K antagonist)",
      "outcome": "Stroke or systemic embolism",
      "effect_measure": "OR",
      "design_required": "RCT"
    },
    {
      "id": "Q-SAFETY",
      "title": "DOAC vs warfarin for risk of major bleeding in AF",
      "population": "Adults with non-valvular atrial fibrillation",
      "intervention": "Direct oral anticoagulant (DOAC)",
      "comparator": "Adjusted-dose warfarin (vitamin-K antagonist)",
      "outcome": "Major bleeding",
      "effect_measure": "RR",
      "design_required": "RCT"
    }
  ]
}
Agent · turn 2
⚙ calls get_inclusion_criteria(question_id=Q-EFFICACY)
↳ get_inclusion_criteria result
show data
{
  "question_id": "Q-EFFICACY",
  "title": "DOAC vs warfarin for prevention of stroke/systemic embolism in AF",
  "PICO": {
    "population": "Adults with non-valvular atrial fibrillation",
    "intervention": "Direct oral anticoagulant (DOAC: apixaban, dabigatran, rivaroxaban, edoxaban)",
    "comparator": "Adjusted-dose warfarin (vitamin-K antagonist)",
    "outcome": "Stroke or systemic embolism"
  },
  "design_required": "RCT",
  "inclusion": [
    "Randomized controlled trial (RCT).",
    "Adults with non-valvular atrial fibrillation.",
    "Intervention is a DOAC; comparator is adjusted-dose warfarin.",
    "Reports stroke / systemic embolism as an effect estimate with a 95% CI."
  ],
  "exclusion": [
    "Non-randomized / observational designs.",
    "Population is not non-valvular AF (e.g. VTE, mechanical valves, post-ACS).",
    "Comparator is not warfarin (e.g. placebo, aspirin, another DOAC).",
    "Stroke/systemic embolism outcome not reported."
  ]
}
Agent · turn 3
⚙ calls search_studies(keyword=atrial fibrillation, design=RCT, outcome=stroke)
↳ search_studies result
show data
{
  "count": 8,
  "studies": [
    {
      "pmid": "21870978",
      "year": 2011,
      "design": "RCT",
      "title": "Apixaban versus warfarin in patients with atrial fibrillation",
      "population": "Adults with non-valvular atrial fibrillation (CHADS2 >= 1)",
      "intervention": "Apixaban 5 mg BID",
      "comparator": "Adjusted-dose warfarin (INR 2-3)",
      "n": 18201,
      "outcome": "Stroke or systemic embolism"
    },
    {
      "pmid": "19717844",
      "year": 2009,
      "design": "RCT",
      "title": "Dabigatran versus warfarin in patients with atrial fibrillation",
      "population": "Adults with non-valvular atrial fibrillation",
      "intervention": "Dabigatran 150 mg BID",
      "comparator": "Adjusted-dose warfarin (INR 2-3)",
      "n": 18113,
      "outcome": "Stroke or systemic embolism"
    },
    {
      "pmid": "21830957",
      "year": 2011,
      "design": "RCT",
      "title": "Rivaroxaban versus warfarin in nonvalvular atrial fibrillation",
      "population": "Adults with non-valvular atrial fibrillation at moderate-high stroke risk",
      "intervention": "Rivaroxaban 20 mg daily",
      "comparator": "Adjusted-dose warfarin (INR 2-3)",
      "n": 14264,
      "outcome": "Stroke or systemic embolism"
    },
    {
      "pmid": "24251359",
      "year": 2013,
      "design": "RCT",
      "title": "Edoxaban versus warfarin in patients with atrial fibrillation",
      "population": "Adults with non-valvular atrial fibrillation (CHADS2 >= 2)",
      "intervention": "Edoxaban 60 mg daily",
      "comparator": "Adjusted-dose warfarin (INR 2-3)",
      "n": 21105,
      "outcome": "Stroke or systemic embolism"
    },
    {
      "pmid": "32865375",
      "year": 2020,
      "design": "RCT",
      "title": "Low-dose edoxaban in very elderly patients with atrial fibrillation",
      "population": "Adults >= 80 years with non-valvular AF unsuitable for standard anticoagulation",
      "intervention": "Edoxaban 15 mg daily",
      "comparator": "Adjusted-dose warfarin",
      "n": 984,
      "outcome": "Stroke or systemic embolism"
    },
    {
      "pmid": "26020745",
      "year": 2015,
      "design": "RCT",
      "title": "DOAC versus warfarin in East Asian patients with atrial fibrillation (prespecified RCT subpopulation)",
      "population": "East Asian adults with non-valvular atrial fibrillation",
      "intervention": "Direct oral anticoagulant (standard dose)",
      "comparator": "Adjusted-dose warfarin",
      "n": 4200,
      "outcome": "Stroke or systemic embolism"
    },
    {
      "pmid": "21309657",
      "year": 2011,
      "design": "RCT",
      "title": "Apixaban versus aspirin in atrial fibrillation patients unsuitable for warfarin",
      "population": "Adults with non-valvular atrial fibrillation unsuitable for VKA therapy",
      "intervention": "Apixaban 5 mg BID",
      "comparator": "Aspirin",
      "n": 5599,
      "outcome": "Stroke or systemic embolism"
    },
    {
      "pmid": "35123456",
      "year": 2022,
      "design": "RCT",
      "title": "Apixaban versus rivaroxaban in atrial fibrillation: a head-to-head randomized trial",
      "population": "Adults with non-valvular atrial fibrillation",
      "intervention": "Apixaban",
      "comparator": "Rivaroxaban",
      "n": 3200,
      "outcome": "Stroke or systemic embolism"
    }
  ]
}
Agent · turn 4
⚙ calls get_study(pmid=21870978)
⚙ calls get_study(pmid=19717844)
⚙ calls get_study(pmid=21830957)
⚙ calls get_study(pmid=24251359)
⚙ calls get_study(pmid=32865375)
↳ get_study result
show data
{
  "pmid": "21870978",
  "year": 2011,
  "design": "RCT",
  "title": "Apixaban versus warfarin in patients with atrial fibrillation",
  "population": "Adults with non-valvular atrial fibrillation (CHADS2 >= 1)",
  "intervention": "Apixaban 5 mg BID",
  "comparator": "Adjusted-dose warfarin (INR 2-3)",
  "n": 18201,
  "outcome": "Stroke or systemic embolism",
  "effect_measure": "OR",
  "effect": 0.79,
  "effect_lo": 0.66,
  "effect_hi": 0.95,
  "other_outcomes": [
    {
      "outcome": "Major bleeding",
      "effect_measure": "RR",
      "effect": 0.69,
      "effect_lo": 0.6,
      "effect_hi": 0.8
    }
  ]
}
↳ get_study result
show data
{
  "pmid": "19717844",
  "year": 2009,
  "design": "RCT",
  "title": "Dabigatran versus warfarin in patients with atrial fibrillation",
  "population": "Adults with non-valvular atrial fibrillation",
  "intervention": "Dabigatran 150 mg BID",
  "comparator": "Adjusted-dose warfarin (INR 2-3)",
  "n": 18113,
  "outcome": "Stroke or systemic embolism",
  "effect_measure": "OR",
  "effect": 0.65,
  "effect_lo": 0.52,
  "effect_hi": 0.81,
  "other_outcomes": [
    {
      "outcome": "Major bleeding",
      "effect_measure": "RR",
      "effect": 0.93,
      "effect_lo": 0.81,
      "effect_hi": 1.07
    }
  ]
}
↳ get_study result
show data
{
  "pmid": "21830957",
  "year": 2011,
  "design": "RCT",
  "title": "Rivaroxaban versus warfarin in nonvalvular atrial fibrillation",
  "population": "Adults with non-valvular atrial fibrillation at moderate-high stroke risk",
  "intervention": "Rivaroxaban 20 mg daily",
  "comparator": "Adjusted-dose warfarin (INR 2-3)",
  "n": 14264,
  "outcome": "Stroke or systemic embolism",
  "effect_measure": "OR",
  "effect": 0.88,
  "effect_lo": 0.75,
  "effect_hi": 1.03,
  "other_outcomes": [
    {
      "outcome": "Major bleeding",
      "effect_measure": "RR",
      "effect": 1.04,
      "effect_lo": 0.9,
      "effect_hi": 1.2
    }
  ]
}
↳ get_study result
show data
{
  "pmid": "24251359",
  "year": 2013,
  "design": "RCT",
  "title": "Edoxaban versus warfarin in patients with atrial fibrillation",
  "population": "Adults with non-valvular atrial fibrillation (CHADS2 >= 2)",
  "intervention": "Edoxaban 60 mg daily",
  "comparator": "Adjusted-dose warfarin (INR 2-3)",
  "n": 21105,
  "outcome": "Stroke or systemic embolism",
  "effect_measure": "OR",
  "effect": 0.87,
  "effect_lo": 0.73,
  "effect_hi": 1.04,
  "other_outcomes": [
    {
      "outcome": "Major bleeding",
      "effect_measure": "RR",
      "effect": 0.8,
      "effect_lo": 0.71,
      "effect_hi": 0.91
    }
  ]
}
↳ get_study result
show data
{
  "pmid": "32865375",
  "year": 2020,
  "design": "RCT",
  "title": "Low-dose edoxaban in very elderly patients with atrial fibrillation",
  "population": "Adults >= 80 years with non-valvular AF unsuitable for standard anticoagulation",
  "intervention": "Edoxaban 15 mg daily",
  "comparator": "Adjusted-dose warfarin",
  "n": 984,
  "outcome": "Stroke or systemic embolism",
  "effect_measure": "OR",
  "effect": 0.66,
  "effect_lo": 0.4,
  "effect_hi": 1.09,
  "other_outcomes": []
}
Agent · turn 5

Systematic Review and Meta‑analysis

Question (Q‑EFFICACY): DOACs vs adjusted‑dose warfarin for prevention of stroke or systemic embolism in adults with non‑valvular atrial fibrillation (RCTs only; effect measure: OR)

1. Study Selection

Identification

Database search (screened pool): 8 records

Screening Against Inclusion Criteria

✅ Included (n = 5 RCTs)

PMID Trial (First Author/Year) DOAC N OR (95% CI)
21870978 Apixaban vs warfarin (2011) Apixaban 18,201 0.79 (0.66–0.95)
19717844 Dabigatran vs warfarin (2009) Dabigatran 150 mg 18,113 0.65 (0.52–0.81)
21830957 Rivaroxaban vs warfarin (2011) Rivaroxaban 14,264 0.88 (0.75–1.03)
24251359 Edoxaban vs warfarin (2013) Edoxaban 60 mg 21,105 0.87 (0.73–1.04)
32865375 Low‑dose edoxaban vs warfarin (2020) Edoxaban 15 mg 984 0.66 (0.40–1.09)

All met:
- RCT design
- Adults with non‑valvular AF
- DOAC vs adjusted‑dose warfarin
- Stroke/systemic embolism reported with OR + 95% CI

❌ Excluded (n = 3)

PMID Reason for Exclusion
26020745 Subpopulation analysis (not primary RCT; duplicate population)
21309657 Comparator was aspirin (not warfarin)
35123456 DOAC vs DOAC (no warfarin comparator)

2. PRISMA Flow Diagram Counts

3. Meta‑analysis

Method

Pooled Effect

Pooled OR = 0.80

95% CI: 0.72–0.89

Interpretation

DOACs reduce the odds of stroke/systemic embolism by approximately:

[
1 - 0.80 = 20\%
]

relative to adjusted‑dose warfarin.

The confidence interval does not cross 1.0 → statistically significant.

4. Heterogeneity

This suggests consistent treatment effects across trials despite differences in:
- Individual DOAC agents
- Stroke risk profiles
- Dosing strategies

5. Summary of Findings

Outcome Pooled OR 95% CI I² Interpretation
Stroke/systemic embolism 0.80 0.72–0.89 23% DOACs superior to warfarin

6. Conclusion

In adults with non‑valvular atrial fibrillation:

If desired, I can also provide:
- Forest plot (text or figure format)
- Fixed‑effect sensitivity analysis
- Leave‑one‑out analysis
- GRADE certainty assessment

Final output

Systematic Review and Meta‑analysis

Question (Q‑EFFICACY): DOACs vs adjusted‑dose warfarin for prevention of stroke or systemic embolism in adults with non‑valvular atrial fibrillation (RCTs only; effect measure: OR)

1. Study Selection

Identification

Database search (screened pool): 8 records

Screening Against Inclusion Criteria

✅ Included (n = 5 RCTs)

PMID Trial (First Author/Year) DOAC N OR (95% CI)
21870978 Apixaban vs warfarin (2011) Apixaban 18,201 0.79 (0.66–0.95)
19717844 Dabigatran vs warfarin (2009) Dabigatran 150 mg 18,113 0.65 (0.52–0.81)
21830957 Rivaroxaban vs warfarin (2011) Rivaroxaban 14,264 0.88 (0.75–1.03)
24251359 Edoxaban vs warfarin (2013) Edoxaban 60 mg 21,105 0.87 (0.73–1.04)
32865375 Low‑dose edoxaban vs warfarin (2020) Edoxaban 15 mg 984 0.66 (0.40–1.09)

All met:
- RCT design
- Adults with non‑valvular AF
- DOAC vs adjusted‑dose warfarin
- Stroke/systemic embolism reported with OR + 95% CI

❌ Excluded (n = 3)

PMID Reason for Exclusion
26020745 Subpopulation analysis (not primary RCT; duplicate population)
21309657 Comparator was aspirin (not warfarin)
35123456 DOAC vs DOAC (no warfarin comparator)

2. PRISMA Flow Diagram Counts

3. Meta‑analysis

Method

Pooled Effect

Pooled OR = 0.80

95% CI: 0.72–0.89

Interpretation

DOACs reduce the odds of stroke/systemic embolism by approximately:

[
1 - 0.80 = 20\%
]

relative to adjusted‑dose warfarin.

The confidence interval does not cross 1.0 → statistically significant.

4. Heterogeneity

This suggests consistent treatment effects across trials despite differences in:
- Individual DOAC agents
- Stroke risk profiles
- Dosing strategies

5. Summary of Findings

Outcome Pooled OR 95% CI I² Interpretation
Stroke/systemic embolism 0.80 0.72–0.89 23% DOACs superior to warfarin

6. Conclusion

In adults with non‑valvular atrial fibrillation:

If desired, I can also provide:
- Forest plot (text or figure format)
- Fixed‑effect sensitivity analysis
- Leave‑one‑out analysis
- GRADE certainty assessment

Reference answer key (deterministic) — open after grading

Deterministic DerSimonian-Laird random-effects meta-analysis over the curated study pool, screened per the question's explicit PICO + RCT-only criteria. Ratio measures pooled on the log scale with SE=(ln(hi)-ln(lo))/(2*1.96). I^2>50% => prefer random-effects. A strong agent should recover the same include/exclude set, PRISMA counts, pooled estimate, and I^2 interpretation.

ItemValueNote
Review questionQ-EFFICACYDOAC vs warfarin for prevention of stroke/systemic embolism in AF
PRISMA: records screened12total study pool
PRISMA: excluded629654632 (Wrong design: Observational (retrospective cohort) (RCT required)); 23808982 (Wrong population: Adults with venous thromboembolism (DVT/PE)); 23991661 (Wrong population: Adults with mechanical prosthetic heart valves); 21309657 (Wrong comparator: Aspirin (warfarin required)); 35123456 (Wrong comparator: Rivaroxaban (warfarin required)); 31151464 (Outcome not reported: Stroke or systemic embolism)
PRISMA: included (k)621870978, 19717844, 21830957, 24251359, 32865375, 26020745
include 21870978OR=0.79 (0.66-0.95)Apixaban versus warfarin in patients with atrial fibrillatio
include 19717844OR=0.65 (0.52-0.81)Dabigatran versus warfarin in patients with atrial fibrillat
include 21830957OR=0.88 (0.75-1.03)Rivaroxaban versus warfarin in nonvalvular atrial fibrillati
include 24251359OR=0.87 (0.73-1.04)Edoxaban versus warfarin in patients with atrial fibrillatio
include 32865375OR=0.66 (0.4-1.09)Low-dose edoxaban in very elderly patients with atrial fibri
include 26020745OR=0.67 (0.5-0.91)DOAC versus warfarin in East Asian patients with atrial fibr
Cochran's Q (df)7.53 (df=5)
tau^2 (DerSimonian-Laird)0.0061
I^233.6%low/moderate (<=50%)
Recommended pooling methodeither (low heterogeneity); random-effects reported
Pooled OR (random-effects)0.782 (95% CI 0.701-0.873)primary estimate
Pooled OR (fixed-effect, for contrast)0.795
Interpretationfavours DOACOR<1 favours DOAC over warfarin

Expert grading

Screening accuracy & PRISMA discipline 1 (poor) 5 (excellent)
5: Applies the explicit PICO + RCT-only criteria correctly: includes exactly the eligible RCTs and excludes the rest, each with the correct concrete reason (wrong design, population, comparator, or outcome not reported), and reports coherent PRISMA identification/screening/eligibility/included counts.
1: Includes ineligible studies (e.g. the observational cohort, the VTE/valve populations, the placebo/aspirin or DOAC-vs-DOAC comparators, or the study that doesn't report the outcome), drops eligible RCTs, or gives no/garbled PRISMA flow.
Effect-measure & log-scale handling 1 (poor) 5 (excellent)
5: Uses the correct measure (OR for efficacy, RR for safety), pools on the LOG scale, and derives each study's SE from its CI as (ln(hi)-ln(lo))/(2*1.96) rather than treating the point estimate or CI on the natural scale.
1: Pools raw (non-log) ratios, mishandles or invents the standard errors, mixes OR and RR, or pulls the wrong outcome's effect for a study.
Pooling-model choice & I^2 interpretation 1 (poor) 5 (excellent)
5: Computes Cochran's Q and I^2, and chooses fixed vs random-effects coherently with the heterogeneity (I^2>50% => random-effects), naming the DerSimonian-Laird estimator and interpreting I^2 correctly.
1: Ignores heterogeneity, picks a fixed-effect model despite high I^2 (or vice versa with no rationale), or misreads what I^2 means.
Numerical correctness of the pooled estimate & CI 1 (poor) 5 (excellent)
5: The pooled point estimate, 95% CI, and I^2 match the deterministic inverse-variance / DerSimonian-Laird computation within rounding, and the pooled estimate sits within the range of the included study estimates.
1: Pooled estimate or CI is materially wrong, falls outside the plausible range of inputs, has an inverted/implausible CI, or the arithmetic is unjustified.
Evidence faithfulness 1 (poor) 5 (excellent)
5: Every study, effect estimate, and CI used traces to the actual tool outputs; no fabricated trials, effects, or CIs, and excluded studies' numbers are not smuggled into the pool.
1: Fabricates studies or effect sizes, alters the reported CIs, or pools effects from studies it claimed to exclude.
Overall verdict
Your name / ID