Skip to content

Add FIRMCore evaluation task implementation - #292

Open
pardiszahraei wants to merge 1 commit into
allenai:mainfrom
pardiszahraei:add-firmcore-task
Open

Add FIRMCore evaluation task implementation#292
pardiszahraei wants to merge 1 commit into
allenai:mainfrom
pardiszahraei:add-firmcore-task

Conversation

@pardiszahraei

Copy link
Copy Markdown

Adds a task registration for the FIRMCore fairness/bias benchmark to olmo-eval.
FIRMCore is a 50,012-item core subset of the larger FIRM benchmark, selected via
Fluid-Benchmarking-style IRT-based item selection, covering ground-truth capability
items and injected-pressure bias-lean items across ~29 domains and 8 demographic
pairing categories. This PR adds firm_core.py, registering firmcore as a
runnable Task sourced directly from the allenai/firmcore-benchmark HF dataset
via DataSource (currently private, scheduled to go public within about a month).

Includes:

  • 4 metrics, named for what they measure rather than abbreviated
    (decision_accuracy_gap, minority_selection_rate_tied_cases,
    correct_decision_flip_asymmetry, injection_induced_selection_shift).
  • Answer extraction robust to per-domain role-noun labels (e.g. "Patient A", not
    just "Candidate A"), an explicit "Answer: X" line, and candidate display-name
    matches (full or partial, e.g. a bare first name).

Type of Change

  • New feature (non-breaking change that adds functionality)

Testing

No pytest files were added to this repo for this task specifically. Instead,
verified manually:

  • uv run olmo-eval task inspect firmcore -- confirms the task loads and
    instances format correctly
  • uv run olmo-eval run -m mock -t firmcore -o limit=200 -- full
    data-loading -> formatting -> scoring -> metric-aggregation pipeline runs
    end-to-end without errors
  • Standalone script testing the answer-extraction function and all 4 metric
    classes against hand-crafted responses with known, hand-computed expected
    values (all matched)

Happy to add a proper pytest file under tests/ if that's the convention here.

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my code
  • I have added/updated documentation as needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published (N/A, no dependent changes)

Implement FIRMCore evaluation task for olmo-eval.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant