Add FIRMCore evaluation task implementation - #292
Open
pardiszahraei wants to merge 1 commit into
Open
Conversation
Implement FIRMCore evaluation task for olmo-eval.
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a task registration for the FIRMCore fairness/bias benchmark to olmo-eval.
FIRMCore is a 50,012-item core subset of the larger FIRM benchmark, selected via
Fluid-Benchmarking-style IRT-based item selection, covering ground-truth capability
items and injected-pressure bias-lean items across ~29 domains and 8 demographic
pairing categories. This PR adds
firm_core.py, registeringfirmcoreas arunnable Task sourced directly from the
allenai/firmcore-benchmarkHF datasetvia DataSource (currently private, scheduled to go public within about a month).
Includes:
(
decision_accuracy_gap,minority_selection_rate_tied_cases,correct_decision_flip_asymmetry,injection_induced_selection_shift).just "Candidate A"), an explicit "Answer: X" line, and candidate display-name
matches (full or partial, e.g. a bare first name).
Type of Change
Testing
No pytest files were added to this repo for this task specifically. Instead,
verified manually:
uv run olmo-eval task inspect firmcore-- confirms the task loads andinstances format correctly
uv run olmo-eval run -m mock -t firmcore -o limit=200-- fulldata-loading -> formatting -> scoring -> metric-aggregation pipeline runs
end-to-end without errors
classes against hand-crafted responses with known, hand-computed expected
values (all matched)
Happy to add a proper pytest file under tests/ if that's the convention here.
Checklist