docs: fix modifying-prompts-metrics guide with correct API - #2486
Merged
anistark merged 10 commits intoDec 14, 2025
Conversation
- Fix incorrect class names (FaithfulnessInput → StatementGeneratorInput, etc.) - Fix wrong attribute names (scorer.prompt → scorer.statement_generator_prompt) - Remove non-existent save/load section (feature doesn't exist on new BasePrompt) - Add cross-references to DiscreteMetric/NumericMetric for simpler use cases - Convert from notebook to pure markdown - Add comprehensive test coverage for all code examples - Add mkdocstrings API reference rule to docs-structure.mdc
sanjeed5
force-pushed
the
fix/modifying-prompts-metrics-guide
branch
from
December 9, 2025 13:32
ba6384f to
050e3fc
Compare
anistark
reviewed
Dec 9, 2025
Member
There was a problem hiding this comment.
Should we move it to e2e tests directory and import from there in docs if needed?
* test: add claude docs auto-update workflow Testing the new workflow that automatically updates documentation when code changes. * fix: add id-token permission and fix max_turns parameter
Contributor
Author
|
@anistark removed the test for now as discussed |
Resolve mkdocs.yml conflict: keep new modifying-prompts-metrics.md path, adopt main's _metrics_language_adaptation.md and train_your_own_metric.md entries.
anistark
approved these changes
Dec 14, 2025
cgaswin
pushed a commit
to cgaswin/ragas
that referenced
this pull request
Dec 17, 2025
…bsai#2486) ## Issue Link / Problem Description The modifying-prompts-metrics guide had significant factual errors that would cause code examples to fail: - Used non-existent class names (`FaithfulnessInput`, `FaithfulnessPrompt`, etc.) - Referenced wrong attribute names (`scorer.prompt` instead of `scorer.statement_generator_prompt`) - Documented save/load functionality that doesn't exist on the collections `BasePrompt` class ## Changes Made - **Fixed incorrect class names**: `FaithfulnessInput` → `StatementGeneratorInput`, `FaithfulnessOutput` → `StatementGeneratorOutput`, etc. - **Fixed wrong attribute names**: `scorer.prompt` → `scorer.statement_generator_prompt` / `scorer.nli_statement_prompt` - **Removed non-existent save/load section**: ~175 lines documenting functionality that doesn't exist on `ragas.prompt.metrics.base_prompt.BasePrompt` - **Added cross-references**: Quick start pointing to `DiscreteMetric`/`NumericMetric` for simpler custom metrics - **Converted to pure markdown**: Deleted `.ipynb` source, created clean `.md` file - **Fixed mkdocstrings link**: Changed `[BasePrompt][ragas.prompt.metrics.base_prompt.BasePrompt]` to `[BasePrompt][ragas.prompt.BasePrompt]` (public API) - **Added docs rule**: Added API references guidance to `.cursor/rules/docs-structure.mdc` ## Testing ### How to Test - [x] Automated tests added/updated - [x] Manual testing steps: 1. Run `uv run python tests/docs/test_modifying_prompts_metrics.py` - all 7 tests pass 2. Run `make serve-docs` and verify the guide renders correctly 3. Verify all code examples in the guide work ### Test Results ``` ✓ Faithfulness has statement_generator_prompt and nli_statement_prompt ✓ Generated prompt string successfully ✓ Custom FactualCorrectness prompt evaluation completed. Score: 1.0 ✓ Custom Faithfulness NLI prompt evaluation completed. Score: 1.0 ✓ NLI prompt string generated successfully ✓ Faithfulness evaluation completed. Score: 1.0 ✓ Language adaptation completed. Score: 1.0 ✓ All tests passed! ``` ## References - Related to PR vibrantlabsai#2394 (documentation structure update) - Related to PR vibrantlabsai#2407 (metrics collections API documentation) - Fixes errors introduced in commits `691ad9f` and `e62ff19`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Link / Problem Description
The modifying-prompts-metrics guide had significant factual errors that would cause code examples to fail:
FaithfulnessInput,FaithfulnessPrompt, etc.)scorer.promptinstead ofscorer.statement_generator_prompt)BasePromptclassChanges Made
FaithfulnessInput→StatementGeneratorInput,FaithfulnessOutput→StatementGeneratorOutput, etc.scorer.prompt→scorer.statement_generator_prompt/scorer.nli_statement_promptragas.prompt.metrics.base_prompt.BasePromptDiscreteMetric/NumericMetricfor simpler custom metrics.ipynbsource, created clean.mdfile[BasePrompt][ragas.prompt.metrics.base_prompt.BasePrompt]to[BasePrompt][ragas.prompt.BasePrompt](public API).cursor/rules/docs-structure.mdcTesting
How to Test
uv run python tests/docs/test_modifying_prompts_metrics.py- all 7 tests passmake serve-docsand verify the guide renders correctlyTest Results
References
691ad9fande62ff19