Skip to content

Commit 5164ddc

Browse files
sanjeed5cgaswin
authored andcommitted
docs: fix modifying-prompts-metrics guide with correct API (vibrantlabsai#2486)
## Issue Link / Problem Description The modifying-prompts-metrics guide had significant factual errors that would cause code examples to fail: - Used non-existent class names (`FaithfulnessInput`, `FaithfulnessPrompt`, etc.) - Referenced wrong attribute names (`scorer.prompt` instead of `scorer.statement_generator_prompt`) - Documented save/load functionality that doesn't exist on the collections `BasePrompt` class ## Changes Made - **Fixed incorrect class names**: `FaithfulnessInput` → `StatementGeneratorInput`, `FaithfulnessOutput` → `StatementGeneratorOutput`, etc. - **Fixed wrong attribute names**: `scorer.prompt` → `scorer.statement_generator_prompt` / `scorer.nli_statement_prompt` - **Removed non-existent save/load section**: ~175 lines documenting functionality that doesn't exist on `ragas.prompt.metrics.base_prompt.BasePrompt` - **Added cross-references**: Quick start pointing to `DiscreteMetric`/`NumericMetric` for simpler custom metrics - **Converted to pure markdown**: Deleted `.ipynb` source, created clean `.md` file - **Fixed mkdocstrings link**: Changed `[BasePrompt][ragas.prompt.metrics.base_prompt.BasePrompt]` to `[BasePrompt][ragas.prompt.BasePrompt]` (public API) - **Added docs rule**: Added API references guidance to `.cursor/rules/docs-structure.mdc` ## Testing ### How to Test - [x] Automated tests added/updated - [x] Manual testing steps: 1. Run `uv run python tests/docs/test_modifying_prompts_metrics.py` - all 7 tests pass 2. Run `make serve-docs` and verify the guide renders correctly 3. Verify all code examples in the guide work ### Test Results ``` ✓ Faithfulness has statement_generator_prompt and nli_statement_prompt ✓ Generated prompt string successfully ✓ Custom FactualCorrectness prompt evaluation completed. Score: 1.0 ✓ Custom Faithfulness NLI prompt evaluation completed. Score: 1.0 ✓ NLI prompt string generated successfully ✓ Faithfulness evaluation completed. Score: 1.0 ✓ Language adaptation completed. Score: 1.0 ✓ All tests passed! ``` ## References - Related to PR vibrantlabsai#2394 (documentation structure update) - Related to PR vibrantlabsai#2407 (metrics collections API documentation) - Fixes errors introduced in commits `691ad9f` and `e62ff19`
1 parent c4b44e1 commit 5164ddc

7 files changed

Lines changed: 191 additions & 629 deletions

File tree

.cursor/rules/docs-structure.mdc

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,12 @@ Follow these conventions when creating or editing documentation:
3131
• Reference images via `_static/…` paths so they work in both dev and hosted docs.
3232
• Custom templates/CSS live in [docs/extra/](mdc:docs/extra/) — avoid editing `material` theme defaults directly.
3333

34-
6. **Do not modify generated or third-party files** in `_static/`, `extra/overrides/`, or `extra/components/` without good reason.
34+
6. **API References (mkdocstrings)**
35+
• Always use public API paths in `[ClassName][ragas.module.ClassName]` references.
36+
• Check what's exported in `__init__.py` — if a class isn't in `__all__`, mkdocstrings can't link to it.
37+
• Example: Use `[BasePrompt][ragas.prompt.BasePrompt]` not `[BasePrompt][ragas.prompt.base.BasePrompt]` or internal module paths.
38+
39+
7. **Do not modify generated or third-party files** in `_static/`, `extra/overrides/`, or `extra/components/` without good reason.
3540

3641
---
3742

docs/concepts/metrics/overview/index.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ from ragas.metrics import FactualCorrectness
6161
scorer = FactualCorrectness(llm=evaluation_llm)
6262
```
6363

64-
Each LLM based metrics also will have prompts associated with it written using [Prompt Object](./../../components/prompt.md).
64+
Each LLM based metrics also will have prompts associated with it written using [Prompt Object](./../../components/prompt.md). You can customize these prompts to suit your domain and use-case. Learn more in the [Modifying Prompts in Metrics](../../../howtos/customizations/metrics/modifying-prompts-metrics.md) guide.
6565

6666

6767
     **Non-LLM-based metrics**: These metrics do not use LLM underneath to do the evaluation. These metrics are deterministic and can be used to evaluate the performance of the AI application without using LLM. These metrics rely on traditional methods to evaluate the performance of the AI application, such as string similarity, BLEU score, etc. Due to the same, these metrics are known to have a lower correlation with human evaluation.
@@ -95,7 +95,7 @@ In Ragas, we categorize metrics based on the type of output they produce. This c
9595

9696
#### 1. Discrete Metrics
9797

98-
These return a single value from a predefined list of categorical classes. There is no implicit ordering among the classes. Common use cases include classifying outputs into categories such as pass/fail or good/okay/bad.
98+
These return a single value from a predefined list of categorical classes. There is no implicit ordering among the classes. Common use cases include classifying outputs into categories such as pass/fail or good/okay/bad. Discrete metrics accept custom prompts directly, making them ideal for quick custom evaluations.
9999

100100
Example:
101101
```python
@@ -106,6 +106,8 @@ def my_metric(predicted: str, expected: str) -> str:
106106
return "pass" if predicted.lower() == expected.lower() else "fail"
107107
```
108108

109+
For modifying prompts in existing collection metrics (like Faithfulness, FactualCorrectness), see [Modifying prompts in metrics](../../../howtos/customizations/metrics/modifying-prompts-metrics.md).
110+
109111
#### 2. Numeric Metrics
110112

111113
These return an integer or float value within a specified range. Numeric metrics support aggregation functions such as mean, sum, or mode, making them useful for statistical analysis.

docs/howtos/customizations/metrics/_modifying-prompts-metrics.md

Lines changed: 0 additions & 350 deletions
This file was deleted.

0 commit comments

Comments
 (0)