Skip to content

Commit f846e4c

Browse files
acoultonacoulton
authored
fix: Ensure that suggested step definitions actually match the step text (#1656)
Previously, when generating snippets for undefined step definitions, there was no guarantee that the pattern would actually match the step text. In some cases (in particular with turnip) this proved to be an issue when feature files contained complex or unexpected syntax: * The first run would apparently provide a valid snippet. * The second run would still not match this snippet, so would propose a new step with the same pattern. * The third run would fail due to the presence of duplicate patterns in the steps from the first and second run. Instead, check that the generated snippet actually matches the step text at the time of generating it. If not, show the user a warning with information about either generating these as regex or manually defining a step to match their features. Fixes #1653
1 parent 036e638 commit f846e4c

12 files changed

Lines changed: 249 additions & 7 deletions

File tree

‎features/snippet_generation_failures.feature‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
Feature: Handle failures when generating snippets
2+
In order to avoid problems when a step cannot be automatically converted to a snippet pattern
3+
As a feature developer
4+
I need Behat to report any steps that need to be created manually or with a different pattern type
5+
6+
Background:
7+
Given I initialise the working directory from the SnippetGenerationFailures fixtures folder
8+
And I provide the following options for all behat invocations:
9+
| option | value |
10+
| --no-colors | |
11+
| --format-settings | '{"paths": false}' |
12+
| --format | progress |
13+
| --snippets-for | FeatureContext |
14+
15+
Scenario: See warnings when printing snippets
16+
When I run behat with the following additional options:
17+
| option | value |
18+
| --snippets-type | turnip |
19+
Then it should pass with:
20+
"""
21+
UU
22+
23+
1 scenario (1 undefined)
24+
2 steps (2 undefined)
25+
26+
--- FeatureContext has missing steps. Define them with these snippets:
27+
28+
#[Given('a step with :arg1 inside a quoted parameter')]
29+
public function aStepWithInsideAQuotedParameter($arg1): void
30+
{
31+
throw new PendingException();
32+
}
33+
34+
--- Don't forget these 2 use statements:
35+
36+
use Behat\Behat\Tester\Exception\PendingException;
37+
use Behat\Step\Given;
38+
39+
--- Could not automatically generate snippets matching the following steps:
40+
(try using --snippets-type=regex, or manually define the step)
41+
42+
- a step with (Parentheses) in the actual step text
43+
"""
44+
45+
Scenario: See warnings when appending snippets
46+
When I run behat with the following additional options:
47+
| option | value |
48+
| --snippets-type | turnip |
49+
| --append-snippets | |
50+
Then it should pass with:
51+
"""
52+
UU
53+
54+
1 scenario (1 undefined)
55+
2 steps (2 undefined)
56+
57+
u features/bootstrap/FeatureContext.php - `a step with "(parentheses)" inside a quoted parameter` definition added
58+
59+
--- Could not automatically generate snippets matching the following steps:
60+
(try using --snippets-type=regex, or manually define the step)
61+
62+
- a step with (Parentheses) in the actual step text
63+
"""
64+
And "features/bootstrap/FeatureContext.php" file should contain:
65+
"""
66+
<?php
67+
68+
use Behat\Step\Given;
69+
use Behat\Behat\Tester\Exception\PendingException;
70+
use Behat\Behat\Context\Context;
71+
72+
class FeatureContext implements Context
73+
{
74+
75+
#[Given('a step with :arg1 inside a quoted parameter')]
76+
public function aStepWithInsideAQuotedParameter($arg1): void
77+
{
78+
throw new PendingException();
79+
}
80+
}
81+
"""

‎i18n.php‎

Lines changed: 35 additions & 0 deletions
Large diffs are not rendered by default.

‎src/Behat/Behat/Context/Snippet/Generator/CannotGenerateStepPatternException.php‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
namespace Behat\Behat\Context\Snippet\Generator;
6+
7+
use UnexpectedValueException;
8+
9+
final class CannotGenerateStepPatternException extends UnexpectedValueException
10+
{
11+
public function __construct(
12+
public readonly string $stepText,
13+
) {
14+
parent::__construct(
15+
sprintf(
16+
'Cannot automatically generate a step pattern matching `%s`',
17+
$stepText,
18+
),
19+
);
20+
}
21+
}

‎src/Behat/Behat/Context/Snippet/Generator/ContextSnippetGenerator.php‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,12 @@ public function generateSnippet(Environment $environment, StepNode $step): Snipp
109109
$stepText = $step->getText();
110110
$pattern = $this->patternTransformer->generatePattern($patternType, $stepText);
111111

112+
if ($this->patternTransformer->matchPattern($pattern->getPattern(), $stepText) === false) {
113+
// The generated pattern does not actually match the step text. For example because the step
114+
// text contains literal characters that are treated as placeholders or similar.
115+
throw new CannotGenerateStepPatternException($stepText);
116+
}
117+
112118
$methodName = $this->getUniqueMethodName(
113119
$contextClass,
114120
$pattern->getPattern(),

‎src/Behat/Behat/Definition/Pattern/PatternTransformer.php‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,24 @@ public function transformPatternToRegex($pattern)
7979
return $this->patternToRegexpCache[$pattern];
8080
}
8181

82+
/**
83+
* Tests if a pattern matches a given step text.
84+
*
85+
* If the pattern does not match, returns false. Otherwise, returns an array of the portions of the
86+
* step text that match any placeholders within the pattern.
87+
*
88+
* @return false|array<int|string,string>
89+
*/
90+
public function matchPattern(string $pattern, string $stepText): false|array
91+
{
92+
$regex = $this->transformPatternToRegex($pattern);
93+
if (!preg_match($regex, $stepText, $matches)) {
94+
return false;
95+
}
96+
97+
return $matches;
98+
}
99+
82100
/**
83101
* @param string $pattern
84102
*

‎src/Behat/Behat/Definition/Search/RepositorySearchEngine.php‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -110,9 +110,8 @@ public function searchDefinition(
110110
*/
111111
private function match(Definition $definition, $stepText, array $multiline)
112112
{
113-
$regex = $this->patternTransformer->transformPatternToRegex($definition->getPattern());
114-
115-
if (!preg_match($regex, $stepText, $match)) {
113+
$match = $this->patternTransformer->matchPattern($definition->getPattern(), $stepText);
114+
if ($match === false) {
116115
return null;
117116
}
118117

‎src/Behat/Behat/Snippet/Cli/SnippetsController.php‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,10 @@ public function execute(InputInterface $input, OutputInterface $output)
100100
$this->eventDispatcher->addListener(ExerciseCompleted::AFTER, [$this, 'printAllSnippets'], -999);
101101
}
102102

103+
if (!$input->getOption('no-snippets')) {
104+
$this->eventDispatcher->addListener(ExerciseCompleted::AFTER, $this->printSnippetGenerationFailures(...), -999);
105+
}
106+
103107
if (!$input->getOption('no-snippets')) {
104108
$this->eventDispatcher->addListener(ExerciseCompleted::AFTER, [$this, 'printUndefinedSteps'], -995);
105109
}
@@ -123,7 +127,9 @@ public function registerUndefinedStep(AfterStepTested $event)
123127
public function appendAllSnippets()
124128
{
125129
$snippets = $this->registry->getSnippets();
126-
count($snippets) && $this->output->writeln('');
130+
if ($snippets) {
131+
$this->output->writeln('');
132+
}
127133

128134
$this->writer->appendSnippets($snippets);
129135
}
@@ -134,18 +140,32 @@ public function appendAllSnippets()
134140
public function printAllSnippets()
135141
{
136142
$snippets = $this->registry->getSnippets();
137-
count($snippets) && $this->output->writeln('');
143+
if ($snippets) {
144+
$this->output->writeln('');
145+
}
138146

139147
$this->writer->printSnippets($this->printer, $snippets);
140148
}
141149

150+
private function printSnippetGenerationFailures(): void
151+
{
152+
$failures = $this->registry->getGenerationFailures();
153+
if ($failures) {
154+
$this->output->writeln('');
155+
}
156+
157+
$this->printer->printSnippetGenerationFailures($failures);
158+
}
159+
142160
/**
143161
* Prints all undefined steps.
144162
*/
145163
public function printUndefinedSteps()
146164
{
147165
$undefined = $this->registry->getUndefinedSteps();
148-
count($undefined) && $this->output->writeln('');
166+
if ($undefined) {
167+
$this->output->writeln('');
168+
}
149169

150170
$this->writer->printUndefinedSteps($this->printer, $undefined);
151171
}

‎src/Behat/Behat/Snippet/Generator/SnippetGenerator.php‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010

1111
namespace Behat\Behat\Snippet\Generator;
1212

13+
use Behat\Behat\Context\Snippet\Generator\CannotGenerateStepPatternException;
1314
use Behat\Behat\Snippet\Snippet;
1415
use Behat\Behat\Snippet\SnippetRegistry;
1516
use Behat\Gherkin\Node\StepNode;
@@ -35,6 +36,8 @@ public function supportsEnvironmentAndStep(Environment $environment, StepNode $s
3536
* Generates snippet from search.
3637
*
3738
* @return Snippet
39+
*
40+
* @throws CannotGenerateStepPatternException if the step text cannot be automatically converted to a pattern
3841
*/
3942
public function generateSnippet(Environment $environment, StepNode $step);
4043
}

‎src/Behat/Behat/Snippet/Printer/ConsoleSnippetPrinter.php‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010

1111
namespace Behat\Behat\Snippet\Printer;
1212

13+
use Behat\Behat\Context\Snippet\Generator\CannotGenerateStepPatternException;
1314
use Behat\Behat\Definition\Translator\TranslatorInterface;
1415
use Behat\Behat\Snippet\AggregateSnippet;
1516
use Behat\Gherkin\Node\StepNode;
@@ -46,6 +47,7 @@ public function __construct(OutputInterface $output, TranslatorInterface $transl
4647

4748
$output->getFormatter()->setStyle('snippet_keyword', new OutputFormatterStyle(null, null, ['bold']));
4849
$output->getFormatter()->setStyle('snippet_undefined', new OutputFormatterStyle('yellow'));
50+
$output->getFormatter()->setStyle('snippet_failure', new OutputFormatterStyle('red'));
4951
}
5052

5153
/**
@@ -108,4 +110,25 @@ public function outputClassesUsesStatements(array $usedClasses): void
108110
$this->output->writeln(sprintf(' <snippet_undefined>use %s;</snippet_undefined>', $usedClass));
109111
}
110112
}
113+
114+
/**
115+
* @param array<CannotGenerateStepPatternException> $exceptions
116+
*/
117+
public function printSnippetGenerationFailures(array $exceptions): void
118+
{
119+
if ([] === $exceptions) {
120+
return;
121+
}
122+
123+
$title = $this->translator->trans('snippet_generation_failure_title', [], 'output');
124+
$hint = $this->translator->trans('snippet_generation_failure_hint', [], 'output');
125+
126+
$this->output->writeln('<snippet_failure>--- '.$title.'</snippet_failure>');
127+
$this->output->writeln('<snippet_failure> '.$hint.'</snippet_failure>');
128+
$this->output->writeln('');
129+
130+
foreach ($exceptions as $exception) {
131+
$this->output->writeln('<snippet_failure> - '.$exception->stepText.'</snippet_failure>');
132+
}
133+
}
111134
}

‎src/Behat/Behat/Snippet/SnippetRegistry.php‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010

1111
namespace Behat\Behat\Snippet;
1212

13+
use Behat\Behat\Context\Snippet\Generator\CannotGenerateStepPatternException;
1314
use Behat\Behat\Snippet\Generator\SnippetGenerator;
1415
use Behat\Gherkin\Node\StepNode;
1516
use Behat\Testwork\Environment\Environment;
@@ -38,6 +39,11 @@ final class SnippetRegistry implements SnippetRepository
3839
*/
3940
private $snippetsGenerated = false;
4041

42+
/**
43+
* @var list<CannotGenerateStepPatternException>
44+
*/
45+
private array $exceptions = [];
46+
4147
/**
4248
* Registers snippet generator.
4349
*/
@@ -82,6 +88,16 @@ public function getUndefinedSteps()
8288
return $this->undefinedSteps;
8389
}
8490

91+
/**
92+
* @return list<CannotGenerateStepPatternException>
93+
*/
94+
public function getGenerationFailures(): array
95+
{
96+
$this->generateSnippets();
97+
98+
return $this->exceptions;
99+
}
100+
85101
/**
86102
* Generates snippets for undefined steps.
87103
*/
@@ -93,7 +109,13 @@ private function generateSnippets(): void
93109

94110
$snippetsSet = [];
95111
foreach ($this->undefinedSteps as $i => $undefinedStep) {
96-
$snippet = $this->generateSnippet($undefinedStep->getEnvironment(), $undefinedStep->getStep());
112+
try {
113+
$snippet = $this->generateSnippet($undefinedStep->getEnvironment(), $undefinedStep->getStep());
114+
} catch (CannotGenerateStepPatternException $e) {
115+
$this->exceptions[] = $e;
116+
unset($this->undefinedSteps[$i]);
117+
continue;
118+
}
97119

98120
if (!$snippet) {
99121
continue;

0 commit comments

Comments
 (0)