Migrate on own parser

[alexander-akait]

/cc @czosel

Current parser does not improve and does not fix bugs, I propose to go to your own parser, we can use https://github.com/nikic/PHP-Parser/tree/master/grammar as example. I know it is require time, but unfortunately it seems we have no other way.

[loilo]

Any chance we could viably compile the original PHP parser to WebAssembly?

[loilo]

PHP as a whole has already been compiled to WASM, but I have no insight into that field and can't tell whether compiling parts (i.e. just the parser) would be feasible.

[alexander-akait]

@loilo i am afraid what original php parsers is not enough for prettier, a lot of information about nodes is missing because php no need this

[karptonite]

Any possibility the plugin could use the https://github.com/nikic/PHP-Parser directly, or would this have the same issue with missing nodes? Obviously that means running php processes as part of prettier PHP (no necessarily simple), but it would have the benefit that it would be easier to stay up to date with future language changes.

[alexander-akait]

@karptonite same problem not enough node information 😞 I think I can implement parser using bison/yacc (https://github.com/nikic/PHP-Parser using modified version of yacc, but it is not a problem) for two weeks, but i don't have time right now, if somebody want to help me will be great.

[karptonite]

Perhaps the PHP version is extensible enough to allow it to be extended to get the additional node information? I'm looking at this issue as an example of getting information that doesn't normally get stored by the parser: nikic/PHP-Parser#384
It also looks like there are some lexer options that might give you more information: https://github.com/nikic/PHP-Parser/blob/1.x/doc/component/Lexer.markdown#lexer-options
I don't know much about how these things work, though, so I'm sort of guessing here.

[alexander-akait]

To be honestly for us interesting only https://github.com/nikic/PHP-Parser/blob/master/grammar/php5.y and https://github.com/nikic/PHP-Parser/blob/master/grammar/php7.y and https://github.com/nikic/PHP-Parser/blob/master/grammar/tokens.y

What we should do:

• Install bison/yacc
• Create js version https://github.com/nikic/PHP-Parser/blob/master/grammar/parser.template
• Rewrite this on js https://github.com/nikic/PHP-Parser/blob/master/grammar/rebuildParsers.php
• Implement each Node (example https://github.com/nikic/PHP-Parser/blob/master/grammar/php7.y#L39) on js, given the names and structure of the current nodes what we use in code, most likely we will have even less of them
• Copy https://github.com/nikic/PHP-Parser/blob/master/grammar/php5.y , https://github.com/nikic/PHP-Parser/blob/master/grammar/php7.y and https://github.com/nikic/PHP-Parser/blob/master/grammar/tokens.y

It is not hard, node(s) is very simple

[alexander-akait]

I can try to find time on PoC

[czosel]

Hi @evilebottnawi,
first of all sorry about the delay, I've been swamped with work lately.
This sounds like a big step, and before jumping right into it, I'm wondering

• Which are the show-stopping issues in the parser we're currently using right now?
• Can those issues not be fixed in the current parser?
• If we decide to implement a new parser, what would be fundamentally different in it's design that would solve the issues we're having now?

[alexander-akait]

Which are the show-stopping issues in the parser we're currently using right now?

• maximum supporting php version is 7.2 (7.3 features doesn't support)
• invalid ast in some cases (parens and *lookup nodes)
• bad ast for inline nodes
• invalid position for comments in some cases

Can those issues not be fixed in the current parser?

I think yes, but nobody do this. Also bison/yacc parser better maintenance.

If we decide to implement a new parser, what would be fundamentally different in it's design that would solve the issues we're having now?

BFN (EBFN) grammar and generate parser based on this rules. You can see https://github.com/nikic/PHP-Parser/tree/master/grammar here how it can be done.

[czosel]

Alright, thanks for sharing the details - I'll try to take a closer look when I find the time 🙂

[ichiriac]

Hi @evilebottnawi and @czosel, interresting approach. In the past I've used https://www.npmjs.com/package/jison but at that time the LR(1) generated code was not good enough to parse some syntax collisions on PHP, and by resolving them is was too permissive and also too slow compared to now, but agree that the maintenance of the syntax may be easier on grammar files than code.

Did you succeed with a POC or prototype ?

[alexander-akait]

@ichiriac not, is is just idea, it it no high priority, the problem is that we get fixes/features extremely slowly and very few guys have experience with parsers, supporting parser generator based on bfn/ebfn will be easy