Inference Documentation

[alando46]

This is a mostly done (although not totally complete) PR with a technical overview of the inference architecture. I'm looking forward to high level feedback (general layout, flow of documentation) or specific suggestions (I'm sure I made some errors or missed some details.) I will try to wrap up the final section soon.

See related discussion on the issue: #1473 (comment)

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[GuilleHoardings]

Nice work!

It looks like you went for a quite detailed explanation of the code. As a suggestion, I think that a diagram of how all the components are connected, and a brief description of each one's goal, would help.

Also, I think it's important to try to use always the same word for the different components. A couple of examples of things I've seen which I think they are the same, but I'm not sure:

• In the architecture document, it talks about a text client, that it's referred to as Text Client, text-client, and a REPL client in the inference document.

• In the architecture document, it talks about a FastAPI webserver, that it's referred to as OA server in the mermaid diagram and OA Inference Server in the inference document.

Anyway, these are minor things. Thank you again for the work!

[yk]

thanks a lot for this :)

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[andreaskoepf]

@alando46 this PR is currently still in draft status. Is it ready for review?

[alando46]

@andreaskoepf, would it be possible to raise this as-is (w/o last section)? I keep trying to get back to this but things have gotten unexpectedly busy.

I'd propose raising this as-is, and then down the road I or someone else can update the final section. The plugin addition (very cool) will likely change some things on the server/worker inference system so might make sense to update the inference documentation once again anyways once that api stabilizes a bit.

Thoughts?