Currently, the inference system lives under /inference. After installing the various dependencies, plus tmux, you can run it using the full-dev-setup.sh script (or use the docker compose inference profile).
The inference system consists of multiple parts:
- one central inference server that connects to a postgres and a redis database.
- many external workers that perform the actual inference
- each worker consist again of two parts, a python connector and a text-generation backend
- the text-generation backend exposes an HTTP API
- the python connector connects to the HTTP API and also to the central server via websocket
- optionally: the text-client. this one is mainly for testing
The goal of this issue is to document the inference pipeline, both how the individual parts are built (e.g. how the central server manages users, workers, etc.), and how they connect. This can be done using text, diagrams, etc. whatever. If a diagram, then preferrably a mermaid diagram directly in markdown.
Currently, the inference system lives under
/inference. After installing the various dependencies, plus tmux, you can run it using thefull-dev-setup.shscript (or use the docker compose inference profile).The inference system consists of multiple parts:
The goal of this issue is to document the inference pipeline, both how the individual parts are built (e.g. how the central server manages users, workers, etc.), and how they connect. This can be done using text, diagrams, etc. whatever. If a diagram, then preferrably a mermaid diagram directly in markdown.