Benchmarks should serve three functions. First prevent performance regression when changes are made on the libray's core. Then give potential users an idea of what to expect when they use the library on their problem. Finally inform users about the trade-offs of each of the algorithms.
We should thus select a few problems that are notoriously hard to sample from and setup a few of benchmarks. The selection should include problems where one (family of) algorithm(s) clearly outperforms all others to fulfill goal (3), as well as common problems to fulfill goal (2). We should measure raw speed, but also other important metrics that assess sampling quality like the number of ESS/s.
The repo already uses pytest-benchmark to run a few benchmarks, but this is rather useless since the results are not stored. In order to keep track of history, and display the results we can use this Github action. In the benchmark branch of this repo there's an example of model implementation and data fetching in the /benchmark directory. We will need to add a datastructure that specifies which algorithm can be run on this example with which parameter values.
The results should be computed at every merge on main and should be displayed in the documentation (and not be erased by a new build).
Documentation build should not erase previous benchmarks, benchmark publishing should not erase docs
Benchmarks should serve three functions. First prevent performance regression when changes are made on the libray's core. Then give potential users an idea of what to expect when they use the library on their problem. Finally inform users about the trade-offs of each of the algorithms.
We should thus select a few problems that are notoriously hard to sample from and setup a few of benchmarks. The selection should include problems where one (family of) algorithm(s) clearly outperforms all others to fulfill goal (3), as well as common problems to fulfill goal (2). We should measure raw speed, but also other important metrics that assess sampling quality like the number of ESS/s.
The repo already uses
pytest-benchmarkto run a few benchmarks, but this is rather useless since the results are not stored. In order to keep track of history, and display the results we can use this Github action. In thebenchmarkbranch of this repo there's an example of model implementation and data fetching in the/benchmarkdirectory. We will need to add a datastructure that specifies which algorithm can be run on this example with which parameter values.The results should be computed at every merge on
mainand should be displayed in the documentation (and not be erased by a new build).Documentation build should not erase previous benchmarks, benchmark publishing should not erase docs