Cargotest enhancements

[nellshamrell]

The Infra team recently made some decisions around the future of cargotest - and there is more to decided and implemented. This issue tracks this work.

Reference: https://paper.dropbox.com/doc/The-future-of-cargotest--AxQ6EFIbNcslFkGPv4Kbr8yhAg-KuqIf3UWJnfrfrukx8o4G

[nellshamrell]

CargoTest Summary

What is Cargotest?

• component of rustc's CI
• ensures a predefined suite of libraries and applications successfully compile and pass tests with the rustc and cargo binaries
• test, called cargotest, which builds some projects on every auto CI run

Goal of Cargotest

• decided by vote on 2020-03-25 (Discord log starts here)
• "The goal of cargotest should be to make sure artifacts produced by CI work across a wide spectrum of projects, such as small HTTP microservices, medium-sized CLI tools, embedded (no_std) applications and big production codebases. All these kinds of projects rely on different language features, code patterns and build system capabilities, so testing a sample of each category should give us a broad coverage."

How does Cargotest work?

• We keep a constant listing test repos in the Cargotest main.rs file
• For each repo, cargo test...
• clones the repo
• runs the cargo test command on each repo

What does Cargotest test right now

Libraries

• iron

CLI apps

• ripgrep
• tokei
• treeify
• xsv

Big production code bases

• servo stylo
• webrender

What do we still need to decide?

• Requirements for a project to be included in Cargotest
• Projects we want to test with Cargotest
• How to keep the pinned commits up to date (each repo is pinned to a specific commit in Cargotest's main.rs

Useful Links

Code Links

• Cargo Test main.rs

Discussions on this topic

• 2020-03-11
• 2020-03-18
• 2020-03-25

[nellshamrell]

Problems with Cargotest

• we need to manually update the pinned commit of each repo we test every time the repo is update (currently pinned to a version of xsv that was committed in 2017)
• rarely breaks
• takes about 20 min to run in CI
• crates like iron are outdated

[nellshamrell]

Requirements for a project to be included in Cargotest

• actively maintained
• maintainer awareness that we are testing their project with Cargotest and a willingness to help if there are spurious failures
• no subverting the Rust stability guarantee
• must compile on stable

Possible projects to include

(This list is based on libraries that are already tested with Cargotest (and fit the above requirements), the most commonly used libraries as found by the 2019 State or Rust Survey, and in this Discord conversation

Libraries

• serde
• regex

CLI apps

• ripgrep
• tokei

Big production code bases

• servo stylo
• webrender

Async

• tokio
• reqwest

To Do

• How to keep the pinned commits up to date (each repo is pinned to a specific commit in Cargotest's main.rs)