Skip to content

[BUG] resolve failing and flaky tests in CI, and race conditions in tests using server #1586

@fkiraly

Description

@fkiraly

Currently, the remote CI is not reliable and will result in faliures and sporadic failures that are unrelated for the PR in question - that means the remote CI is not usable for practical purposes. Problems:

  • it may still be that not all failures are temporarily decorated with xfail and skip properly
  • I think there is a even race condition in tests for the server, which results in sporadic failures, especially if multiple CI runs happen in parallel (which happens in every PR since there is a test matrix)

I am not sure how to address this best long-term; short-term I would suggest to simply skip all the flaky tests.

Mid-term, I think the server tests need to be decoupled from openml-python SDK tests, i.e., replaced by three things:

  • mock backends for simple API tests
  • a local server option that runs on the VM. This can follow the extension pattern introduced in the v1-v2 migration, see [ENH] V1 → V2 API Migration #1575
  • tests for the server API decoupled from tests from openml-python, possibly on a CRON and controlling for race conditions

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions