[Bug]: Premature container reaping under concurrent multi-package `go test ./...`

[karsov]

Testcontainers version

v0.43.0

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host arch

x86

Go version

1.26.4

Docker version

Client: Docker Engine - Community
 Version:           29.6.0
 API version:       1.55
 Go version:        go1.26.4
 Git commit:        fb59821
 Built:             Thu Jun 18 20:01:09 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.6.0
  API version:      1.55 (minimum version 1.40)
  Go version:       go1.26.4
  Git commit:       70eaf5e
  Built:            Thu Jun 18 19:57:14 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.5
  GitCommit:        e53c7c1516c3b2bff98eb76f1f4117477e6f4e66
 runc:
  Version:          1.3.6
  GitCommit:        v1.3.6-0-g491b69ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client: Docker Engine - Community                                                                                                                                                           
 Version:    29.6.0                                                                                                                                                                         
 Context:    default                                                                                                                                                                        
 Debug Mode: false                                                                                                                                                                          
 Plugins:                                                                                                                                                                                   
  buildx: Docker Buildx (Docker Inc.)                                                                                                                                                       
    Version:  v0.34.1                                                                                                                                                                       
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx                                                                                                                                 
  compose: Docker Compose (Docker Inc.)                                                                                                                                                     
    Version:  v5.1.4                                                                                                                                                                        
    Path:     /usr/libexec/docker/cli-plugins/docker-compose                                                                                                                                

Server:                                                                                                                                                                                     
 Containers: 75                                                                                                                                                                             
  Running: 0                                                                                                                                                                                
  Paused: 0                                                                                                                                                                                 
  Stopped: 75                                                                                                                                                                               
 Images: 96
 Server Version: 29.6.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e53c7c1516c3b2bff98eb76f1f4117477e6f4e66
 runc version: v1.3.6-0-g491b69ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 7.0.12-101.fc43.x86_64
 Operating System: Fedora Linux 43 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 31.06GiB
 ID: e301f56b-a96b-4def-a297-bd861c894943
 Docker Root Dir: /home/docker-data
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables+firewalld
  EnableUserlandProxy: true
  UserlandProxyPath: /usr/bin/docker-proxy

What happened?

After the upgrade from v0.41.0 to v0.43.0 in my project I started to regularly observe the testcontainers-backed tests in some of the packages failing when I run all tests in parallel (go test ./...) but they always pass if I run only a single package (go test ./pkg/db, or run the tests consecutively (go test ./... -p 1).

Looking more closely at the logs I noticed that for all but 1 of the testcontainers-backed packages, I get Reaper handshake errors like this:

2026/06/29 11:58:58 Reaper handshake failed: read ack: read tcp [::1]:45156->[::1]:33074: read: connection reset by peer

or like this

2026/06/29 12:09:49 Reaper handshake failed: read ack: EOF

I also, noticed that the packages with the slower tests fail approximately 10 seconds after the testcontainer-backed package with quicker tests succeeds.

All of this led me to think that my issue is that for most of my packages testcontainers is not able to connect and register into ryuk, and then 10s after the tests, that are successfully registered, complete, a cleanup is deleting my still running containers and causing the tests to fail.

My theory seems to be confirmed by the code:

1. The Ryuk reconnect timeout is 10s (which I read as - if no tests reconnect in 10s then trigger a cleanup); code: https://github.com/testcontainers/testcontainers-go/blob/main/internal/config/config.go#L72
2. When a reaper container is reused the code doesn't check if it's actually started (like checking the logs for "Started" for a new reaper), which means the registering to it will not happen:
   1. code for new reaper: https://github.com/testcontainers/testcontainers-go/blob/main/reaper.go#L379
   2. code when reusing: https://github.com/testcontainers/testcontainers-go/blob/main/reaper.go#L348
3. Handshake failures with the reaper are practically ignored and not retried: https://github.com/testcontainers/testcontainers-go/blob/main/reaper.go#L544
4. It seems that slower ryuk starup in 0.43.0 causing race conditions was already observed in other cases, judging by the comment here: https://github.com/testcontainers/testcontainers-go/blob/main/reaper.go#L370

I was able to create a relatively simple reproduction (described in the "Additional information" below). It still has a random component to it, so if you don't get the behaviour the first time, just re-run it.

Also, when I added a wait for log "Started" in the reuse path my tests started to work again ( https://github.com/testcontainers/testcontainers-go/blob/main/reaper.go#L348 ):

	if err := wait.ForAll(
		wait.ForLog("Started"),
		wait.ForExposedPort().
			WithPollInterval(100*time.Millisecond).
			SkipInternalCheck(),
	).WaitUntilReady(ctx, dockerContainer); err != nil {
		return nil, fmt.Errorf("wait for reaper %s: %w", dockerContainer.ID[:8], err)
	}

I may be wrong, as I'm new to the code in this repo. So please suggest.

P.S. I've double-checked and the issue is not present in v0.42.0, so for now I've rolled back to it.

Relevant log output

=== RUN   TestFast                                                                                                                                                                          
2026/06/29 12:10:33 github.com/testcontainers/testcontainers-go - Connected to docker:                                                                                                      
  Server Version: 29.6.0                                                                                                                                                                    
  API Version: 1.55                                                                                                                                                                         
  Operating System: Fedora Linux 43 (Workstation Edition)                                                                                                                                   
  Total Memory: 31803 MB                                                                                                                                                                    
  Testcontainers for Go Version: v0.43.0                                                                                                                                                    
  Resolved Docker Host: unix:///var/run/docker.sock                                                                                                                                         
  Resolved Docker Socket Path: /var/run/docker.sock                                                                                                                                         
  Test SessionID: 5bc62bf6d10e7d04e301fdae36fd6b3b65ea9c87cae57ebfc59d1b5c4bcafa5a                                                                                                          
  Test ProcessID: 35ba4fa7-2206-432f-89a7-23eb28d0f46f                                                                                                                                      
2026/06/29 12:10:33 🐳 Creating container for image alpine:3.20                                                                                                                             
2026/06/29 12:10:34 🐳 Creating container for image testcontainers/ryuk:0.14.0
2026/06/29 12:10:34 ✅ Container created: 3e36d02eae2a
2026/06/29 12:10:34 🐳 Starting container: 3e36d02eae2a
2026/06/29 12:10:34 ✅ Container started: 3e36d02eae2a
2026/06/29 12:10:34 ⏳ Waiting for container id 3e36d02eae2a image: testcontainers/ryuk:0.14.0. Waiting for: all of: [log message "Started", port 8080/tcp to be listening]
2026/06/29 12:10:34 Shell not found in container
2026/06/29 12:10:34 🔔 Container is ready: 3e36d02eae2a
2026/06/29 12:10:34 ✅ Container created: 1c7a90b17cdf
2026/06/29 12:10:34 🐳 Starting container: 1c7a90b17cdf
2026/06/29 12:10:35 ✅ Container started: 1c7a90b17cdf
2026/06/29 12:10:35 🔔 Container is ready: 1c7a90b17cdf
2026/06/29 12:10:35 🐳 Stopping container: 1c7a90b17cdf
2026/06/29 12:10:45 ✅ Container stopped: 1c7a90b17cdf
2026/06/29 12:10:45 🐳 Terminating container: 1c7a90b17cdf
2026/06/29 12:10:45 🚫 Container terminated: 1c7a90b17cdf
--- PASS: TestFast (11.49s)
PASS
ok      example.com/tc-reaper-repro/fast1       11.543s
=== RUN   TestFast
2026/06/29 12:10:33 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 29.6.0
  API Version: 1.55
  Operating System: Fedora Linux 43 (Workstation Edition)
  Total Memory: 31803 MB
  Testcontainers for Go Version: v0.43.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 5bc62bf6d10e7d04e301fdae36fd6b3b65ea9c87cae57ebfc59d1b5c4bcafa5a
  Test ProcessID: 8fd57afe-6c3b-45fc-ad1e-b75fcab72f18
2026/06/29 12:10:33 🐳 Creating container for image alpine:3.20
2026/06/29 12:10:34 🐳 Creating container for image testcontainers/ryuk:0.14.0
2026/06/29 12:10:34 ⏳ Waiting for Reaper "3e36d02e" to be ready
2026/06/29 12:10:34 🔥 Reaper obtained from Docker for this test session 3e36d02e
2026/06/29 12:10:34 ✅ Container created: d0251609dd0f
2026/06/29 12:10:34 🐳 Starting container: d0251609dd0f
2026/06/29 12:10:34 Reaper handshake failed: read ack: read tcp [::1]:56352->[::1]:33079: read: connection reset by peer
2026/06/29 12:10:34 ✅ Container started: d0251609dd0f
2026/06/29 12:10:34 🔔 Container is ready: d0251609dd0f
2026/06/29 12:10:34 🐳 Stopping container: d0251609dd0f
2026/06/29 12:10:44 ✅ Container stopped: d0251609dd0f
2026/06/29 12:10:44 🐳 Terminating container: d0251609dd0f
2026/06/29 12:10:45 🚫 Container terminated: d0251609dd0f
--- PASS: TestFast (11.18s)
PASS
ok      example.com/tc-reaper-repro/fast2       11.237s
=== RUN   TestFast                                                                                                                                                                          
2026/06/29 12:10:33 github.com/testcontainers/testcontainers-go - Connected to docker:                                                                                                      
  Server Version: 29.6.0                                                                                                                                                                    
  API Version: 1.55                                                                                                                                                                         
  Operating System: Fedora Linux 43 (Workstation Edition)                                                                                                                                   
  Total Memory: 31803 MB                                                                                                                                                                    
  Testcontainers for Go Version: v0.43.0                                                                                                                                                    
  Resolved Docker Host: unix:///var/run/docker.sock                                                                                                                                         
  Resolved Docker Socket Path: /var/run/docker.sock                                                                                                                                         
  Test SessionID: 5bc62bf6d10e7d04e301fdae36fd6b3b65ea9c87cae57ebfc59d1b5c4bcafa5a                                                                                                          
  Test ProcessID: d6172199-ff94-43e4-b3fc-60ee0d67513d                                                                                                                                      
2026/06/29 12:10:33 🐳 Creating container for image alpine:3.20                                                                                                                             
2026/06/29 12:10:34 🐳 Creating container for image testcontainers/ryuk:0.14.0                                                                                                              
2026/06/29 12:10:34 ⏳ Waiting for Reaper "3e36d02e" to be ready                                                                                                                            
2026/06/29 12:10:34 🔥 Reaper obtained from Docker for this test session 3e36d02e                                                                                                           
2026/06/29 12:10:34 ✅ Container created: 902a97f89481
2026/06/29 12:10:34 🐳 Starting container: 902a97f89481
2026/06/29 12:10:34 Reaper handshake failed: read ack: read tcp [::1]:56340->[::1]:33079: read: connection reset by peer
2026/06/29 12:10:34 ✅ Container started: 902a97f89481
2026/06/29 12:10:34 🔔 Container is ready: 902a97f89481
2026/06/29 12:10:34 🐳 Stopping container: 902a97f89481
2026/06/29 12:10:45 ✅ Container stopped: 902a97f89481
2026/06/29 12:10:45 🐳 Terminating container: 902a97f89481
2026/06/29 12:10:45 🚫 Container terminated: 902a97f89481
--- PASS: TestFast (11.26s)
PASS
ok      example.com/tc-reaper-repro/fast3       11.297s
=== RUN   TestFast
2026/06/29 12:10:33 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 29.6.0
  API Version: 1.55
  Operating System: Fedora Linux 43 (Workstation Edition)
  Total Memory: 31803 MB
  Testcontainers for Go Version: v0.43.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 5bc62bf6d10e7d04e301fdae36fd6b3b65ea9c87cae57ebfc59d1b5c4bcafa5a
  Test ProcessID: 7eb61e7f-77c5-4844-8066-0c9d179717b0
2026/06/29 12:10:33 🐳 Creating container for image alpine:3.20
2026/06/29 12:10:34 🐳 Creating container for image testcontainers/ryuk:0.14.0
2026/06/29 12:10:34 ⏳ Waiting for Reaper "3e36d02e" to be ready
2026/06/29 12:10:34 🔥 Reaper obtained from Docker for this test session 3e36d02e
2026/06/29 12:10:34 ✅ Container created: b5fbae9feee2
2026/06/29 12:10:34 🐳 Starting container: b5fbae9feee2
2026/06/29 12:10:34 Reaper handshake failed: read ack: read tcp [::1]:56360->[::1]:33079: read: connection reset by peer
2026/06/29 12:10:34 ✅ Container started: b5fbae9feee2
2026/06/29 12:10:34 🔔 Container is ready: b5fbae9feee2
2026/06/29 12:10:34 🐳 Stopping container: b5fbae9feee2
2026/06/29 12:10:45 ✅ Container stopped: b5fbae9feee2
2026/06/29 12:10:45 🐳 Terminating container: b5fbae9feee2
2026/06/29 12:10:45 🚫 Container terminated: b5fbae9feee2
--- PASS: TestFast (11.30s)
PASS
ok      example.com/tc-reaper-repro/fast4       11.352s
=== RUN   TestSlow
2026/06/29 12:10:33 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 29.6.0
  API Version: 1.55
  Operating System: Fedora Linux 43 (Workstation Edition)
  Total Memory: 31803 MB
  Testcontainers for Go Version: v0.43.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 5bc62bf6d10e7d04e301fdae36fd6b3b65ea9c87cae57ebfc59d1b5c4bcafa5a
  Test ProcessID: 8c51e7a3-2428-4d2f-982e-9b62cd327076
2026/06/29 12:10:33 🐳 Creating container for image alpine:3.20
2026/06/29 12:10:34 🐳 Creating container for image testcontainers/ryuk:0.14.0
2026/06/29 12:10:34 ⏳ Waiting for Reaper "3e36d02e" to be ready
2026/06/29 12:10:34 🔥 Reaper obtained from Docker for this test session 3e36d02e
2026/06/29 12:10:34 ✅ Container created: bb337d9cbc11
2026/06/29 12:10:34 🐳 Starting container: bb337d9cbc11
2026/06/29 12:10:34 Reaper handshake failed: read ack: EOF
2026/06/29 12:10:34 ✅ Container started: bb337d9cbc11
2026/06/29 12:10:34 🔔 Container is ready: bb337d9cbc11
    slow_test.go:31: slow: container stopped after 21s (reaped by Ryuk)
2026/06/29 12:10:55 🐳 Stopping container: bb337d9cbc11
2026/06/29 12:10:55 ✅ Container stopped: bb337d9cbc11
2026/06/29 12:10:55 🐳 Terminating container: bb337d9cbc11
2026/06/29 12:10:55 🚫 Container terminated: bb337d9cbc11
--- FAIL: TestSlow (21.91s)
FAIL
FAIL    example.com/tc-reaper-repro/slow        21.954s
FAIL

Additional information

Code to reproduce the issue:

go.mod:

module example.com/tc-reaper-repro

go 1.26

require github.com/testcontainers/testcontainers-go v0.43.0

Create folders fast1, fast2, fast3 & fast4, and put the following inside a fast_test.go file (just change the package name). I tried with a smaller number of "fast" packages but the issue reproduces less often then.

package fast2

import (
	"context"
	"testing"

	"github.com/testcontainers/testcontainers-go"
)

func TestFast(t *testing.T) {
	ctx := context.Background()
	ctr, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: testcontainers.ContainerRequest{
			Image: "alpine:3.20",
			Cmd:   []string{"sleep", "600"},
		},
		Started: true,
	})
	if err != nil {
		t.Fatalf("fast1: start container: %v", err)
	}
	t.Cleanup(func() { _ = ctr.Terminate(ctx) })
}

Create folder slow and put a file slow_test.go inside:

package slow

import (
	"context"
	"testing"
	"time"

	"github.com/testcontainers/testcontainers-go"
)

func TestSlow(t *testing.T) {
	ctx := context.Background()
	ctr, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: testcontainers.ContainerRequest{
			Image: "alpine:3.20",
			Cmd:   []string{"sleep", "600"},
		},
		Started: true,
	})
	if err != nil {
		t.Fatalf("slow: start container: %v", err)
	}
	t.Cleanup(func() { _ = ctr.Terminate(ctx) })

	for i := 0; i < 30; i++ {
		state, err := ctr.State(ctx)
		if err != nil {
			t.Fatalf("slow: container gone after %ds (reaped by Ryuk): %v", i, err)
		}
		if !state.Running {
			t.Fatalf("slow: container stopped after %ds (reaped by Ryuk)", i)
		}
		time.Sleep(time.Second)
	}
}

Finally, run the tests:

go mod tidy
go test -v -count=1 ./...

[karsov]

@mdelapenya Should I simply open a PR with the fix that worked for me?

I'm not completely sure if it's the best solution, however.
Plus I wasn't able to figure out a good enough unit test.

[mdelapenya]

@karsov hi! thanks for taking the time in trying to address this. So your solution would be to compose the current wait strategy of the reaper with a ForLog one, right? Yeah, submit the PR and see it on the CI. It you can add some tests for it, like the ones you posted, it would be cool.

I'd add the tests in the first commit, see them fail, and then the fix, seeing them pass. WDYT?

[karsov]

@mdelapenya I created the PR as you recommended - first the commit with the unit-test change (I think I was able to come up with a good enough non-flaky test), which you can see failing, and then the actual change fixing the issue.
The tests in the CI ran for my unit-test commit, but did not for the code fix. Is there something I'm missing?