Skip to content

cleanup = true doesn't work for local cleanup in hybrid workflows #7334

Description

@rzelle-lallemand

Bug report

We have an in-house Nextflow 26.04 pipeline that mostly runs on-prem, but some of the processes are performed with AWS Batch. These runs create a local work directory and a remote one (specified with -bucket-dir). I understand that remote cleanup isn't supported (see #5091 (comment) and https://docs.seqera.io/nextflow/reference/config#cleanup). However, I would expect local cleanup to work in hybrid workflows, but this doesn't work.

Expected behavior and actual behavior

With "cleanup = true", I expect the local work directory to be cleaned up, even if Nextflow doesn't want to touch the remote work directory. Instead, Nextflow fails to perform a local cleanup.

Steps to reproduce the problem

Use "cleanup = true" in nextflow.config in a hybrid on-prem/AWS Batch workflow.

Program output

Jul-14 14:06:31.475 [main] WARN  nextflow.Session - Failed to cleanup work dir: /projectA/temp/nextflow-pipeline/run_2026-07-14/work
java.lang.IllegalStateException: Missing plugin 'nf-amazon' required to read file: s3://private-bucket/48/299e411a526eb1453a5a2acfa0f721
	at nextflow.file.FileHelper.asPath(FileHelper.groovy:409)
	at nextflow.file.FileHelper.asPath0(FileHelper.groovy:362)
	at nextflow.file.FileHelper.asPath(FileHelper.groovy:345)
	at nextflow.Session$_cleanup_closure41.doCall(Session.groovy:1165)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:280)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1009)
	at groovy.lang.Closure.call(Closure.java:433)
	at nextflow.cache.CacheDB.eachRecord(CacheDB.groovy:212)
	at nextflow.Session.cleanup(Session.groovy:1161)
	at nextflow.script.ScriptRunner.shutdown(ScriptRunner.groovy:246)
	at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:136)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:442)
	at nextflow.cli.Launcher.run(Launcher.groovy:526)
	at nextflow.cli.Launcher.main(Launcher.groovy:686)

Environment

  • Nextflow version: 26.04.2
  • Java version: [?]
  • Operating system: Linux
  • Bash version: GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)

Additional context

(Add any other context about the problem here)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions