ESGF-NG status as of August 1, 2025.
NOTE: ESGF-NG will be used for CMIP7 and subsequent activities. None of the below applies to CMIP6 or earlier activities.
ESGF-NG is the next generation of the core architecture for the Earth System Grid Federation. Designed in 2024 and deployed in 2025, it replaces the ESGF-1.0 core architecture. See ESGF-NG Docs for further information.
Our current work is still consistent with the design as written.
One significant addition (that doesn't change any of the written design) is a dependency in the STAC Transaction API on STAC extensions created by the esg-vocab (aka ESGVOC) project.
- When a catalog item is added or updated using the STAC API, the item's metadata schema and controlled vocabulary are validated against the JSON schemas defined by the relevant STAC extensions.
- This validation occurs before an event is created on the ESGF event stream.
- All ESGF STAC extensions are registered with the STAC community's extensions repository.
- IMPORTANT: This addition means that before an ESGF project can publish items, the project must register its metadata schema and controlled vocabularies with the ESGVOC project and the ESGVOC project must publish a STAC extension for the project.
| The Past | Activity |
|---|---|
| 2019-11 | ESGF Future Architecture workshop held in UK - Report |
| 2024-04 | ESGF-NG proposed at 10th ESGF conference in Bethesda, MD (USA) |
| 2024-11 | ESGF mini-conference in Abingdon (UK); Data challenges proposed and challenges 1-3 designed and scheduled for Jan/Feb 2025 |
| 2024-12-13 | Globus performs Data Challenge 01 round 1. (Includes end-to-end basic CMIP6 dataset publication.) |
| 2025-01-30 | CEDA performs Data Challenge 01 round 1. |
| 2025-03-25 | Data Challenge 01 round 2 completed. (Includes both Globus and EGI-Check In authorization and schema-level validation.) |
| 2025-05-01 | ESGVOC team provided first version of STAC generator capability for ESGF projects |
| 2025-07-01 | STAC Transaction API uses STAC extensions generated by ESGVOC for item validation. Extensions are available for CMIP6, Input4MIPS, and Obs4MIPS. |
| 2025-07-01 | Onboarded engineers from the ESGF PID team to the Event Stream for preliminary evaluation. (See Event Stream below for outcomes.) |
The following timeline is estimated and subject to change.
| The Future | Activity |
|---|---|
| 2025-08-08 | Data Challenges 02, 03, and 05 completed. (Includes dataset updates and retractions and catalog consistency errors reported on the event stream.) East & West catalogs available for client testing. |
| August 1-August 15 | Onboard ESGF REF developers to data challenge event stream so they can explore events from the ESGF-NG data challenges and design an integration with the ESGF REF. (See Event Stream below.) |
| 2025-08-15 | Data Challenge 01 round 3 and Data Challenge 04 completed. (Includes controlled vocabulary validation and support for ESGVOC projects beyond CMIP6.) East & West catalogs available for client testing. |
| 2025-09-01 | Target for establishing production ESGF-NG publication capability for "friendly projects" (STAC catalogs, event stream, publisher and related client tools) |
| 2025-09-30 | Target for production ESGF-NG publication capability for all ESGF projects |
Updates are divided into two sections: data challenges and component status. Data challenges are end-to-end tests of the ESGF-NG core architecture and related components, by which we assure ourselves that the system is ready for CMIP7 publishing and discovery. The component status sections provide details about individual sub-components of the ESGF-NG core architecture. (Component details become less important as time passes.)
The ESGF-NG team is currently conducting a series of data challenges (trial runs of the ESGF-NG system) to validate the expected behavior of the end-to-end ESGF-NG system.
- Completed: Data Challenge 1 (rounds 1 & 2), end-to-end publication. CEDA and Globus each published 500+ unique datasets (drawn from CMIP6) to their own STAC catalogs. When finished, the two STAC catalogs were compared to confirm they were synchronized and both catalogs had all entries.
- In Progress: Data Challenges 2 and 3 and 5, end-to-end publication, update, and retraction, including ESGF client application testing. We expect to complete this the first week of August. Code changes have been merged in STAF FastAPI, STAC Transaction Extension, ESGF-NG STAC Transaction API, and ESGF-NG Event Consumer, including posting errors on the ESGF event stream. We need ESGF client applications to test against the resulting STAC catalogs. (We are counting on the ESGF XC to coordinate the latter testing activity with us and with individual client application teams.) The resulting event stream will also be used by the ESGF PID team to evaluate use in their project.
- Next Up: Data Challenge 1 (Round 3) and Data Challenge 4, schema and CV validation for multiple ESGF projects. Code changes to STAC catalog components, input datasets and STAC extensions for CMIP6 and Obs4MIPS are available. We will perform the publications and updates in early August. We will need ESGF client applications to test against the resulting STAC catalogs. The resulting event stream will also be used by the ESGF PID team to evaluate use in their project.
IMPORTANT NOTE: We expect ESGF client application testing against the STAC catalogs produced by the data challenges above to identify issues with the STAC catalogs that we will need to prioritize and address.
Reminder: We still have no technical design for handling catalog consistency errors published on the ESGF event stream by the event consumer. Our plan is to monitor the errors that occur and use that to develop an error-handing strategy, which will likely be a combination of manual and automated processing.
The ESGF STAC catalogs enforce metadata schema and controlled vocabulary validation for all catalog additions and updates. STAC extensions (JSON schemas) published by the ESGVOC team (one per ESGF project) provide the basis for this validation. The data challenge to be run in August will test this feature.
The ESGVOC team has prepared initial versions of STAC extensions for CMIP6, Input4MIPS, and Obs4MIPS. ESGF client teams have identified defects with these, so the ESGVOC team is preparing revisions. We will isolate some of the previously published Input4MIPS datasets and use it as part of our input data for the data challenges described above. (The remainder of the input data will come from CMIP6.) This will serve as a test of our ability to validate new items and item updates for multiple ESGF projects.
We have informed the ESGF WIP and the ESGVOC team that it is essential that each ESGF project supply JSON schema and STAC extensions before the project will be able to publish data using ESGF-NG.
The Transaction API enables STAC clients to make changes to the catalog. These changes may be addition of new items (POST) or changes to an existing item (PUT, PATCH). ESGF's STAC catalogs will understand DELETE requests but no client will be authorized to perform them. For each request, the STAC Transaction API:
- confirms the authorization of the request,
- validates the item metadata, and
- posts an event on the ESGF Event Stream.
It does not implement the change in the catalog. Instead, it posts an event on the ESGF event stream with the details of the required change. The catalog change itself is performed by the Event Consumers for each ESGF STAC catalog.
The data challenge currently in progress is testing all of the functionality described above. We believe the validation code (both for new items and for item changes) is complete and have tested it internally. The metadata validation is still causing issues with this data challenge because the ESGF project schema and controlled vocabularies are not yet stable.
The Event Consumer subscribes to the ESGF Event Stream, reads events from the stream, and performs the corresponding changes in a STAC Catalog. These changes may be addition of new items (PUT) or changes to existing items (POST, PATCH). If the Event Consumer detects a consistency error, it publishes an error to the ESGF Event Stream's error topic. The ESGF event consumer supports (via configuration) either an ElasticSearch or Globus Search index as its backend.
The data challenge currently in progress is testing all of the functionality described above.
NOTE: The error event schema is preliminary and may change to address issues identified during testing or early data publication.
The ESGF event stream service is operated by Globus as part of the ESGF2-US project. We are currently using Confluent Data Streaming, a managed Apache Kafka deployment hosted in AWS.
The event stream is currently being used for our ESGF-NG data challenges.
- Write access (ability to post events) is restricted to the East and West STAC catalogs.
- Read access (ability to subscribe to topics and read events) is limited to the CEDA and Globus teams who are conducting the data challenges.
- We also granted Read access to engineers for the ESGF PID team. They are evaluating the Event Stream for use in a new version of the ESGF PID service. They are using events generated from our data challenges in this evaluation. Their feedback has already led to improvements to ESGVOC's CMIP6 metadata schema.
We expect to onboard engineers for the ESGF Rapid Evaluation Framework (REF) team in August or September to enable them to explore the event stream with events from one or more ESGF-NG data challenges.
Before beginning production operation, we expect we will redeploy the service on a new Kafka cluster within the Confluent service. This should have no effect on clients other than an endpoint address change.
[No significant changes since last month.]
The STAC Transaction API (see above) authenticates all client requests and authorizes requests against active project and data node registrations. The system used for authentication and authorization is configurable.
- For the ESGF-West STAC catalog, authentication is performed via Globus, and registration is performed via Globus Group memberships.
- For the ESGF-East STAC catalog, authentication is performed via EGI-Check In and registration is performed via EGI-Check In attributes.
As of August 1, 2025, all of the above functionality is implemented and has been tested.
NOTE: The current EGI-Check In implementation uses the OIDC authorization code flow. At some point in the future (as a low-priority development), CEDA plans to change this to the OIDC device flow.
[No significant changes since last month.]
When catalog changes are requested via the STAC API (new items or updates to existing items), the API is not designed to return an error code to the calling client application if the change cannot be made in the catalog because of a consistency error. (E.g., if a new item is published and it conflicts with an existing item, or if a retraction is requested and the item does not exist.) Instead, the API always returns a code indicating a deferred operation. If the change cannot be made in the STAC catalog, the catalog will generate an error on the ESGF Event Stream's "errors" topic.
We currently do not have a plan for how to respond to errors reported on the Event Stream.
All pre-CMIP7 ESGF discovery and data access applications must be updated to use the new STAC catalog(s). Because we are now using STAC, we can now use STAC applications that have been in widespread use by the geospatial data community.
On completion of each data challenge, we have made the resulting STAC catalogs (both East and West) available for client application testing. We are counting on the ESGF XC to coordinate this testing with us and the individual client application teams. We expect this testing will result in issues that we will need to resolve (it already is!), so it's important that this testing happen as soon as possible and with as many client applications as possible.
| Application | Status |
|---|---|
| Radiant Earth STAC Browser | 2/11: Done |
| DKRZ’s customisation of the STAC Browser. (Includes support for JupyterLite and visualisations) | https://eerie.cloud.dkrz.de/ & https://gitlab.dkrz.de/data-infrastructure-services/cloudify/-/tree/main/workshop?ref_type=head |
| Metagrid | 3/31: Started in early March and still in progress |
| Intake-esgf | 3/31: Started by 2/11; still underway |
| Esg-pull | 3/31: Not started, might be used for data replication so would be covered as part of that effort |
| Wget-API | 3/31 - Not started |
Other tools outside our control (who need to be reached out to)
- CMCC
- Climate4Impact
- ESMValTool
- ESGF-py-client - we intend to deprecate this application
[No significant changes since last month.]
- Both catalogs:
- Both East and West have performed multiple deployments of the STAC Transaction API, Event Consumer, and STAC Discovery API for the data challenges. We are preparing documentation and plans for these deployments to be handled by local operations personnel by the time we launch the public ESGF-NG service.
- East and West are using common STAC Transaction API code (with separate configs for auth, Globus v. EGI Check-In) and common event consumer code (with separate backends: Globus Search and ElasticSearch).
- The catalogs still use separate STAC Discovery API code. (No plans to change that, presently.)
- Code to support controlled vocabulary validation is pending a feature release from the ESGVOC project.
- West catalog:
- The West catalog supports Globus authentication and authorization and uses a Globus Search backend.
- East catalog:
- The East catalog supports EGI Check-In authentication and authorization and uses an ElasticSearch backend (hosted on JASMIN).
The CEDA team is looking into moving its STAC API to the production URL.
The Globus team has begun working with our Operations team to plan production deployments of the ESGF-West core architecture including the ESGF Event Stream.
- We anticipate two production environments for ESGF-West: Production (what the ESGF community sees and uses) and Staging (the next version of the Production environment deployed for testing).
- Releases to Production and Staging will be coordinated via a GitHub tagging convention. For components shared with the ESGF-East team (STAC Transaction API and Event Consumer), we will coordinate a release tagging convention that allows each team to release independently of the other.
- Production and Staging will each have a distinct Event Stream service, allowing testing in the Staging environment without changing the contents of the Production system.
- We are targeting July 15 for availability of both Production and Staging environments.
- Our development team will continue operating development deployments separately from Production and Staging.
The Globus team is also developing a contingency plan for handing off the ESGF Event Stream to another operator in the event that we are no longer able to operate it.
[No significant changes since last month.]
Publisher registration for both East and West STAC catalogs is currently a manual process. Catalog operators must manually set up authorization policy.
[No significant changes since last month.]
The ESGF Replicator application is a planned ESGF application to be used by ESGF data node operators. When deployed and configured by an ESGF data node operator, the application will monitor the ESGF event stream for new and updated datasets. When datasets are seen that match a local replication profile, the application will create replicas of those datasets on the data node. If the data node is authorized to do so, the application will also update the datasets' entries in the ESGF STAC catalog to register the new replicas.
There is currently no written technical design for the ESGF Replicator application. See the GitHub Repository for user stories, requirements, and key design points.
A new design team met for the first time on 2025-04-30 to review the materials above and begin planning an implementation strategy.
- We confirmed the approach of using esgpull as the basis for the replicator. This will represent a new usage mode for esgpull. Specifically, instead of the current mode which performs a catalog search for relevant datasets, this new mode will review new publication events on the ESGF event stream to identify relevant datasets.
- Several of the features on the replicator list are already on esgpull's agenda, and we'll rely on the existing team to implement these, with possible help from the replicator team.
- The ESGF2-US/Globus team will contribute Globus Transfer support to esgpull.
- The ESGF-NG team (mainly ESGF2-US and CEDA) will contribute event stream support (as an alternative to catalog search).