ESGF-NG status as of May 5, 2025.
NOTE: ESGF-NG will be used for CMIP7 and subsequent activities. None of the below applies to CMIP6 or earlier activities.
ESGF-NG is the next generation of the core architecture for the Earth System Grid Federation. Designed in 2024 and deployed in 2025, it replaces the ESGF-1.0 core architecture. See ESGF-NG Docs for further information.
Our current work is still consistent with the design as written.
| The Past | Activity |
|---|---|
| 2019-11 | ESGF Future Architecture workshop held in UK - Report |
| 2024-04 | ESGF-NG proposed at 10th ESGF conference in Bethesda, MD (USA) |
| 2024-11 | ESGF mini-conference in Abingdon (UK); Data challenges proposed and challenges 1-3 designed and scheduled for Jan/Feb 2025 |
| 2024-12-13 | Globus performs Data Challenge 01 round 1. (Includes end-to-end basic CMIP6 dataset publication.) |
| 2025-01-30 | CEDA performs Data Challenge 01 round 1. |
| 2025-03-25 | Data Challenge 01 round 2 completed. (Includes both Globus and EGI-Check In authorization and schema-level validation.) |
The following timeline is estimated and subject to change.
| The Future | Activity |
|---|---|
| 2025-05-30 | Data Challenges 02 and 03 completed. (Includes dataset updates and retractions.) |
| 2025-05-30 | Data Challenge 01 round 3 completed. (Includes controlled vocabulary validation, including support for projects beyond CMIP6.) |
| 2025-05-30 | Data Challenge 05 completed. (Includes event consumer error reporting on the event stream's error topic.) |
| 2025-06-30 | Data Challenge 04 completed. (Includes OBS4MIPS dataset publication with schema and controlled vocabulary validation.) |
| June 15-July 15 | Onboard ESGF REF developers to ESGF event stream so they can explore events from the ESGF-NG data challenges and design an integration with the ESGF REF. (See Event Stream below.) |
| 2025-07-01 | Target for establishing production ESGF-NG publication capability (STAC catalogs, event stream, publisher and related client tools) |
The ESGF-NG team is currently conducting a series of data challenges (trial runs of the ESGF-NG system) to validate the expected behavior of the end-to-end ESGF-NG system. The first section below contains updates on these data challenges. Subsequent sections provide technical details about the state of the STAC catalogs, the ESGF discovery and data access applications, the ESGF event stream, and the ESGF replicator application.
- Data Challenge 1 is an end-to-end publication test. CEDA and Globus each publish 500 unique datasets (drawn from CMIP6) to their own STAC catalog. When finished, the two STAC catalogs are compared to verify that they are synchronized and have all 1000 catalog entries.
- Round 1 - Globus and CEDA have both completed rounds 1 and 2 with 500 datasets each (one duplicate). The duplicate dataset was detected and rejected by both catalogs. The asset lists for all datasets were different because of an encoding/decoding issue in the West catalog.
- Round 2 - Globus and EGI-Check In authorization worked. Schema validation worked for positive cases, but the input data contained no negative cases. We will include negative cases in Round 3. (See timeline above.)
- Round 3 - ESGVOC code is available from IPSL/CEDA. We performed an initial integration with the STAC transaction API in April, but have subsequently decided to use a different approach. The new approach requires ESGVOC to generate STAC extensions for onboarded ESGF projects, which we'll use in the STAC transaction API. We're currently waiting on ESGVOC to implement this new feature. Support in the STAC transaction API will be greatly simplified by this approach.
- Data Challenge 2 is an end-to-end publication and retraction test. We've completed the code changes to support updates (inc. retraction) in the STAC Transaction API and Event Consumer components. Both ESGF2-US and CEDA expect to run this challenge by the end of May.
- Data Challenge 3 is an end-to-end publication and update test. We've completed the code changes to support updates (inc. retraction) in the STAC Transaction API and Event Consumer components. Both ESGF2-US and CEDA expect to run this challenge by the end of May.
- Design of Data Challenges 4 and beyond is underway. Challenges 04 and 05 will involve support for projects beyond CMIP6 and support for the ESGF Publication Errors event stream. We hope to onboard the ESGF Rapid Evaluation Framework team to an Event Stream used in these data challenge so they can begin planning an integration with the event stream.
- Both catalogs:
- Both East and West have performed multiple deployments of the STAC Transaction API, Event Consumer, and STAC Discovery API for the data challenges. We are preparing documentation and plans for these deployments to be handled by local operations personnel by the time we launch the public ESGF-NG service.
- East and West are using common STAC Transaction API code (with separate configs for auth, Globus v. EGI Check-In) and common event consumer code (with separate backends: Globus Search and ElasticSearch).
- The catalogs still use separate STAC Discovery API code. (No plans to change that, presently.)
- Support for item updates and retractions is being tested presently.
- Code to support error reporting on the event stream is in development.
- Code to support controlled vocabulary validation is pending a feature release from the ESGVOC project.
- West catalog:
- The West catalog supports Globus authentication and authorization and uses a Globus Search backend.
- The Globus ESGF team is also preparing the processes and documentation necessary to transition deployment and operation of these components to the Globus operations team.
- East catalog:
- The East catalog supports EGI Check-In authentication and authorization and uses an ElasticSearch backend (hosted on JASMIN).
The ESGF STAC catalogs will enforce metadata schema and controlled vocabulary validation for all catalog additions and updates.
- We tested an initial implementation of CMIP6 schema and controlled vocabulary validation and determined that the approach required too much custom code. We are now pursuing a new approach that heavily leverages existing STAC community code.
- The new approach requires the ESGVOC module (where new ESGF project schemas and controlled vocabularies are registered) to automatically generate (and update) STAC extensions for each ESGF project. Our STAC transaction API can then rely on existing community code to validate item metadata against the matching STAC extension.
- We are now waiting for the ESGVOC team to add the new feature to ESGVOC and generate a fresh STAC extension for CMIP6 (and other projects as they're registered).
[No significant changes since last month.]
Publisher registration for both East and West STAC catalogs is currently a manual process. Catalog operators must manually set up authorization policy.
[No significant changes since last month.]
When catalog changes are requested via the STAC API (e.g., new items, item updates including retractions), the API is not designed to return an error code to the calling client application if the change cannot be made in the catalog because of a consistency error. (E.g., if a new item is published and it conflicts with an existing item, or if a retraction is requested and the item does not exist.) Instead, the API will return a code indicating a deferred operation and the STAC catalog will generate an error on the ESGF event stream's "errors" topic.
We currently do not have a plan for what to do with errors reported on the event stream.
The ESGF Replicator application is a planned ESGF application to be used by ESGF data node operators. When deployed and configured by an ESGF data node operator, the application will monitor the ESGF event stream for new and updated datasets. When datasets are seen that match a local replication profile, the application will create replicas of those datasets on the data node. If the data node is authorized to do so, the application will also update the datasets' entries in the ESGF STAC catalog to register the new replicas.
There is currently no written technical design for the ESGF Replicator application. See the GitHub Repository for user stories, requirements, and key design points.
A new design team met for the first time on 2025-04-30 to review the materials above and begin planning an implementation strategy.
- We confirmed the approach of using esgpull as the basis for the replicator. This will represent a new usage mode for esgpull. Specifically, instead of the current mode which performs a catalog search for relevant datasets, this new mode will review new publication events on the ESGF event stream to identify relevant datasets.
- Several of the features on the replicator list are already on esgpull's agenda, and we'll rely on the existing team to implement these, with possible help from the replicator team.
- The ESGF2-US/Globus team will contribute Globus Transfer support to esgpull.
- The ESGF-NG team (mainly ESGF2-US and CEDA) will contribute event stream support (as an alternative to catalog search).
[No significant changes since last month.]
The ESGF event stream service is operated by Globus as part of the ESGF2-US project. We are currently using Confluent Data Streaming, a managed Apache Kafka deployment hosted in AWS.
The event stream is currently being used for our ESGF-NG data challenges.
- Write access (ability to post events) is restricted to the East and West STAC catalogs.
- Read access (ability to subscribe to topics and read events) is limited to the CEDA and Globus teams who are conducting the data challenges.
Before beginning production operation, we expect we will redeploy the service on a new Kafka cluster within the Confluent service. This should have no effect on clients other than an endpoint address change.
We are aiming for May/June to onboard engineers for the ESGF Rapid Evaluation Framework (REF) team to enable them to explore the event stream with events from one or more ESGF-NG data challenges.
[No significant changes since last month.]
All pre-CMIP7 ESGF discovery and data access applications must be updated to use the new STAC catalog(s). Because we are now using STAC, we can now use STAC applications that have been in widespread use by the geospatial data community.
| Application | Status |
|---|---|
| Radiant Earth STAC Browser | 2/11: Done |
| DKRZ’s customisation of the STAC Browser. (Includes support for JupyterLite and visualisations) | https://eerie.cloud.dkrz.de/ & https://gitlab.dkrz.de/data-infrastructure-services/cloudify/-/tree/main/workshop?ref_type=head |
| Metagrid | 3/31: Started in early March and still in progress |
| Intake-esgf | 3/31: Started by 2/11; still underway |
| Esg-pull | 3/31: Not started, might be used for data replication so would be covered as part of that effort |
| Wget-API | 3/31 - Not started |
Other tools outside our control (who need to be reached out to)
- CMCC
- Climate4Impact
- ESMValTool
- ESGF-py-client - we intend to deprecate this application