Sync Summary

[Lloyd K (sw-lkinsella)]

Is there any way of getting the more verbose sync summary on Cloud? At the moment I'm only seeing https://api.airbyte.com/v1/jobs/{jobId} which includes a very high level summary of rows/bytes.

In the raw logs though is a JSON dump of the entire sync.

It would be really useful to be able to get at this.

For example:

024-12-09 15:07:09 info sync summary: {
  "status" : "completed",
  "recordsSynced" : 78,
  "bytesSynced" : 53522,
  "startTime" : 1733756523353,
  "endTime" : 1733756829882,
  "totalStats" : {
    "bytesCommitted" : 53522,
    "bytesEmitted" : 53522,
    "destinationStateMessagesEmitted" : 83,
    "destinationWriteEndTime" : 1733756827132,
    "destinationWriteStartTime" : 1733756523802,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 1,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 111,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 111,
    "recordsEmitted" : 78,
    "recordsCommitted" : 78,
    "recordsFilteredOut" : 0,
    "bytesFilteredOut" : 0,
    "replicationEndTime" : 1733756827172,
    "replicationStartTime" : 1733756523353,
    "sourceReadEndTime" : 1733756576114,
    "sourceReadStartTime" : 1733756523804,
    "sourceStateMessagesEmitted" : 83
  },
  "streamStats" : [ {
    "streamName" : "table_a",
    "streamNamespace" : "public",
    "stats" : {
      "bytesCommitted" : 0,
      "bytesEmitted" : 0,
      "recordsEmitted" : 0,
      "recordsCommitted" : 0,
      "recordsFilteredOut" : 0,
      "bytesFilteredOut" : 0
    }
  }, {
    "streamName" : "table_b",
    "streamNamespace" : "public",
    "stats" : {
      "bytesCommitted" : 0,
      "bytesEmitted" : 0,
      "recordsEmitted" : 0,
      "recordsCommitted" : 0,
      "recordsFilteredOut" : 0,
      "bytesFilteredOut" : 0
    }
  ...

[Airbyte Support Bot (airbyte-support-bot)]

Hi! This discussion hasn't had activity in over a year. Is this still relevant to you? If so, please reply and we'll keep it open. If we don't hear back within 14 days, we'll close this as part of our housekeeping. You can always reopen or create a new discussion later. Thanks!

---

Devin session

[M. Clay Johns (johnsmclay)]

This would be amazingly useful bit of data. Allowing us to track usage on a granular (stream) level helps us make sure we can detect and manage runaway jobs and things on our end that generate a ton of unnecessary changes. Especially since sources like Postgres may have 40+ streams and your Cloud UI doesn't have useful tracking or even a way to order the stats from the last sync by rows/bytes.