[+] implement parallel source discovery

[pashagolub]

Improves dead-source handling with parallel resolution and instance_up=0 on discovery failure.

Sources.ResolveDatabases() previously resolved each source sequentially. A single slow or unresponsive source (e.g. a continuous-discovery endpoint behind a firewall) would block discovery of all subsequent sources for the full connection timeout duration.

Sources are now resolved concurrently using sync.WaitGroup.Go(). Results are collected into a pre-allocated indexed slice to preserve deterministic ordering. Per-source error logging with source name is included in the resolver itself.

When a SourcePostgresContinuous or SourcePatroni source fails to resolve any databases, LoadSources() now emits instance_up=0 to the configured sinks. This makes the failure visible in dashboards and alerting, consistent with how unreachable directly-monitored sources are handled.

[coveralls]

Coverage Report for CI Build 25120163997

Coverage decreased (-0.3%) to 83.065%

Details

• Coverage decreased (-0.3%) from the base build.
• Patch coverage: 23 uncovered changes across 2 files (6 of 29 lines covered, 20.69%).
• 6 coverage regressions across 1 file.

Uncovered Changes

File	Changed	Covered	%
internal/sources/resolver.go	21	0	0.0%
internal/reaper/reaper.go	6	4	66.67%

Coverage Regressions

6 previously-covered lines in 1 file lost coverage.

File	Lines Losing Coverage	Coverage
internal/reaper/reaper.go	6	35.53%

---

Coverage Stats

Relevant Lines:	5338
Covered Lines:	4434
Line Coverage:	83.06%
Coverage Strength:	0.95 hits per line

---

💛 - Coveralls

[0xgouda]

Is this related to #1377?

[pashagolub]

Is this related to #1377?

In some way. I couldn't reproduce that issue but I found that misconfigured discovery sources could cause a huge delays.

[0xgouda]

I have a concern here that can be reproduced with the following steps:

1. Define a target that happens to be unreachable and is of the kind postgres-continuous-discovery
2. pgwatch writes instance_up=0 for the target for a while with dbname = sourceName
3. The target becomes alive
4. pgwatch runs for a while and now writes the updated instance_up = 1, but with a new dbname = sourceName + _ + realDbname
5. The target is down again and its instance_up = 0 is written with dbname = sourceName + _ + realDbname

So the full instance uptime history becomes a bit disconnected, with different dbname[s].

But generally, I think that's the best we can do, just wanted to note this behaviour.

[pashagolub]

Good catch! We could use source instead of dbname. This way we will know for sure at which point that happened

[0xgouda]

Can you explain more?

[pashagolub]

// WriteInstanceDown writes instance_up = 0 metric to sinks for the given source
func (r *Reaper) WriteInstanceDown(md *sources.SourceConn) {
	r.measurementCh <- metrics.MeasurementEnvelope{
		DBName:     md.Name,
		MetricName: specialMetricInstanceUp,
		Data: metrics.Measurements{metrics.Measurement{
			metrics.EpochColumnName: time.Now().UnixNano(),
			"kind": string(md.Kind),
		//  ^^^^^^^^^^^^^^^^^^^^^^^
			specialMetricInstanceUp: 0},
		},
	}
}

[pashagolub]

this way grafana could distinguish regular databases vs all other

[0xgouda]

I don't think adding the kind to the measurements will help.

What if we have 2+ postgres-continuous-discovery/patroni sources? Grouping by kind then will be useless.

[0xgouda]

And again, I guess what we have already is the best we can do, since we won't be able to add any instance identifiers (i.e., sys_id) to the measurements before connecting at least once.

But then, in Grafana, one might have 2 panels like:
instance uptime history grouped by sys_id
and
instance uptime history for instances with unknown sys_id

and they should complement each other, to give the full uptime history view.