Thoughts on multi-region support

[jayjanssen]

I'm using MiniStack to back local acceptance tests for an Aurora provisioner that supports multi-region global clusters. Today MiniStack flattens all requests into a single region, so I can't acceptance test outside of single-region clusters. This removes some of the advantage of using MiniStack, since we have a lot of still very slow real AWS tests to run for our multi-region stuff.

Myself and a colleague have been thinking about this and have an experimental multi-region patch for MiniStack that passes our acceptance tests and gives a general shape of what would be needed to support multi-region, though obviously this is only the services we have concerns with. One thought to avoid backward compatibility issues was to harness this up to wire up an opt-in (MULTISTACK_MULTI_REGION=1) gate, though it's not particularly pretty.

We're just trying to determine what our next step is here and wanted get your thoughts first @Nahuel990. Totally understand if it's a hard "no", but felt it was worth asking and not assuming.

[Nahuel990]

@jayjanssen thanks for bring this and the POC.

On the framing: we rarely give a "hard no" on anything unless it doesn't make sense to add or moves us further away from AWS. Multi-region is the opposite of that, so this was always going to be a yes in principle. MAin reason we didn't add it yet, was nobody actually raised it

The direction looks good...The ARN parser and persistence scoping should probably land before service work since every later PR will depend on them

I also agree this needs to be split up. 4k lines across 9 services is too much for a clean review. I'd probably do it in stages (same PR but just split it): ARN parser first, then persistence + region scoping then then the remaining services as one bundle

MINISTACK_MULTI_REGION=1 is good as a transition, but I think we should aim to a default-on milestone at some point (once this is finished), otherwise the flag becomes permanent and people will keep getting region-flat behavior without realizing it

The other thing to be careful about is the inconsistency window. With only a subset of services region-aware, tests can silently fall back to flat behavior depending on what they touch. Personally I'd rather fail fast there than allow silent inconsistencies

Last thing: the agent-generated parts should be verified against botocore models and AWS docs, especially Aurora Global Clusters since the behavior there is a bit subtle if you raise a PR I do have a local process for that so I will review it at the moment the PR lands

If the order sounds reasonable, opening the ARN parser PR first is probably the cleanest next step, feel free to raise any concern we are open to options

[Nahuel990]

Cool! there are also docker previews builds for each branch (on each commit) so we can definitly fully test not only the build but how it lands in the image... I think you can just create the branches you need, no point in me creating a separate branch

[Areson]

Will do. I'll make a branch to land this work against and start working on some incremental commits so that we can manage the review.

[jayjanssen]

Thanks @Nahuel990 this sounds great. @Areson will likely be the main implementor from our side

[Areson]

@Nahuel990 Do you want to go ahead and make a branch for us to work against?

[Nahuel990]

@Areson sorry for the delay you can target feat/multi-region and do PRs against it

[Areson]

@Nahuel990 Now that 1.4.0 is out, how to you want me to proceed? I noticed that you closed the two multi-region PRs that I had open. Should I re-open them against main or do you want to put together another feature branch?

[Areson]

I just saw that you had merged them both directly in. For future work do you want PRs against main or another feature branch?

[Nahuel990]

@Areson sorry yes when I merge the multi-region branch automatically removed those PRs so I cherry picked them directly... no, we are good from now on to target directly main... thanks for the feature!