Increase resources, k8s setting to handle overwhelming request

[0yukali0]

Tracking issue

Close #7939

Why are the changes needed?

flyte-benchmark triggers overloading problem of k8s API server, OOM of rustfs and flute-binary.

What changes were proposed in this pull request?

Set default reserved resources for flyte and rustfs.

How was this patch tested?

Running experiment with parameters with -k 10 -n 2000 to trigger the issue.
I recommended that environment 20core and 28G to run the benchmark with parameters -k 25 -n 2000 in devbox.

# git clone flyte-benchmark
uv run scripts/v2/swarm.py    --k 10 --n 2000 #  9core 26G
uv run scripts/v2/swarm.py    --k 25 --n 2000 #  24core 64G

Labels

• fixed: For any bug fixed.

Setup process

My environments includes 9 core, 26G memory and 100G disk. (Colima instance)

1. Setting resources in _common.py in benchmark.

def task_env(name):
    """Parent environment for one benchmark shape."""
    return flyte.TaskEnvironment(
        name=name,
        image=image,
        resources=flyte.Resources(cpu=("50m", "80m"), memory=("300Mi", "512Mi")),
        depends_on=[sleep_env],
    )

2. Setting up devbox.

# git clone flyte
make devbox-build
make devbox-run

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Related #7757

Stack

If you do use git town to manage PR Stacks, the stack relevant to this PR
will show below. Otherwise, you can ignore this section.

Docs link

[popojk]

Suggested change

	reqeusts:
	requests:

[popojk]

It seems to be be a typo in values.yaml, making the resources settings not valid in actual deployment. Could you please:

1. Fix the typo and test it again
2. Run make devbox-build to generate new manifest files under docker/devbox-bundled/manifests

Thanks!

[kumare3]

Are you running the benchmark on devbox?

[0yukali0]

Are you running the benchmark on devbox?

Yes, I ran the benchmark in devbox to validate the result of #7757

[0yukali0]

It seems to be be a typo in values.yaml, making the resources settings not valid in actual deployment. Could you please:

1. Fix the typo and test it again
2. Run make devbox-build to generate new manifest files under docker/devbox-bundled/manifests

Thanks!

Thanks for review, I will update it and validate it later.

[0yukali0]

[0yukali0]

Hi @popojk, thanks for review.
I reconsidered the problem. I will keep the k3s setting and remove the default resources in devbox.

The flyte-binary and RustFS resource issues are sparked by my Mac’s limited resources when running lots of task actions.
I don’t think the default resource requirements should be increased for everyone, as this would make Devbox harder for new users to get started with.