Supports the scheduling capabilities of SubGroup

[ouyangshengjia]

What type of PR is this?

/kind feature

What this PR does / why we need it:

1. Divide the pods in a PodGroup into multiple SubGroup. Each SubGroup has its own network topology constraint.
2. Schedule pods in the PodGroup according to network topology constraints at the PodGroup and SubGroup levels.

Which issue(s) this PR fixes:

Partial fixes for #4188

Special notes for your reviewer:

@wangyang0616 @JesseStutler @hzxuzhonghu

Does this PR introduce a user-facing change?

Example YAML for vcJob after modification

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: network-topology-job
spec:
  minAvailable: 6
  schedulerName: volcano
  networkTopology:
    mode: hard
    highestTierAllowed: 2
  tasks:
    - replicas: 6
      name: "task"
      partitionPolicy:
        totalPartitions: 2
        partitionSize: 3
        minPartitions: 2
        networkTopology:
            mode: hard
            highestTierAllowed: 1
      template:
        metadata:
          name: task
        spec:
          containers:
            - image: ubuntu
              imagePullPolicy: IfNotPresent
              name: task
              resources:
                requests:
                  cpu: "2"
                  memory: 2Gi
          restartPolicy: OnFailure

Example YAML for podgroup after modification

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: network-topology-podgroup
spec:
  minMember: 6
  networkTopology:
    mode: hard 
    highestTierAllowed: 2
  subGroupPolicy: 
    - subGroupSize: 3
      minSubGroups: 2
      name: task
      matchPolicy:
        - labelKey: volcano.sh/task-subgroup-id
      networkTopology:
        mode: hard 
        highestTierAllowed: 1

[gemini-code-assist]

Summary of Changes

Hello @ouyangshengjia, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the Volcano scheduler's capabilities by introducing a hierarchical scheduling model centered around 'PodBunches'. This new abstraction allows for the definition and enforcement of network topology constraints at a finer granularity than before, enabling more sophisticated and efficient resource allocation for complex workloads. The changes involve substantial modifications to the scheduler's internal data structures and allocation algorithms to accommodate the new PodBunch concept and ensure proper state management.

Highlights

• Hierarchical Pod Scheduling with PodBunches: Introduces the concept of 'PodBunches' to allow for more granular and hierarchical scheduling within a PodGroup. Pods can now be divided into multiple PodBunches, each with its own network topology constraints.
• Network Topology Awareness: Extends network topology scheduling capabilities to the PodBunch level, in addition to the existing PodGroup level. This enables scheduling decisions to consider network constraints for smaller groups of pods.
• Refactored Allocation Logic: The core allocation logic has been significantly refactored to support PodBunches. This includes new data structures like JobWorksheet and PodBunchWorksheet to organize tasks, and new functions for allocating resources at the job and podbunch levels.
• Scheduling Decision Recorder: A new Recorder component has been added to track and manage scheduling decisions for jobs and podbunches, including their allocated HyperNodes. This helps in maintaining state and recovering information across scheduling cycles.
• Enhanced HyperNode Management: The HyperNodeInfo structure now includes parent-child relationships, and the session initialization logic has been improved to handle a virtual 'ClusterTopHyperNode' and recover/validate allocated HyperNode information for jobs and podbunches.
• Temporary Preemption Restriction: Preemption is temporarily disabled for jobs that utilize the new network topology features, indicating a current limitation or ongoing development in this area.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces significant changes to support PodBunch scheduling capabilities, enhancing network topology awareness within the Volcano scheduler. The changes involve new data structures (allocateContext, JobWorksheet, PodBunchWorksheet, PodBunchInfo) to manage groups of pods with specific network topology constraints. The core allocation logic has been refactored to handle these PodBunches, including new functions for organizing, allocating, and selecting the best hypernodes for jobs and individual pod bunches. Additionally, the HyperNodeInfo structure has been extended to include parent-child relationships, improving the hierarchical representation of network topology. A new Recorder component is introduced to track allocation decisions. Test cases have been updated to reflect these new functionalities and configurations.

Overall, the changes are substantial and well-structured, addressing a complex scheduling requirement. However, there are a few areas that could benefit from clarification or further refinement to ensure robustness and full adherence to intended behavior, particularly concerning the handling of fit errors and the NeedContinueAllocating logic for jobs with BunchPolicy.

[gemini-code-assist]

The comment // todo: job contains bunch policy does not supports the strategies below and the subsequent return true indicate a known limitation. For jobs with BunchPolicy, the detailed logic for checking task role minimums and other strategies is bypassed, causing NeedContinueAllocating to always return true.

This could lead to situations where scheduling continues even if specific pod bunch or task role constraints are not met, potentially resulting in inefficient allocations or unexpected behavior. This todo should be addressed to ensure that jobs with BunchPolicy are evaluated against their granular requirements.

[ouyangshengjia]

The function NeedContinueAllocating is a fast-fail judgement when allocating pods in a job. Not supporting this performance optimization feature in SubGroup scheduling scenarios does not cause an error. It would be better to support this feature through a seperate ISSUE/PR.

[gemini-code-assist]

The job.ResetFitErr() call here clears all fit errors for the entire job before evaluating a new hypernode candidate. Subsequently, within the loop for each pod bunch, job.ResetPodBunchFitErr(podBunch.UID) is called (L419).

Since job.ResetFitErr() already clears the entire NodesFitErrors map, the job.ResetPodBunchFitErr call for each pod bunch becomes redundant in this context. While not a functional bug, it adds unnecessary operations. Consider if job.ResetFitErr() should be more granular or if job.ResetPodBunchFitErr is intended for a different scenario not covered by the current outer loop structure.

[ouyangshengjia]

ResetPodBunchFitErr clears all fit errors for the podBunch in the job before a new bunch-level hyperNode allocate attemption.

[gemini-code-assist]

The ready atomic boolean is initialized to true here. However, the actual building of the hypernode cache (which populates hyperNodes, hyperNodesSetByTier, realNodesSet) happens later via UpdateHyperNode calls. If the scheduler attempts to use hypernode information (e.g., by checking HyperNodesReadyToSchedule) before the cache is fully populated and UpdateHyperNode has successfully completed its initial build, it might operate on incomplete or uninitialized data.

It would be safer to initialize ready to false and only set it to true after the first successful build of the hypernode cache, ensuring that the scheduler only proceeds when the data is reliably ready.

[ouyangshengjia]

the ready flag should be true if there is no hypernode in the cluster.

[Copilot]

Pull Request Overview

This PR adds support for scheduling capabilities of SubGroup (PodBunch) within PodGroup. It divides pods in a podgroup into multiple podbunches, each with its own network topology constraint, and schedules pods according to network topology constraints at both podgroup and podbunch levels.

Key Changes

• Introduced PodBunchInfo to represent groups of tasks within a job with their own topology constraints
• Added podBunch-level scheduling functions (PodBunchReady, PodBunchPipelined, PodBunchOrder, HyperNodeGradient)
• Refactored allocation logic to support hierarchical job/podBunch scheduling with network topology

Reviewed Changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/scheduler/api/pod_bunch_info.go	New file defining PodBunchInfo structure and related logic
pkg/scheduler/api/job_info.go	Added podBunch management and network topology helper methods
pkg/scheduler/api/types.go	Updated function signatures to use PodBunchInfo instead of JobInfo
pkg/scheduler/framework/session.go	Added ClusterTopHyperNode, podBunch functions, and allocated hyperNode recovery
pkg/scheduler/actions/allocate/allocate.go	Complete refactoring to support hierarchical job/podBunch allocation
pkg/scheduler/plugins/gang/gang.go	Added podBunch order, ready, and pipelined functions
pkg/scheduler/plugins/priority/priority.go	Added podBunch order function
pkg/scheduler/plugins/network-topology-aware/network_topology_aware.go	Added hyperNode gradient and binpacking logic for podBunches
pkg/scheduler/cache/cache.go	Added updateJobInfo for updating allocated hyperNodes
pkg/scheduler/util/scheduler_helper.go	Updated to return both list and set of nodes by hyperNode

Comments suppressed due to low confidence (3)

pkg/scheduler/plugins/network-topology-aware/network_topology_aware.go:1

• Corrected spelling of 'hypeNode' to 'hyperNode' in log message.

/*

pkg/scheduler/api/hyper_node_info.go:1

• When addChild returns an error at line 442, the function exits early but parentHn.Children.Insert(member) at line 461 is never executed. However, when the error is returned at line 456, childHn.Parent may have already been set at line 460, creating an inconsistent state. The Parent field should only be set after all validations pass.

/*

pkg/scheduler/framework/session.go:1

• The condition nta.maxTier < tier will cause ZeroScore to be returned when the tier exceeds maxTier. However, this doesn't prevent potential division by zero in line 501 if minTier equals maxTier and the first condition isn't caught. The logic should ensure minTier < maxTier before division.

/*

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull Request Overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The TODO comment 'todo handle collision' indicates unfinished collision handling logic for hash-based SubJobID generation. This could lead to incorrect SubJob identification if collisions occur. Consider implementing proper collision detection or using a more robust ID generation strategy (e.g., UUID or cryptographic hash).

[Copilot]

The TODO comment 'todo: job contains subJob policy does not supports the strategies below' indicates incomplete implementation for jobs with SubJob policies. The function returns true without checking failed roles when ContainsSubJobPolicy() is true, which may bypass important allocation logic. Consider implementing the full allocation strategy for jobs with SubJob policies.

Suggested change

	if ji.ContainsSubJobPolicy() {
	return true
	}
	// Allocation logic for jobs with SubJob policies is now handled below.

[ouyangshengjia]

The function NeedContinueAllocating is a fast-fail judgement when allocating pods in a job. Not supporting this performance optimization feature in SubGroup scheduling scenarios does not cause an error. It would be better to support this feature through a seperate ISSUE/PR.

[Copilot]

The TODO comment indicates that only ExactMatch selectors are currently supported, while other selector methods (e.g., regex or label-based matching) are not implemented. This limitation should be documented or the other selector types should be implemented to avoid unexpected behavior.

Suggested change

	if member.Selector.ExactMatch == nil { // todo support other selector method
	if member.Selector.ExactMatch == nil { // Only ExactMatch selectors are currently supported; other selector methods (e.g., regex or label-based matching) are not implemented.
	fmt.Printf("Warning: Only ExactMatch selectors are supported in createSchedulerCache; selector for member %v will be ignored.\n", member)

[ouyangshengjia]

I think retain todo tag is better

[Copilot]

[nitpick] This check duplicates similar logic that existed in the original code but was removed from the allocate loop. Consider centralizing the HyperNodesReadyToSchedule check to avoid scattered validation logic and potential inconsistencies.

[ouyangshengjia]

HyperNodesReadyToSchedule will not be changed within a session, so there is no potential inconsistencies.

[wangyang0616]

The podBunch field should no longer be present after the update; it's recommended to refresh the PR description.

[wangyang0616]

Please clearly describe your next steps. It would be even better if you had issue tracking information. Please also include your GitHub username for easier communication with other contributors.

[ouyangshengjia]

done. related to issue: #4374

[JesseStutler]

Should this PR be updated with a new design doc so that more contributors can understand the current framework? Our framework seems to be getting increasingly complex.

[wangyang0616]

/lgtm

[JesseStutler]

/approve
/lgtm

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JesseStutler

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JesseStutler]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[yccharles]

Does the subGroup support different priorityClass with PodGroup in future ? @ouyangshengjia @JesseStutler

Consider such a scenario： In one job，

• set some task's priorityclass to guaranteed
• set the other task's priorityclass to best-effort

this scenario support job can use elastic resource or idle resource in k8s cluster. And those best-effort tasks can be preempting in any time without manual operation or job’s autoscaling.

[hzxuzhonghu]

@yccharles What's your use case, please file a new issue? So others can see it and discuss there

[JesseStutler]

Does the subGroup support different priorityClass with PodGroup in future ? @ouyangshengjia @JesseStutler

Consider such a scenario： In one job，

• set some task's priorityclass to guaranteed
• set the other task's priorityclass to best-effort

this scenario support job can use elastic resource or idle resource in k8s cluster. And those best-effort tasks can be preempting in any time without manual operation or job’s autoscaling.

I think it's a good suggestion! @yccharles