使用说明与事实红线(先看这一页)
How to use this handbook & fact red lines (read this first)
这份手册把调度器系列五篇深度文档(Kueue 原理、Volcano 原理、两者对比、协同架构、AI/GPU 调度方案)与 Firmus JD-A 的岗位要求融合,面向英文面试。中文供理解脉络,🇬🇧 英文是可以直接说出口的话术,建议背熟。岗位是 JD-A:Senior AI Engineer (Kubernetes & Customised Scheduler),AI & Applications / Model-to-Grid,自研 K8s 原生专属调度器。若主攻切换到 JD-C(平台底座岗),Volcano/Kueue 融合、功耗/电网、FCAS/DSX、NVL72 白板约七成内容不再考,不要用错口径。
🇬🇧 This is the 'technical-depth enhanced edition' for JD-A, used alongside the JD-C primary handbook. In an interview, use only the current primary track — never mix the two material lines.
1.1 六条事实红线(会被圈内人当场核查)
1.1 Six fact red lines (a practitioner will check these live)
| 绝对不要说 NEVER SAY | 正确口径 CORRECT LINE |
|---|---|
| NVL72 机内有“多个 NVSwitch 域” / TP 锁同一 NVSwitch 域、PP/DP 跨域走 NVLink 主干 | 一个 NVL72 rack = 单一 72-GPU NVLink 域(all-to-all,~130 TB/s);机内无多域。集群里是“多个 rack 级域”。≤72 卡作业整体落在单 rack,scale-up 走 NVLink,跨 rack 才走 IB/Spectrum-X。 |
| “没人有多年 NVL72 生产经验” / “GB300 太新” | 墨尔本站点已在生产跑 GB300 NVL72(2026-09-28 Meta 新闻稿)。准确说法:舰队仍在 build phase,缺经验的是“我”,不是市场。 |
| #140097 / #140306 / #140143 说成 merged;#101822 说成 203 条评论 | 只有 #140308(v1.38)/ #101822 / #100853 / #101859 / drain 系列讲 shipped;#140097 是 open WIP 复现器;#140306/#140143 closed 未合入;#101822 是 165 条 review。 |
| “K8s 调度器参与秒级 FCAS / 频率事件” | 秒级 FCAS 归 UPS、电力电子、硬件联锁;K8s 调度器只做分钟级及以上的软塑形,且只作用于尚未启动的作业。 |
| “Kueue 不支持 Gang Scheduling” | Kueue 通过准入层 all-or-nothing + PodsReady 超时驱逐提供“作业级 gang 语义”;Volcano 提供“调度级 gang”(PodGroup 占坑后原子绑定)。层级不同,不是有无。 |
| Volcano 有 6 个 Action(含 recycle);Scheduling Framework 12 个扩展点;DRA 资源可被抢占 | Volcano 官方 5 个 Action:enqueue/allocate/preempt/reclaim/backfill;Scheduling Framework 13 个扩展点(含 PreEnqueue);DRA 不支持 Pod 抢占。 |
1.2 三条表述纪律
1.2 Three phrasing disciplines
- 数字必须带口径:凡“降本 30%”“成功率提升”,都要能答“分母、基期、窗口、采集来源”。手册中以【】标注的占位符必须换成真实数字。🇬🇧 Every number needs a basis: for '30% cost-down' or 'success-rate up' you must answer 'denominator, baseline, window, source'. Replace every 【】 placeholder with a real, traceable number.
- 官网动态计数器数值(0.6 PUE / 19 NVL / ~9%)一律不引用;只用可交叉验证的硬数字:32 NVL72 racks、2 Scale Units、288 MW Bell Bay、900+ MW、7 站 4 国 / 2 在运 + 5 在建。🇬🇧 Never quote the site's animated counters (0.6 PUE / 19 NVL / ~9%). Only cross-checkable hard numbers: 32 NVL72 racks, 2 Scale Units, 288 MW Bell Bay, 900+ MW, 7 sites/4 countries (2 live + 5 in build).
- “No colocation, no containers”里的 containers 指预制集装箱式数据中心模块,不是 Docker/K8s 容器——别读成“这家公司不用 K8s”。🇬🇧 In 'No colocation, no containers', containers means prefabricated containerised data-centre modules, not Docker/K8s containers — don't read it as 'they don't use Kubernetes'.
作战地图:岗位 × 证据 × 缺口
Battle map: role × evidence × gap
核心矛盾来自 Giuseppe(用人经理)的质疑:这岗要“own a production, topology- and power-aware scheduler on NVL72-class hardware”,而你没有 NVL72 生产实操。整份手册的策略,就是把这条质疑从“你没做过 NVL72”改写为“你已掌握调度器控制面全栈、且能把它接到 AI Factory 的功耗/热/电网遥测上——这是 NVL72 调度器 80% 的工程难点,剩下 20% 硬件规格上手即会”。
2.1 JD-A 核心职责逐条映射
2.1 JD-A core responsibilities, mapped one by one
| JD 要求(英文原词) | 你的证据 / 本手册对应章节 |
|---|---|
| Build proprietary K8s-native job scheduling platform; CRDs, controllers, admission webhooks, scheduling plugins, APIs, CLI | 自研 Permit 阶段 gang(§9.2)+ kube-scheduler 插件工程(§3)+ 上游 #101822/#100853(§9.1) |
| Topology-aware placement: GPU locality, NVLink/NVSwitch, NUMA, NIC, RDMA, fabric, storage locality, fault-domain | 拓扑感知全章(§7)+ TAS 精确机制(§5.4)+ NVL72 口径(§1.1)+ #140308(§9.1) |
| AI-factory-resource-aware: capacity, health, storage, power limits, thermal, maintenance, reservations | 功耗/热/电网感知(§8.2)+ Model-to-Grid 减震器(§8.3)+ AdmissionCheck 设计(§8.1) |
| Workload control: admission, queueing, priorities, quotas, fair sharing, reservations, preemption, gang, co-scheduling, backfill, aging, retry, checkpoint-aware, deferred, failure recovery | Kueue 治理(§5)+ Volcano 机制(§4)+ 对比与选型(§6)+ 弹性容错(§7.4) |
| Integrate with Slurm, Slinky, vCluster, Kueue, KAI, Volcano, YuniKorn — 'rather than replace' | 协同架构(§7)+ 正交组合融合设计(§7.2)+ 调度器横评(§8.5) |
| Observability, scheduler explanations, operator views | 可观测与调度解释(§8.4)+ Kueue 可见性(§5.5) |
| Go + Python; production controllers/operators/plugins/webhooks/REST/gRPC/CLI | Go 后端(Megvii §10.4)+ 上游 Go 代码(§9.1)+ 平台服务 |
2.2 四档自评与诚实边界
2.2 Four-tier self-rating and honest boundaries
- 深度 K8s 多租户 = advanced;作业调度器 = good(自研 Permit gang + Volcano 生产级;Kueue/KAI/Slurm 选型+集成级);IaC/CI-CD + 可观测 = very good;裸金属/GPU 开通 = 4 年前小米做过、形态与今日 LLM 不同(诚实口径,不升级成“做过”)。🇬🇧 Multi-tenant K8s = advanced; job schedulers = good (custom Permit gang + production Volcano; Kueue/KAI/Slurm at selection+integration level); IaC/CI-CD + observability = very good; bare-metal/GPU provisioning = did it at Xiaomi 4 years ago, different from today's LLM form (honest — don't upgrade it to 'done').
- sig-instrumentation 与 scheduler 的统一口径:早期在 sig-instrumentation 做指标/日志(KLog),后因平台工作延伸到 sig-scheduler 的调度框架与节点生命周期;两者都属实,是时间线。🇬🇧 Unifying sig-instrumentation and scheduler: early on in sig-instrumentation on metrics/logging (KLog), later extended through platform work into sig-scheduler's framework and node lifecycle. Both are true — it's a timeline.
公司与产品栈速记(只记可交叉验证的硬事实)
Company & product stack (cross-checkable hard facts only)
Firmus 2019 年创立于澳洲,AI Factory 是垂直整合的新一代数字基础设施:设计、建造、运营全栈自研,可再生供能,把 AI Factory 当作电网资产(“make every watt count”“every watt into tokens”)。是 NVIDIA Cloud & Engineering partner。面试中多用 “your site says / your release states” 引出公司事实,不要说成已独立验证的事实。
3.1 三层产品栈
3.1 Three-layer product stack
- HyperCube™ = 物理模块(液冷 + 供电 + 机械),官网称每模块 32 个 NVL72 rack / 2 个 NVIDIA Scale Unit,NSW 预制。🇬🇧 HyperCube™ = the physical module (liquid cooling, power, mechanical); the site says each holds 32 NVL72 racks / 2 NVIDIA Scale Units, prefabricated in NSW.
- AI FactoryOS™ = 软件定义控制面(编排、GPU/热/功耗遥测、电网交互);客户侧另有 Managed Slurm 作作业调度器。🇬🇧 AI FactoryOS™ = the software-defined control plane (orchestration, GPU/thermal/power telemetry, grid interaction); customers also get Managed Slurm as their job scheduler.
- AI FactoryX = 数字孪生,runbook 先在孪生里验证再上真设备。JD-A 要做的 Model-to-Grid 是这三层之上的作业级决策层。🇬🇧 AI FactoryX = the digital twin where runbooks are validated before touching real equipment. JD-A's Model-to-Grid is the job-level decision layer above these three.
3.2 规模、客户与电网(公司自述)
3.2 Scale, customers and grid (company-stated)
- 900+ MW 已签约;7 座 AI Factory / 4 国(澳、新、印尼、马来西亚),2 在运 + 5 在建(24 个月内 RFS);anchor 客户 OpenAI / Meta / Thinking Machines Lab;墨尔本已在生产跑 GB300 NVL72,后续站点用 Vera Rubin NVL72(VR200)+ Spectrum-X。🇬🇧 900+ MW contracted; 7 factories / 4 countries (AU, SG, ID, MY), 2 live + 5 in build (RFS within 24 months); anchor customers OpenAI / Meta / Thinking Machines Lab; Melbourne runs GB300 NVL72 in production, later sites use Vera Rubin NVL72 (VR200) + Spectrum-X.
- 塔州 Bell Bay(Southgate)288 MW、3×220 kV、官方定义 dispatchable;UPS 提供 FCAS / load modulation / firming。电网互动不是比喻,但有时间尺度边界(§8.2)。🇬🇧 Bell Bay, Tasmania (Southgate): 288 MW, 3×220 kV, officially 'dispatchable'; UPS delivers FCAS / load modulation / firming. Grid interaction is not a metaphor, but has time-scale boundaries (§8.2).
3.3 最小电网词汇集(必背)
3.3 Minimum grid vocabulary (memorize)
- NEM = National Electricity Market;AEMO = 市场运营方;FCAS = Frequency Control Ancillary Services(10 个市场:Regulation raise/lower + Contingency 1s/6s/60s/5min 各 raise/lower)。🇬🇧 NEM = National Electricity Market; AEMO = the market operator; FCAS = Frequency Control Ancillary Services (10 markets: Regulation raise/lower plus Contingency raise/lower at 1s/6s/60s/5min).
- 对数据中心这种“负载”:raise = 减少用电(降载、推迟作业);lower = 增加用电(把可延迟作业提前,吸收弃风弃光 / stranded power)。只说“电网紧张就降载”是半吊子。🇬🇧 For a data centre, a load: raise = consume less (curtail, defer); lower = consume more (pull deferrable jobs forward to absorb surplus renewables / stranded power). 'We throttle down when the grid is tight' is only half the story.
- NVIDIA DSX:DSX OS(多租户/编排)/ DSX Sim(仿真)/ DSX MaxLPS(固定功率包络内动态功率分配)/ DSX Flex(电网信号编排)/ DSX Exchange(IT-OT 信号)。被问“DSX 已有调度为何自研”:DSX 提供信号与厂级编排,自研层负责作业级决策与产品化。🇬🇧 NVIDIA DSX: DSX OS (multi-tenancy/orchestration) / DSX Sim (simulation) / DSX MaxLPS (dynamic power allocation in a fixed envelope) / DSX Flex (grid-signal orchestration) / DSX Exchange (IT-OT signals). Asked 'DSX already schedules — why build your own?': DSX supplies signals and factory-level orchestration; our layer owns job-level decisions and productisation.
开场钩子(背 1 分钟)
Opening hook (memorize a 1-minute version)
中文对照(供理解,据此组织英文):谢谢 Giuseppe。直说:我过去 12 年都在做云原生 SRE 与平台工程,近期重心是把“让服务稳定”转成“让 GPU 不闲着”。我从 2020 起是 kubernetes/kubernetes 上游贡献者(@yuzhiquan),在调度框架、拓扑感知调度、节点生命周期都有合入代码;原生 gang GA 前就在 Permit 阶段自研过 gang 调度器。我确实没在生产管过 NVL72——如实说。但 NVL72 调度最难的部分(rack 级 NVLink 拓扑、故障域隔离、动态功耗/热约束)本质是调度器控制面工程,正是我反复做过的事。AI FactoryOS 把遥测融合成一层,调度器消费它做准入、放置、需求塑形——这是我最强的控制面模式。硬件规格入职后与团队对齐。
🇬🇧 Closer that scores: still active in sig/scheduling in 2026 — #140308 (TAS topology-domain fix, merged into v1.38), #140097 (gang+DRA conflict reproducer, open WIP). These are exactly NVL72's hardest problems: topology-domain boundaries, gang+GPU-device deadlock, cache robustness. 'The code I write upstream is the same class of problem.'
Part A · kube-scheduler 内核(必考底座)
Part A · kube-scheduler internals (the foundation they will test)
先立职责边界:kube-scheduler 只做一件事——为 .spec.nodeName 为空的 Pod 选一个 Node 并写回绑定,不运行容器、不保证 Pod 真能起来。ResourceQuota 是 admission 的事,LimitRange 默认值是 admission 的事,Node 故障驱逐是 node-lifecycle controller + kubelet 的事。记住这张边界表,能避开一半送命题。
🇬🇧 Interview line: all performance numbers must be measured by you (Kubemark/KWOK + scheduler-perf); never quote unmeasured throughput/latency. Giving the measurement method is itself a plus.
Part B · Volcano:作业级调度与插件体系
Part B · Volcano: job-level scheduling and plugin system
Volcano 是 CNCF 旗下首个也是唯一的官方容器批处理调度项目。本质:k8s 调度的是“一个个 Pod”,Volcano 调度的是“一组组作业 + 一层层队列”。它把批处理领域完整语义(作业、队列、公平、抢占回收、拓扑)搬进调度器。核心是 Session + Action + Plugin 三层架构。
B.1 扩展点四大类(20+ 个 Fn)与 gang 实现
B.1 Four categories of extension points (20+ Fns) and gang implementation
- 四大类:① 排序 Ordering(jobOrderFn/queueOrderFn/taskOrderFn/nodeOrderFn/batchNodeOrderFn…);② 过滤与准入 Validation(predicateFn/jobValidFn/jobReadyFn/jobStarvingFn…);③ 抢占与回收(preemptableFn/reclaimableFn/victimTasksFn/overusedFn…);④ 状态与预留(reservedNodesFn/targetJobFn/jobPipelineFn)。同一 Fn 可被多插件注册,靠 tier 顺序组合。🇬🇧 Four categories: (1) Ordering (jobOrderFn/queueOrderFn/taskOrderFn/nodeOrderFn/batchNodeOrderFn…); (2) Validation (predicateFn/jobValidFn/jobReadyFn/jobStarvingFn…); (3) Preempt/Reclaim (preemptableFn/reclaimableFn/victimTasksFn/overusedFn…); (4) State/Reserve (reservedNodesFn/targetJobFn/jobPipelineFn). The same Fn can be registered by several plugins, composed by tier order.
- gang 实现:PodGroup 的 minMember + gang 插件的 JobReady = (allocated + pipelined) ≥ minMember。allocate 阶段未达 minMember 就不绑定——等待;suicide 插件超时释放整组避免死锁。占坑(jobPipelineFn)是原子性的关键。🇬🇧 Gang: PodGroup minMember + gang plugin's JobReady = (allocated + pipelined) ≥ minMember. In allocate, if minMember isn't met the job isn't bound — it waits; the suicide plugin releases the whole job on timeout to avoid deadlock. Pipelining (jobPipelineFn) is the key to atomicity.
- preempt vs reclaim:preempt 同队列内按优先级抢占;reclaim 跨队列按权重把借出的应得份额收回——两个独立 Action,各管一层。conformance 插件保护 kube-system 关键 Pod。🇬🇧 preempt vs reclaim: preempt takes from lower priority within a queue; reclaim recovers lent-out entitled share across queues by weight — two independent Actions, each one layer. The conformance plugin protects kube-system Pods.
- 拓扑与设备:task-topology 按通信关系把频繁通信的 Task 共置同一 NVLink 域/leaf switch;numaaware 对齐 CPU/内存到 GPU 的 NUMA;deviceshare 支持 GPU 共享/MIG/vCUDA。到 NVL72:层级换成 rack(单一 NVLink 域)→ 液冷回路/PDU → node。🇬🇧 Topology & devices: task-topology co-locates chatty Tasks in the same NVLink domain / leaf switch; numaaware aligns CPU/memory to the GPU's NUMA node; deviceshare supports GPU sharing / MIG / vCUDA. On NVL72 the levels become rack (single NVLink domain) → cooling loop / PDU → node.
B.2 三种扩展方式(从易到难)
B.2 Three ways to extend (easy to hard)
- A 配置内置插件:ConfigMap 的 tiers 增减/排序插件,零代码改行为(首选);B Extender:HTTP 回调外部服务,钩子最全、可跨语言,无需重编译,但每次调度 HTTP 串行阻塞、挂了会卡调度,需超时+降级;C 自定义 Plugin:实现接口 + RegisterPluginBuilder,编译进调度器。🇬🇧 A configure built-in plugins: add/reorder plugins in the ConfigMap's tiers, zero code (first choice); B Extender: HTTP callback to an external service, most hooks, cross-language, no recompile — but every scheduling cycle blocks on HTTP and a down extender stalls scheduling, so add timeout + degradation; C custom Plugin: implement the interface + RegisterPluginBuilder, compile into the scheduler.
🇬🇧 Red-line phrasing: Kueue does not 'lack gang' — it's job-level vs scheduling-level. Third-party comparison tables saying 'Kueue gang = ❌' were roughly true before 2024 and are now outdated — see §6 for the accurate statement.
Part C · Kueue:准入、配额与治理
Part C · Kueue: admission, quota and governance
Kueue 不是 Pod 调度器,而是准入 + 配额 + 公平 + 抢占系统,不替换 kube-scheduler。流程:提交 Job → Kueue 创建 Workload(Pending)→ 准入阶段决定“准不准/给多少/哪个 flavor” → Admitted=true 后把资源请求 patch 回 Pod → kube-scheduler 负责放置。一句话:Kueue 决定“能不能”,kube-scheduler 决定“放哪儿”。价值是把“调度之前的治理”产品化。
C.1 核心对象与配额模型
C.1 Core objects and quota model
- 六类核心对象:ResourceFlavor(资源“口味”,如 gpu-h100-nvl72,带 selector/taint 与 topologyName);ClusterQueue CQ(配额主体:resourceGroups、cohort 借用、抢占、admissionChecks、namespaceSelector);LocalQueue LQ(namespace 级,指向 CQ,Job 打 queue-name 标签进入);Workload(由 Job 生成,记录 podSets 与 status.admission);AdmissionCheck(准入闸门);Topology/TAS(拓扑感知)。🇬🇧 Six core objects: ResourceFlavor (resource 'flavour', e.g. gpu-h100-nvl72, with selector/taint and topologyName); ClusterQueue CQ (quota subject: resourceGroups, cohort borrowing, preemption, admissionChecks, namespaceSelector); LocalQueue LQ (namespace-scoped, points to a CQ, Jobs label queue-name to enter); Workload (generated from the Job, records podSets and status.admission); AdmissionCheck (admission gate); Topology/TAS (topology-aware scheduling).
- 配额与借用:CQ 有 nominalQuota;cohort 内互相借用(borrowingLimit 上限,lendingLimit 控制借出);借出方来高优先级作业时 reclaim/preempt 收回。公平用 cohort 内 DRS(Dominant Resource Share)。抢占三个开关默认都是 Never:withinClusterQueue / reclaimWithinCohort / borrowWithinCohort。FlavorFungibility 让首选 flavor 满时回退到可互换的。🇬🇧 Quota & borrowing: a CQ has nominalQuota; CQs in a cohort borrow from each other (borrowingLimit cap, lendingLimit controls lending); when the lender gets a high-priority job, reclaim/preempt recovers it. Fairness uses DRS (Dominant Resource Share) inside the cohort. Three preemption knobs all default Never: withinClusterQueue / reclaimWithinCohort / borrowWithinCohort. FlavorFungibility lets a full preferred flavour fall back to an interchangeable one.
- Job → Workload 生成靠 PodSets 抽象:把不同框架(Job/JobSet/PyTorchJob/RayJob)的差异用 PodSets() 抹平,调度只认统一结构。配额用量 = Σ(podSet 请求 × count)。新增框架只需实现 manager 接口。🇬🇧 Job → Workload uses the PodSets abstraction: differences across frameworks (Job/JobSet/PyTorchJob/RayJob) are flattened by PodSets(); scheduling only sees a uniform structure. Quota usage = Σ(podSet requests × count). Adding a framework only needs a manager implementation.
C.2 gangs、TAS 与 AdmissionCheck(Firmus 杀手锏)
C.2 Gangs, TAS and AdmissionCheck (the Firmus killer feature)
- Kueue 的 gang = 准入层 all-or-nothing:每个 podSet 必须在 Admitted 前同时满足配额 + TAS 域;All-or-nothing with PodsReady 在 podsReadyTimeout 内未全部 Ready 就驱逐 + 指数退避重排;podSets[].minCount 允许部分;另有 Partial Admission 与 Dynamic Reclaim。它比 Volcano 运行时拼装失败得更早、释放得更快。🇬🇧 Kueue's gang = admission-level all-or-nothing: every podSet must fit quota + TAS domain before Admitted; All-or-nothing with PodsReady evicts and requeues with exponential backoff if not all Ready within podsReadyTimeout; podSets[].minCount allows partial; plus Partial Admission and Dynamic Reclaim. It fails earlier and releases faster than Volcano's runtime assembly.
- TAS 精确机制(口语“整域预留”的准确版本):定义 Topology CRD(层级如 rack→block→node,每层按域打标签),ResourceFlavor 引用它;在 PodTemplate 上用 podset-required-topology / preferred / unconstrained 等注解逐层评估;准入时只选能完整填满的域、域间互斥避免碎片,然后 kube-scheduler 在域内放置。NVL72 的 rack(单一 NVLink 域)天然是 TAS 顶层,液冷回路/PDU 为附加层。🇬🇧 TAS precise mechanism (the accurate version of the colloquial 'whole-domain reservation'): define a Topology CRD (levels like rack→block→node, each labelled by domain), referenced by a ResourceFlavor; use podset-required-topology / preferred / unconstrained annotations on the PodTemplate evaluated level by level; at admission only pick a domain that can be completely filled, domains are mutually exclusive to avoid fragmentation, then kube-scheduler places inside the domain. The NVL72 rack (single NVLink domain) is naturally the top TAS level, with cooling loop / PDU as extra levels.
- AdmissionCheck = 可插拔准入闸门:所有 check state=Ready 前 Workload 不被 Admitted。写自定义 controller 设状态。AI FactoryOS 的功耗/热/电网/维护信号恰好就是一个 AIFactoryAdmissionCheck:电价高峰、热降额、维护窗口 → Pending → 不准入。这是接外部遥测最干净的扩展点,比改调度器内核优雅十倍。🇬🇧 AdmissionCheck = a pluggable admission gate: a Workload isn't Admitted until all checks report state=Ready; you write a custom controller to set the state. AI FactoryOS's power/thermal/grid/maintenance signals are exactly an AIFactoryAdmissionCheck: peak price, thermal derate, maintenance window → Pending → not admitted. It's the cleanest extension point for external telemetry — ten times more elegant than hacking the scheduler core.
- 可见性 = Workload 的 status.conditions(Admitted/Pending/Preempted/Finished)+ events + Prometheus 指标(队列深度/准入等待/抢占数)——这就是 JD 里的“可执行的调度解释”。MultiKueue 做多集群;TAS/DRA/Topology Spreading 等新特性以较新版本为准,面试统一说“我讲机制,具体状态以部署版本为准”。🇬🇧 Visibility = Workload status.conditions (Admitted/Pending/Preempted/Finished) + events + Prometheus metrics (queue depth / admission wait / preemptions) — that's the JD's 'actionable scheduler explanations'. MultiKueue handles multi-cluster. For TAS/DRA/Topology Spreading say 'I explain the mechanism; status depends on the deployed version'.
🇬🇧 DRA note: the device plugin only reports 'how many cards locally', blind to NVLink topology / whole-rack; DRA (DeviceClass + structured params) lets the scheduler see 'which NVLink domain, interconnected with which cards' — the prerequisite for NVL72 topology awareness. DRA does not support Pod preemption.
Part D · Volcano vs Kueue:对比与选型
Part D · Volcano vs Kueue: comparison and selection
一句话:Kueue 是“门禁与秩序”(谁能进场、占多少、何时放行),Volcano 是“座位调度引擎”(一组作业怎么排、放在哪)。作用于作业生命周期不同阶段,正交可组合,也可二选一。判断关键不是功能清单长短,而是痛点在“调度之前”还是“调度之中”。
| 维度 Dimension | Kueue | Volcano |
|---|---|---|
| 本质 Essence | 准入控制(admission),不替换 kube-scheduler | 批处理调度引擎,自带 volcano-scheduler |
| 调度单元 Unit | Workload(可含多个 podSet) | Job/Task(Queue→Job→Task→Node 四层) |
| 决策模型 Model | 模拟调度后决定是否 admit | 周期驱动 Session 快照 + Action 流水线 |
| Gang | 准入层 all-or-nothing(作业级) | 调度层 PodGroup 占坑原子绑定(严格) |
| 队列/公平 | CQ + Cohort 借用,DRS | Queue 层级,DRF / Proportion |
| 抢占 | 三开关,默认 Never | preempt / reclaim 两个 Action |
| 扩展 | Webhook + Controller + Job integration + AdmissionCheck | 20+ 扩展点 Fn + Extender + 自定义 Plugin |
| 原生 K8s API | 直接用 Job/JobSet/CronJob | 多用自定义 VCJob CRD |
| 接入成本 | 较低(原生、增量) | 较高(自定义 API + 调度概念) |
| 归属 | K8s SIG Scheduling 子项目 | CNCF 项目 |
D.1 Gang 三层级(高频追问)
D.1 Three levels of gang (frequent follow-up)
- K8s 原生 Coscheduling:kube-scheduler 插件,2024 起 beta 演进,纯原生、能力基础。🇬🇧 K8s-native Coscheduling: a kube-scheduler plugin, beta since 2024, pure native and fairly basic.
- Kueue 准入层:Workload 整体准入 + All-or-nothing with PodsReady;不保证所有 Pod 同一时刻被原子绑定,但保证配额与作业整体放行。🇬🇧 Kueue admission-level: Workload admitted as a whole + All-or-nothing with PodsReady; doesn't guarantee all Pods are atomically bound at the same instant, but guarantees quota and whole-job release.
- Volcano 调度层:PodGroup + minMember + jobPipelineFn 占坑,凑齐后统一绑定——严格调度级 gang。追求严格 gang 时 Volcano 更强,重视原生集成与配额治理时 Kueue 更省心。🇬🇧 Volcano scheduling-level: PodGroup + minMember + jobPipelineFn pipelining, bind together once assembled — strict scheduling-level gang. Volcano is stronger for strict gang; Kueue is easier when you value native integration and quota governance.
D.2 选型决策树与常见误区
D.2 Selection decision tree and common misconceptions
- 小团队先跑稳 → 原生 kube-scheduler + namespace;多团队要配额公平 → Kueue;分布式训练要严格 Gang/Backfill → Volcano;两者都要 → Kueue 准入 + Volcano 调度组合。🇬🇧 Small team, get stable first → native kube-scheduler + namespace; multi-team needing quota/fairness → Kueue; distributed training needing strict gang/backfill → Volcano; both → Kueue admission + Volcano scheduling composition.
- 经验法则(Jimmy Song《AI 基础设施手册》):Kueue 更像资源治理框架(准入+配额+承诺+统计),是平台底座;Volcano 更像批处理调度引擎,是训练负载加速器。🇬🇧 Rule of thumb (Jimmy Song, AI Infrastructure Handbook): Kueue is more a resource-governance framework (admission+quota+commitment+accounting), the platform base; Volcano is more a batch scheduling engine, the training-workload accelerator.
- 误区纠正:①“Kueue 不支持 Gang”已过时;②“Kueue 替代 kube-scheduler”错;③“用了 Volcano 就不需要配额治理”不一定;④“两者只能二选一”错,正交可组合。🇬🇧 Misconceptions corrected: (1) 'Kueue has no gang' is outdated; (2) 'Kueue replaces kube-scheduler' is wrong; (3) 'Volcano means no quota governance needed' — not necessarily; (4) 'pick only one' is wrong — they compose orthogonally.
Part E · 协同:Kueue + Volcano 正交组合
Part E · Composition: Kueue + Volcano, orthogonal by design
两者能配合的根本原因:作用在作业生命周期不同阶段,职责不重叠。Kueue 决定“谁能进场、能占多少、何时放行”(准入、配额、承诺、统计),Volcano 决定“怎么排座位、放在哪”(作业级调度、Gang、抢占回收)。形成“先治理、再调度”的闭环。这正是 JD 的 “integrate … rather than replace”。
E.1 组合架构:数据平面 + 控制面
E.1 Composition architecture: data plane + control plane
- 数据平面(资源单位与隔离):整卡 nvidia.com/gpu / MIG 硬切分 / 共享虚拟化(HAMi 等)。控制面(秩序与策略):Kueue(治理/准入/配额/承诺/统计)+ Volcano(批处理语义/调度策略)。平台层:门户、训练生命周期、SLO、计量。🇬🇧 Data plane (resource unit & isolation): whole GPU nvidia.com/gpu / MIG hard partition / shared virtualisation (HAMi). Control plane (order & policy): Kueue (governance/admission/quota/commitment/accounting) + Volcano (batch semantics/scheduling policy). Platform layer: portal, training lifecycle, SLO, metering.
- 分工边界表:配额准入与统计以 Kueue 为单一事实源;调度策略与 gang 以 Volcano 为准。两者都有队列与抢占——这是最大重叠风险,须避免策略重复/互相打架。🇬🇧 Division of labour: quota admission and accounting take Kueue as the single source of truth; scheduling policy and gang take Volcano as authoritative. Both have queues and preemption — the biggest overlap risk; avoid duplicated or conflicting policy.
- 协同生命周期:提交挂起(suspend:true) → Kueue 配额准入 → 解除挂起 → Pod 由 schedulerName=volcano 接管做 gang + 放置 → 运行/抢占 → 完成归还配额。交接靠 suspend 状态 + schedulerName 衔接。🇬🇧 Lifecycle: submit + suspend:true → Kueue quota admission → unsuspend → Pods taken over by schedulerName=volcano for gang + placement → run/preempt → finish and return quota. The handoff is via suspend state + schedulerName.
# 对接骨架(理念示意,生产以版本文档 + 测试验证为准)
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:/n name: ddp-train
labels:/n kueue.x-k8s.io/queue-name: team-a-queue # Kueue 准入入口
spec:/n schedulerName: volcano # 交给 Volcano 调度
minAvailable: 8 # Volcano gang
tasks:/n - name: worker
replicas: 8
template:/n spec:/n containers:/n - name: train
resources:/n limits: {nvidia.com/gpu: 1}
🇬🇧 Version-integration discipline: Kueue/Volcano support for each other's Workload/Job mapping changes by version. The above is a conceptual skeleton — before production, check Kueue's Job Integration list and Volcano Job's Workload mapping, and validate the suspend↔gang handoff in a test cluster.
E.2 六套参考架构(按组织成熟度选)
E.2 Six reference architectures (choose by maturity)
- A 单队列整卡(小团队跑稳)→ B 多队列整卡 + Kueue(治理先行)→ C MIG 池化 + Kueue(强隔离)→ D HAMi 共享 + Kueue(高利用率)= 架构 E Kueue + Volcano(训练增强)→ F 双池分治(在线/离线,稳定优先)。演进路线:A → B → C/D → E/F。🇬🇧 A single-queue whole GPU (small team) → B multi-queue whole GPU + Kueue (governance first) → C MIG pool + Kueue (strong isolation) → D HAMi sharing + Kueue (high utilisation) → E Kueue + Volcano (training enhancement) → F dual-pool split (online/offline, stability first). Path: A → B → C/D → E/F.
- 拼装原则:A 最小可用闭环优先;B 资源单位越细验收越严格;C 重配置是“变更”必须流程化(尤其 MIG);D 把平台责任写进架构。验收三层指标:调度层(排队时间/成功率/抢占回收)、性能层(吞吐/延迟 P50/95/99、抖动)、运营层(用量归因/兑现率/超卖违约率/成本)。🇬🇧 Composition principles: (A) minimal viable loop first; (B) finer resource units require stricter acceptance; (C) reconfiguration is a 'change' and must be process-controlled (especially MIG); (D) write platform responsibilities into the architecture. Three acceptance layers: scheduling (queue time/success/preempt-reclaim), performance (throughput/latency P50/95/99, jitter), operations (attainment/fulfilment/oversubscription breach/cost).
Part F · AI/GPU 调度方案全景
Part F · The AI/GPU scheduling landscape
当 GPU 成为最贵资源,调度目标从“能不能跑”升级为“有秩序、高效率、可审计地跑”。技术栈三层:框架/作业层(PyTorchJob/JobSet/RayJob/VolcanoJob)、控制面(Kueue 治理 + Volcano/kube-scheduler/YuniKorn/Koordinator 调度)、数据平面(整卡/MIG/vGPU/MPS/时间片/HAMi/DRA)。选型先定资源单位,再定秩序与调度策略,最后定作业表达。
F.1 拓扑感知:聚拢 vs 打散(易错)
F.1 Topology awareness: pack vs spread (easy to get wrong)
- K8s 原生 topologySpreadConstraints 是打散(高可用,跨 zone 均匀分布);Kueue TAS 是聚拢(把同一 Workload 的 Pod 锁在同一拓扑域,低通信延迟)。目标相反,不要混用:训练要聚拢,服务要打散。🇬🇧 K8s-native topologySpreadConstraints is spread (high availability, even across zones); Kueue TAS is pack (lock a Workload's Pods in one topology domain, low communication latency). Opposite goals — don't mix: training wants pack, services want spread.
- RDMA/网络:RoCEv2 依赖 PFC/ECN,NIC 与 GPU/NUMA 的亲和(NIC affinity)决定跨节点通信效率;NVL72 内 scale-up 走 NVLink,跨 rack scale-out 走 IB/Spectrum-X。拓扑感知就是让 TP/EP 的 all-to-all 永远留在 NVLink 域内。🇬🇧 RDMA/network: RoCEv2 relies on PFC/ECN; NIC-to-GPU/NUMA affinity decides cross-node communication efficiency; inside NVL72 scale-up is NVLink, cross-rack scale-out is IB/Spectrum-X. Topology awareness keeps TP/EP all-to-all inside the NVLink domain.
F.2 GPU 共享与虚拟化(隔离 vs 灵活)
F.2 GPU sharing and virtualisation (isolation vs flexibility)
- 隔离强度 MIG > vGPU > MPS > 时间片;灵活性反之。整卡最稳、隔离强但细粒度利用率低;MIG 硬切分强隔离、单位离散、重配置有成本;时间片无内存/故障隔离、延迟不可预测;HAMi 可声明共享、细粒度弹性,但隔离/干扰取决于实现,需更强治理与验收。🇬🇧 Isolation MIG > vGPU > MPS > time-slicing; flexibility is the reverse. Whole GPU is simplest and strongly isolated but coarse; MIG is hard partition, strong isolation, discrete units, reconfiguration cost; time-slicing has no memory/fault isolation and unpredictable latency; HAMi offers declarative sharing, fine-grained elasticity, but isolation/interference depend on the implementation and need stronger governance and acceptance.
- 生产推理 SLA 用 MIG;开发/轻推理用时间片;追求密度用 HAMi 但必须配套治理。Kueue 配额要绑明确资源单位(nvidia.com/gpu / MIG profile / 共享份额),并用 ResourceFlavor 让 H100/A10 等异构显式化,避免“10 张卡”对不同团队价值不同的隐性不公平。🇬🇧 Use MIG for production-inference SLAs; time-slicing for dev/light inference; HAMi for density but with governance. Kueue quota must bind a clear resource unit (nvidia.com/gpu / MIG profile / share) and use ResourceFlavor to make H100/A10 heterogeneity explicit, avoiding the hidden unfairness where '10 GPUs' means different things to different teams.
F.3 弹性容错与主流调度器横评
F.3 Elasticity/fault-tolerance and scheduler landscape
- 弹性容错:Kueue 协作式抢占(默认 Never,给训练留 checkpoint 时间)+ Dynamic Reclaim(reclaimablePods 释放多余配额)+ Partial Admission;框架层 VolcanoJob 的 PodEvicted→RestartJob 从 checkpoint 恢复。调度层与框架层配合,才能“抢占不丢训练”。🇬🇧 Elasticity: Kueue cooperative preemption (default Never, leaves time for checkpoint) + Dynamic Reclaim (reclaimablePods releases surplus quota) + Partial Admission; framework layer VolcanoJob's PodEvicted→RestartJob resumes from checkpoint. Scheduler + framework together are what make 'preemption without losing training' possible.
- 横评:kube-scheduler(原生,单 Pod,适合推理服务);Kueue(准入层 gang,多团队严格配额);Volcano(严格 PodGroup gang,分布式训练/HPC);YuniKorn(替换型,层级队列 + DRF,K8s/Hadoop 混合);Koordinator(阿里开源、CNCF 沙箱,侧重在线离线混部与超卖 CPU burst)。常见组合:Kueue 治理 + Volcano/Koordinator 调度 + HAMi/MIG 数据平面。🇬🇧 Landscape: kube-scheduler (native, per-Pod, inference serving); Kueue (admission-level gang, multi-team strict quota); Volcano (strict PodGroup gang, distributed training/HPC); YuniKorn (replacement, hierarchical queues + DRF, K8s/Hadoop hybrid); Koordinator (Alibaba-open-sourced, CNCF sandbox, online/offline colocation and CPU-burst overcommit). Common composition: Kueue governance + Volcano/Koordinator scheduling + HAMi/MIG data plane.
Part G · Firmus 专属设计题(Giuseppe 必问)
Part G · Firmus-specific design questions (Giuseppe will ask)
把前面所有能力收敛到一张白板:作业级决策层坐在 AI FactoryOS 之上,消费遥测,做准入/放置/需求塑形,并给出调度解释。下面 ASCII 图直接在白板复刻,图内英文标签即讲解用词。
用户/ML平台 --API/CLI--> [ Workload CRD ] submit(train/ft/infer/batch/agentic)
| parallelism TP/PP/DP/EP, topology constraints,
| priority/quota/checkpoint period/power budget
[ Admission & Queue Layer (Kueue/ctrl) ]
quota·priority·fair(DRF)·reserve·preempt·backfill·aging
| admission webhook | demand-shaping
[ Scheduling Framework ] [ Factory Resource Controller ]
plugin Filter/Score/ <-- telemetry -- grid/power/derate:/n Reserve/Permit defer/pace/right-size/resched
| topology+factory |
[ ClusterTopology CRD ] [ AI FactoryOS telemetry ]
NVLink单域·NUMA·PCIe (GPU/cooling/power/grid/maintenance)
NIC/RDMA·fault domain(rack/PDU)
Bind -> GPU UUID annotation -> events: why queued/admitted/placed/preempted/deferred
metrics -> Prometheus/OTel -> Grafana(queue depth/latency/frag/topology/power-thermal)
🇬🇧 Time-scale red line: second-level FCAS belongs to UPS / power electronics / hardware interlocks; the K8s scheduler only does minute-scale-and-slower soft shaping, only on jobs not yet started; software decisions may change placement but must never rewrite security policy or bypass device protection (the CTO's agentic-permission boundary).
Part H · 你的真实证据链(会被查)
Part H · Your real evidence chain (it will be checked)
H.1 上游 PR 清单与口径
H.1 Upstream PR list and the exact wording
| PR/Commit | 状态 Status | 与 Firmus 的关系 |
|---|---|---|
| #140308 TAS 空 topology label → invalid domain | ✅ merged 2026-10 (v1.38) | 直击:TAS 域 = NVL72 rack(单一 NVLink 域)/ PDU;错误域会让 72 卡作业溢到 scale-out fabric |
| #140097 gang+DRA 分配冲突复现器 | 🟡 open WIP 2026-06 | 直击:NVL72 用 DRA 分配整机架 GPU;gang+DRA+高密度是死锁三元组 |
| #140306 PodGroup 缓存 / #140143 cache-miss 指标 | 🟡 closed 未合入 | 缓存一致性;NVL72 节点故障/热事件频繁,决定会不会把作业丢到刚死的节点 |
| #101822 NodeResourcesFit score func | ✅ merged 2021-06 (v1.22, 165 review) | 放置打分核心;pack vs spread 直接影响功耗/热,且是我亲自贡献的代码 |
| #100853 EnqueueExtensions / #101859 panic 修复 / drain 系列 | ✅ merged 2021 | 事件驱动重新入队 / 插件健壮性 / 节点驱逐自动化 |
🇬🇧 Discipline: merged PRs → 'shipped'; #140097 → 'an open reproducer I'm iterating on'; #140306/#140143 → 'PRs I submitted that were closed without merge — still useful as problem statements'. If asked 'were those merged?', answer honestly and add '#140308 is the one that actually landed'. Honesty scores. #101822 had 165 review comments, not 203.
H.2 自研 Permit 阶段 gang 调度器(不在简历,当独立项目讲)
H.2 Custom Permit-phase gang scheduler (not on the résumé — present it as its own project)
- 问题:训练/批作业要全有或全无,原生调度器先放部分 Pod 卡住、资源长期占却不出活;当时 coscheduling/PodGroup 未 GA。🇬🇧 Problem: training/batch jobs need all-or-nothing; native scheduling placed some Pods and stalled, holding resources without progress; coscheduling/PodGroup wasn't GA yet.
- 我负责:在 kube-scheduler 插件链 Permit 扩展点实现——组内 Pod 全进 Permit 才原子放行,超时整体回滚释放。🇬🇧 What I owned: implemented at the Permit extension point — a group's Pods are released atomically only when all reach Permit; on timeout the whole group rolls back and releases.
- 困难:① 多 PodGroup 互等死锁 → 超时回滚 + 准入控制(资源不足撑最大 gang 不接纳)+ 严格优先级;② 并发 Assume/Bind 的 GPU 争抢 → 对 GPU UUID 做 CAS。🇬🇧 Difficulty: (1) cross-group wait deadlock → timeout rollback + admission control (don't admit a gang the cluster can't resource) + strict priority; (2) concurrent Assume/Bind GPU contention → a compare-and-swap on the GPU UUID.
- 决定:放 Permit 而非 Reserve/PostFilter,因为 Permit 是周期最后闸门、最适合“等全组”原子放行;不等社区 GA 因为业务窗口。结果:gang 成功率【补数字】,僵死事件【降 X%】;复盘:社区 coscheduling 成熟后应更早评估迁移而非长期自研(诚实加分)。🇬🇧 Decision: chose Permit over Reserve/PostFilter because it's the last gate in the cycle, ideal for 'wait for the whole group' atomic release; didn't wait for upstream GA because the business window demanded it. Result: gang success rate 【fill】, deadlock incidents 【down X%】; retrospective: once community coscheduling matured, should have evaluated migration earlier (honest retrospective scores).
Part I · 简历项目五轴(HR 必深挖)
Part I · Résumé projects — five-axis drill (HR will dig in)
五轴 = 问题 / 你负责 / 困难 / 决定 / 结果。每条都要能讲“问题→证据→定位→止血→根治→防复发”,所有【】换成真实可追溯数字,否则被判“背团队项目”。以下给出英文骨架,中文人心照组织。
- Tubi:Istio 全舰队(南北向 ingress gateway 先行,再按 namespace 分批注入 sidecar,mTLS + 黄金信号,零应用改造)+ AWS/GCP 多集群 locality-first + Terraform 集群外 IaC / 集群内 GitOps + on-call AI agent。on-call agent 命中 Firmus“受控自动化 / remediation 即软件定义能力”,主动展开。🇬🇧 Tubi: Istio fleet-wide (north-south ingress gateway first, then per-namespace phased sidecar injection, mTLS + golden signals, zero app changes) + AWS/GCP multi-cluster locality-first + Terraform IaC outside / GitOps inside + on-call AI agent. The on-call agent hits Firmus's 'guarded automation / remediation as software-defined capability' — expand proactively.
- Megvii:阿里云 HA + Enigma 发布系统(独立写 Go 后端:快照/回滚/业务 API + Bootstrap 前端);回滚一致性靠配置+镜像版本化+状态快照。🇬🇧 Megvii: K8s on Aliyun HA + Enigma release system (solo Go backend: snapshot/rollback/business APIs + Bootstrap frontend); rollback consistency via config+image versioning+state snapshot.
Part J · 反问清单 + 冲刺 + 避坑
Part J · Reverse-questions + sprint + pitfalls
J.1 Senior 反问(定档用)
J.1 Senior reverse questions (they set your tier)
- “Bell Bay 被定义为 dispatchable、UPS 提供 FCAS——秒级响应与调度器分钟级塑形的边界由谁定?哪些 workload 允许被 lower 信号提前拉起?”🇬🇧 'Bell Bay is described as dispatchable with UPS-delivered FCAS — who owns the boundary between second-level response and the scheduler's minute-level shaping, and which workloads are eligible to be pulled forward on a lower signal?'
- “官网说每 HyperCube 32 NVL72 racks / 2 Scale Units——一个 Scale Unit 对应几个 rack?这个映射会怎样进入 ClusterTopology CRD 的层级?”(高质量反问,不要自己推断数字)🇬🇧 'The site says 32 NVL72 racks / 2 Scale Units per HyperCube — how many racks per Scale Unit, and how does that mapping feed the ClusterTopology CRD levels?' (high-quality question — don't infer the number yourself)
- “既然已有 Managed Slurm 和 DSX Flex/MaxLPS,自研调度器与它们的职责边界画在哪?哪些决策绝不下放到软件层?”🇬🇧 'Given Managed Slurm and DSX Flex/MaxLPS already exist, where exactly is the responsibility boundary of the in-house scheduler, and which decisions are never delegated to the software layer?'
- “NVL72 舰队仍在 build phase——现在最痛的是调度算法,还是遥测/信号契约(freshness、stale 时的降级)?”🇬🇧 'The NVL72 fleet is still in build phase — is the current pain in scheduling algorithms, or in the telemetry/signal contract (freshness, what to do when stale)?'
- “这个岗位的 success metric 更看重 end-user outcome(排队时间、time-to-result),还是 factory efficiency(GPU 利用率、碎片率)?两者冲突时怎么排序?”🇬🇧 'Does this role's success metric weigh end-user outcomes (queue time, time-to-result) or factory efficiency (GPU utilisation, fragmentation) more — and how do you rank them when they conflict?'
J.2 上场前 10 分钟速查
J.2 Ten-minute pre-interview check
- 开场钩子 1 分钟 + §9.2 自研 gang 五轴脱稿;开场不要主动提没跑过 NVL72,被直接问 GPU 才用诚实边界。🇬🇧 1-minute opening hook + §9.2 custom-gang five-axis, off-script; don't volunteer 'no NVL72' in the opening — use the honest boundary only if directly asked about GPU.
- 事实红线 6 条(§1.1)+ 时间尺度红线(§8.2)扫一遍;PR 状态照念,不升级成 merged。🇬🇧 Scan the six fact red lines (§1.1) and the time-scale red line (§8.2); recite PR statuses as-is, never upgrade to merged.
- 所有【】换成真实数字,每个数字能答“怎么来的”;尤其成本降 30% 的核算口径。🇬🇧 Replace every 【】 with a real number, each with a 'how did you get that' answer — especially the 30% cost-down basis.
J.3 避坑
J.3 Pitfalls
- 不要纠正 JD 的 “Syndey” 拼写,也不要把 PDF 里 €sustainable($ 字体错误)当事实。🇬🇧 Don't correct the JD's 'Syndey' typo, and don't treat '€sustainable' (a $-sign font error in the PDF) as a fact.
- 不要说“没人跑过 NVL72”——墨尔本站点已在生产跑 GB300 NVL72;把边界说成“我的”,不是“市场的”。🇬🇧 Don't say 'nobody has run NVL72' — Melbourne already runs GB300 NVL72 in production; frame the boundary as 'mine', not 'the market's'.
- 不要把 Kueue 说成替换 kube-scheduler,也不要说 Kueue 不支持 gang;不要说 Volcano 有 6 个 Action。🇬🇧 Don't say Kueue replaces kube-scheduler, or that Kueue has no gang; don't say Volcano has six Actions.
附录 · 中英术语对照
Appendix · Bilingual glossary
| 中文 | English |
|---|---|
| 准入控制 / 配额治理 | Admission control / quota governance |
| 作业级调度 / 批处理调度 | Job-level scheduling / batch scheduling |
| 全有或全无 / 协同调度 | All-or-nothing / gang (co-)scheduling |
| 拓扑感知调度 | Topology-aware scheduling (TAS) |
| 整域预留 | Whole-domain reservation |
| 主导资源公平 / 主导资源份额 | DRF / Dominant Resource Share (DRS) |
| 抢占 / 回收 / 回填 | Preemption / reclaim / backfill |
| 动态资源分配 | Dynamic Resource Allocation (DRA) |
| 多实例 GPU / 多集群 | MIG / MultiKueue |
| 需求塑形 / 电网感知 | Demand-shaping / grid-aware |
| 频率控制辅助服务 | FCAS (Frequency Control Ancillary Services) |
| 可调度的用电负荷 | Dispatchable (demand) |
| 可执行的调度解释 | Actionable scheduler explanations |
| 技术减震器 | Technical shock absorber |
参考来源:Firmus 官网 firmus.co(2026-11-09 核验)与 JD-A `5417753008`;companies/firmus/ 现有手册与策略卡;调度器系列 5 篇深度文档(Kueue / Volcano / 对比 / 协同 / AI-GPU 方案);Jimmy Song《AI 基础设施手册》;Kueue / Volcano / Kubernetes 官方文档。