Skip to content
Back to blog

Kubernetes 1.38: Native DRA, AI Scheduling, and Production Upgrade Traps

Kubernetes 1.38 locks Dynamic Resource Allocation into core GA, introduces CompositePodGroup, and ends webhook race conditions. Here is the operational audit for platform engineers.

Maicon Berlofa|Published |8 min read
Architectural overview of Kubernetes 1.38 features including native DRA, CompositePodGroup, and admission order
Production ChartZero-Root & Pinned SHA256

Deploy generic with HelmForge

Production-grade chart with automated S3 backup, ESO integration, and 20-layer security gate.

Kubernetes 1.38 arrives as the final minor release of 2026, marking a decisive shift from evolutionary iteration to operational maturity.

For several release cycles, platform teams managing machine learning training, inference serving, and high-performance computing clusters have lived in a world of compromises. Deploying accelerator-heavy workloads meant juggling third-party custom schedulers, integer-only device plugins, fragile sidecar mutating webhook race conditions, and opaque batch failure states.

Kubernetes 1.38 brings architectural closure to those workarounds. With Dynamic Resource Allocation (DRA) permanently integrated into core General Availability, CompositePodGroup promoted to v1beta1 for native gang-scheduling, and deterministic static webhook admission ordering, Kubernetes officially cements its role as the native operating system for artificial intelligence and distributed infrastructure.

Compiled with Go 1.27 (godebug default=go1.27), Kubernetes 1.38 also introduces strict runtime and API parsing validations. For platform engineers and Helm chart maintainers, upgrading requires auditing specific operational edges before workloads hit production.

What Kubernetes 1.38 changes in practice

Reading release notes SIG by SIG often obscures the holistic operational impact. When mapped by operational surface area, Kubernetes 1.38 reorganizes four critical layers:

  1. Hardware & Accelerator Management: The DynamicResourceAllocation feature gate has been completely removed from the codebase. DRA is no longer an optional experiment; it is a permanent core subsystem with stable device binding and fractional capacity.
  2. Workload Co-Scheduling: Native multi-pod group coordination via CompositePodGroup eliminates the requirement for standalone external schedulers during distributed model training.
  3. Admission Control Determinism: Mutating admission webhooks support explicit static ordering phases, giving operators deterministic sequencing between static manifest-defined and dynamic admission plugins.
  4. Validation & Strict Parsing: Kubelet strictly enforces .env comment formatting, and kube-apiserver caps RSA keys at 4096 bits during JWK parsing, catching latent configuration debt.

Architectural topology of Kubernetes 1.38 showing API server admission ordering, composite pod gang-scheduling, and node DRA runtime execution.

Visual summary: Kubernetes 1.38 coordinates control-plane admission determinism with advanced gang-scheduling and core DRA hardware partitioning on worker nodes.

Dynamic Resource Allocation permanently enters core GA

Historically, Kubernetes managed GPUs and hardware accelerators through the legacy Device Plugin framework. Device plugins exposed hardware as coarse, integer-based node capacity (such as nvidia.com/gpu: 4). They offered no native mechanism to request fractional VRAM, negotiate multi-instance GPU (MIG) slices dynamically, or co-schedule device attachment with storage topologies.

Kubernetes 1.38 completes the transition by graduating Dynamic Resource Allocation to General Availability and removing the feature gate entirely. Key architectural milestones include:

  • Device Binding Conditions (KEP-5007): Kubelet and the scheduler coordinate device-readiness conditions as an explicit prerequisite gate before Pod binding is initiated, decoupling device preparation from volume binding while ensuring Pods are only scheduled to nodes capable of satisfying hardware claims.
  • DRAPartitionableDevices in GA: Hardware accelerators can be partitioned dynamically on demand (such as provisioning isolated compute slices or GPU partitions) rather than relying on pre-configured static node pools.
  • DRAConsumableCapacity & DRAFractionalCapacityRange: Applications can declare granular hardware allocations, sharing physical devices safely across multiple pods with explicit capacity guarantees.
  • DRADeviceTaints & Rules: Device-level taints are enabled by default, featuring a None preview effect that enables platform operators to observe taint evaluations in telemetry before enforcing workload eviction (NoExecute).

For Helm chart maintainers, the implications are straightforward: applications providing AI model serving (such as text embeddings, vector databases, or LLM inference engines) should begin offering structured ResourceClaim templates alongside traditional resources.limits overrides.

Native AI and batch co-scheduling: CompositePodGroup

Distributed training jobs (such as PyTorch DDP, DeepSpeed, or Megatron-LM) have a fundamental operational requirement: all pods in a distributed worker set must be scheduled concurrently. If a cluster with 64 GPUs schedules 60 pods and starves the remaining 4, the entire job hangs while holding costly hardware idle.

Historically, solving this required deploying custom batch schedulers (such as Volcano or customized Kueue topologies). Kubernetes 1.38 promotes the CompositePodGroup API (scheduling.k8s.io/v1beta1) to native beta. In v1.38, the feature is disabled by default and requires enabling feature gates on control-plane components: --feature-gates=CompositePodGroup=true,GenericWorkload=true,TopologyAwareWorkloadScheduling=true.

apiVersion: scheduling.k8s.io/v1beta1
kind: CompositePodGroup
metadata:
  name: distributed-llama-training
  namespace: ml-workloads
spec:
  workloadRef:
    name: llama-training-workload
  schedulingPolicy:
    gang:
      minGroupCount: 4

Key scheduling advancements in 1.38 include:

  • Hierarchical Gang Placement: The composite scheduler coordinates child PodGroup instances bound to a parent Workload. The scheduler guarantees that the required quota of leaf groups (minGroupCount) acquire placements and hardware claims concurrently before workloads are dispatched to nodes.
  • Topology-Aware Placement Scoring: Through TopologyAwareWorkloadScheduling, kube-scheduler introduces the percentageOfPlacementsToScore parameter. In large clusters with spine-leaf network topologies, the scheduler adaptively samples candidate placements, optimizing inter-node communication latency without incurring high CPU scheduling overhead.
  • Job Failure Policy Reasons (KEP-4443, Alpha): Under PodFailurePolicy, rules using action: FailJob support custom condition reasons when the Alpha gate --feature-gates=PodFailurePolicyName=true is enabled on kube-apiserver and kube-controller-manager. Higher-level orchestrators (such as JobSet or Kueue) can distinguish between transient infrastructure failures (like an uncorrectable GPU ECC memory error) and fatal application code bugs, preventing wasted retry budgets.

API machinery: ending the mutating webhook race

In complex production clusters, mutating admission webhooks frequently conflict. When a Pod creation request triggers HashiCorp Vault / External Secrets secret injection, Istio or Linkerd ambient sidecars, and Kyverno or OPA Gatekeeper security policies, the evaluation order was historically non-deterministic or dictated strictly by alphabetical webhook registration names.

Kubernetes 1.38 introduces staticOrder in WebhookAdmissionConfiguration (KEP-5793), enabled via the --feature-gates=MutatingAdmissionWebhookStaticOrder=true flag:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: MutatingAdmissionWebhook
    configuration:
      apiVersion: apiserver.config.k8s.io/v1alpha1
      kind: WebhookAdmissionConfiguration
      staticManifestsDir: /etc/kubernetes/admission/static
      staticOrder: StaticFirst # Supported modes: StaticFirst, StaticLast, ByName
  • StaticFirst (Default): Evaluates manifest-defined administrative webhooks located in staticManifestsDir before dynamically registered cluster webhooks.
  • StaticLast: Allows dynamic API-registered webhooks to mutate objects before static policy enforcers inspect the final result.
  • ByName: Maintains interleaved alphabetical ordering for backward compatibility.

While staticOrder organizes the execution phase of static manifests versus dynamic webhooks, it does not prevent subsequent dynamic mutators from modifying an object after early mutators complete. When security policies must strictly validate the final converged state of an object, teams should continue utilizing webhook reinvocation policies and final validating admission (ValidatingAdmissionWebhook or Common Expression Language validations).

Reducing API payload bloat with ManagedFieldsOptOut

Another chronic control-plane challenge has been metadata.managedFields. Introduced with Server-Side Apply, managed fields track object ownership field by field, often doubling or tripling manifest payload sizes in kubectl get and CI/CD controller reconciliations.

Kubernetes 1.38 introduces the alpha feature gate ManagedFieldsOptOut. Clients requesting resources can pass an HTTP Accept header to omit managed fields from API responses entirely, reducing serialization overhead and network bandwidth by up to 60% during bulk synchronization.

The production upgrade gotchas

While Kubernetes 1.38 delivers powerful platform capabilities, several strict parsing and security enforcement changes require validation before rolling control planes and nodes.

1. Kubelet strict environment file parsing

In previous releases, Kubelet’s environment variable parsing was lenient when processing inline comment markers in .env files. If an environment file contained a comment immediately following a closing quote without leading whitespace (e.g., DATABASE_URL='postgresql://app'#production), Kubelet silently discarded the trailing comment or absorbed it.

In Kubernetes 1.38, Kubelet strictly rejects this syntax. Containers referencing malformed env files will fail during startup with a container runtime error (ContainerCannotRun).

Verify all environment ConfigMaps, Secrets, and embedded application templates:

# Search for environment definitions with trailing comments without space
rg -n "='[^']*'#" charts/ manifests/ values*.yaml

Ensure any comments are placed on dedicated lines or preceded by whitespace (KEY='value' # comment).

2. JWK RSA key size limit (4096-bit maximum)

The Kubernetes API server updates its core cryptographic dependency to go-jose/v4. To protect control-plane nodes against algorithmic complexity and denial-of-service vectors during JWT signature verification, RSA keys larger than 4096 bits are strictly rejected during JWK parsing.

If your cluster integrates with custom corporate OpenID Connect (OIDC) identity providers using 8192-bit RSA signing keys, token authentication will fail immediately upon upgrading kube-apiserver. Ensure identity provider keys are rotated to compliant 2048-bit or 4096-bit RSA keys. If migrating to elliptic curve cryptography such as ECDSA (e.g., P-256 with ES256), explicitly configure kube-apiserver with --oidc-signing-algs=ES256,RS256, as the default setting expects RS256 only (note that Ed25519/EdDSA is not supported by the OIDC authenticator).

3. CRI writable cgroups support

The Container Runtime Interface (CRI) adds cgroup_mount_mode support to manage writable cgroups natively inside container namespaces. This change streamlines running nested systemd workloads, development sandboxes, and virtualized container runtimes without requiring dangerous host-level /sys/fs/cgroup bind mounts.

Operating writable cgroups requires worker nodes running cgroup v2 with nsdelegate enabled in systemd, alongside a container runtime implementing CRI v1.38 specifications (such as containerd 2.1+ or CRI-O 1.38+).

4. Kube-proxy cleanup

kube-proxy now includes built-in detection for iptables-legacy versus iptables-nft across its operational modes. Operators maintaining custom or minimal base container images for kube-proxy no longer need to package the legacy iptables-wrapper detection binary. Additionally, the transitional NFTablesProxyMode feature gate has been removed from the active roster as the modern backend reaches full maturity.

A maintainer checklist for Kubernetes 1.38

Platform engineers and Helm chart authors should execute this operational audit across repositories:

1. Audit Environment Manifests:
   Run grep sweeps across ConfigMaps and values templates to ensure no inline comments immediately trail quotes (='val'#note).

2. Evaluate DRA Hardware Claims:
   Review GPU/accelerator Helm values. Prepare opt-in ResourceClaim templates for dynamic hardware partitioning alongside legacy resource limits.

3. Align Admission Webhook Ordering:
   Configure WebhookAdmissionConfiguration on control planes to establish explicit staticOrder (StaticFirst) alongside staticManifestsDir for security and mutation pipelines.

4. Audit OIDC Signing Keys:
   Confirm that all identity provider OIDC JWKS endpoints serve RSA keys <= 4096 bits, and configure --oidc-signing-algs if using ECDSA.

5. Validate Batch Pod Failure Policies:
   Enable PodFailurePolicyName on control planes if testing custom failure condition reasons in Job and JobSet specifications to delineate infrastructure faults.

6. Verify Node Container Runtimes:
   Ensure container runtimes support CRI v1.38 cgroup mount specifications, with host cgroup v2 nsdelegate enabled.

The bigger read

Kubernetes 1.38 is a release about industrialization.

By locking Dynamic Resource Allocation into core General Availability, establishing native primitives for multi-pod gang-scheduling, and eliminating non-deterministic webhook behaviors, Kubernetes removes the friction that previously forced platform teams to invent custom out-of-tree controllers for advanced infrastructure.

For production clusters, upgrading to 1.38 is not merely an API refresh. It represents an opportunity to simplify architecture: retiring third-party batch hacks, adopting declarative hardware sharing, and building on predictable platform contracts.

References

Newsletter

Get the next post in your inbox

Join the HelmForge newsletter for Kubernetes insights, chart updates, and practical operations tips.

Related analysis

More in Releases

Read next

WordPress on Kubernetes Needs More Than a Pod