Curriculum-learning-driven hierarchical multi-agent deep reinforcement learning for collaborative scheduling in complex supply chain networks

With the growing scale, heterogeneity, and dynamic uncertainty of modern supply chain networks, collaborative scheduling across order assignment, manufacturer selection, and logistics operations has become increasingly critical and challenging because of strong inter-stage coupling, high decision complexity, and dynamic operational constraints. To address these challenges, this paper investigates the joint optimization problem of order assignment, heterogeneous manufacturer selection, and logistics vehicle scheduling in dynamic supply chain collaborative networks and proposes a curriculum-learning-driven hierarchical multi-agent deep reinforcement learning framework (CH-MADRL) for coordinated scheduling in complex environments. First, the joint optimization problem is formulated as a hierarchical multi-agent Markov decision process to capture the hierarchical dependencies and dynamic interactions among order assignment, heterogeneous manufacturer selection, and logistics vehicle scheduling, which establishes a unified modeling foundation for multi-stage collaborative scheduling. Second, based on this formulation, a hierarchical multi-agent deep reinforcement learning architecture is developed to decompose the tightly coupled high-dimensional joint scheduling problem into three correlated sub-problems, enabling coordinated optimization across different stages of the supply chain. Third, a constraint-progressive adaptive curriculum-learning mechanism is developed to facilitate policy learning under dynamic constraints, where a stage-conditioned dynamic masking mechanism regulates feasible action spaces, and a dual-gated promotion strategy stabilizes transitions across curriculum stages. Simulation experiments demonstrate that the proposed method surpasses baseline approaches in scheduling performance, training efficiency, and cross-scale generalization capability.

1 Introduction

The global supply chain system is undergoing a significant evolution characterized by heightened uncertainty. The superposition of multiple factors, including trade frictions, geopolitical risks, public health emergencies, and demand fluctuations, imposes continuous shocks on the stable operation of cross-regional supply networks (de Lima et al., 2022). Under macro environments where volatility, uncertainty, complexity, and ambiguity coexist, supply chain management is transitioning from traditional models dominated by informatization and standardized processes to digital-intelligent systems featuring data-driven approaches, intelligent perception, and autonomous decision making (Hao and Demir, 2025; Yu et al., 2026). Meanwhile, enterprises impose higher requirements on the response speed, collaborative efficiency, and resilience of supply chain systems. Multi-stage collaborative optimization and real-time scheduling thus emerge as critical challenges in scenarios with dynamically arriving orders, heterogeneous resource capabilities, and strong coupling between manufacturing and logistics, particularly for intelligent decision making in complex supply chains.

Researchers worldwide conduct systematic studies on collaborative scheduling in complex supply chains, and existing methods generally fall into three categories: mathematical programming and decomposition optimization methods, heuristic and metaheuristic methods, and data-driven methods based on reinforcement learning (Wu et al., 2024). The first two formulate order production and distribution as optimization models subject to process flows, capacity limits, delivery deadlines, and transportation constraints, solving them via exact algorithms or approximate search techniques. Vié et al. (2025) propose a matheuristic framework that combines local search with exact methods to coordinate production and distribution, optimizing total costs of order fulfillment and shipment. Subsequent work has extended production and logistics scheduling from static single-stage settings to complex environments with equipment failures, dynamic order arrivals, and multi-objective trade-offs (Atasagun and Karaoğlan, 2024). These methods remain effective for small- to medium-scale problems or relatively stable environments. However, continuous order arrivals, intensified cross-stage coupling, and frequent disturbances expose critical limitations: model reconstruction is costly, solution times escalate rapidly, and schedule stability deteriorates.

Reinforcement learning (RL) handles dynamic task arrivals, resource state changes, and frequent disturbances by interacting with the environment, learning from trial and error, and optimizing long-term returns. Unlike traditional methods that require explicit modeling and static solution processes, RL updates decision policies through continuous feedback. Zhang et al. (2024a) formulate dynamic flexible scheduling problems with transportation time constraints as multi-agent collaborative decision processes, coordinating machine agents with job agents. Shi et al. (2025) propose a nested hierarchical deep reinforcement learning method for production and logistics collaborative scheduling in dynamic flexible job shops. Li et al. (2025a) develop a real-time scheduling method using multi-agent deep reinforcement learning (MARL) for production and logistics coordination under large-scale dynamic order arrivals. For logistics operations, Li et al. (2024) propose a hierarchical framework for dynamic conflict-free automated guided vehicle (AGV) scheduling in automated container terminals. The above studies validate RL, particularly multi-agent and hierarchical variants, as effective for complex scheduling and logistics coordination (Shi et al., 2025; Li et al., 2025a, 2024). However, existing methods primarily address single workshops, single logistics systems, or local resource coordination. Extending these methods to full-chain supply chain scenarios is difficult because strong coupling among order assignment, heterogeneous manufacturer selection, and vehicle scheduling leads to exponential growth of the joint state and action space. Furthermore, dynamic task arrivals, resource competition, and spatiotemporal dependencies compound challenges, including sparse rewards, inefficient exploration, and training non-stationarity. Recent studies note that, although RL and MARL have advanced rapidly in dynamic scheduling, practical deployment remains limited by scalability, training stability, and cross-scenario generalization (Zhang et al., 2024b; Hady et al., 2025).

For RL in complex supply chain collaborative scheduling, curriculum learning (CL) provides a feasible way to organize training under conditions of cold-start difficulties, sparse rewards, and training instability. Training tasks are arranged progressively so that agents can first learn basic decision rules and then adapt to larger networks, stronger stage couplings, and tighter operational constraints. Results from related scheduling studies support the usefulness of this training paradigm. Iklassov et al. (2023) demonstrate that curriculum-based training performs better than direct training for large-scale instances in job shop scheduling, while Narvekar et al. (2020) show that task ordering based on difficulty and transferability improves reinforcement learning performance. A similar situation arises in supply chain collaborative scheduling, where increasing network scale is usually accompanied by stronger interactions among decision stages and more complex constraints. However, existing CL methods rely on predefined difficulty orders or rough task selection heuristics, and only limited attention has been paid to the gradual regulation of feasible action spaces during training. This issue is particularly important in large discrete decision spaces, where invalid action masking can reduce infeasible exploration and improve training stability (Huang and Ontañón, 2022). Recent work has started to examine automatic curriculum design in sparse-reward cooperative multi-agent reinforcement learning (Chen et al., 2021), but research that combines curriculum learning, hierarchical multi-agent coordination, and dynamic feasible-action control for complex supply chain scheduling is still limited.

To address these limitations, we investigate the joint optimization of order assignment, heterogeneous manufacturer selection, and logistics vehicle scheduling in dynamic supply chain collaborative networks. First, we formulate this problem as a hierarchical multi-agent Markov decision process that captures the coupling among order, manufacturing, and logistics decisions. Then, we propose a curriculum-learning-driven hierarchical multi-agent deep reinforcement learning framework (CH-MADRL), which coordinates scheduling across stages through hierarchical decomposition, curriculum progression, and dynamic constraint handling. Furthermore, we design a stage-conditioned dynamic masking mechanism and a dual-gated promotion strategy to gradually expand feasible action spaces and stabilize curriculum progression. Finally, experiments on dynamic environments validate the proposed method. The main contributions of this paper are as follows:

• 1.

  A CH-MADRL framework is proposed, which decouples order assignment, heterogeneous manufacturer selection, and logistics vehicle scheduling into three hierarchically correlated sub-problems to alleviate the combinatorial complexity caused by high-dimensional joint decision making.

• 2.

  A constraint-progressive adaptive curriculum-learning mechanism is designed, which achieves smooth transition across scale tasks and improves policy training stability through complexity evolution paths from simple to complex and dual-gated promotion strategies.

• 3.

  A stage-conditioned dynamic masking mechanism is constructed, which embeds task dependencies, resource availability, and scale boundaries into the action selection process, achieving progressive unlocking of feasible action spaces and reducing invalid exploration.

• 4.

  Comparative experiments show that the proposed method outperforms baselines in scheduling performance, training efficiency, and cross-scale generalization.

The remainder of this paper is organized as follows. Section 2 introduces the related literature and techniques. Section 3 provides a formal description and mathematical modeling of the research problem. Section 4 elaborates on the proposed CH-MADRL framework in detail. Section 5 presents experimental results. Section 6 concludes and outlines future research directions.

2 Related work

2.1 Collaborative logistics scheduling methods in supply chains

Coordination between production and logistics stages constitutes the basis of supply chain collaborative scheduling. Expansion from single factories and distribution chains to multi-facility, multi-resource, and multi-node networks has shifted research from separate optimization of orders, production, and distribution toward integrated production and distribution and joint manufacturing and transportation optimization. Liang et al. (2025) examine collaborative production and material distribution, showing that independent optimization of either stage lowers system-wide efficiency under strong spatiotemporal coupling. On this basis, Homayouni and Fontes (2021) investigate joint production and transportation scheduling problems in flexible job shops, Yao et al. (2024) incorporate limited AGV resources into flexible job shop models to formalize manufacturing and transportation collaboration, and Sidki et al. (2025) develop batch-centric mixed-integer programming for integrated production and pipeline distribution. These studies demonstrate that production and logistics collaborative scheduling has evolved from traditional single-stage optimization to integrated optimization oriented toward multi-resource coupling.

Regarding solution methodologies, existing research includes exact algorithms, heuristic algorithms, and metaheuristic algorithms. Exact methods guarantee optimality for small-scale instances, yet computational complexity escalates rapidly with order scale, node quantity, and transportation resources, precluding real-time decision making in dynamic scenarios (Yao et al., 2024; Sidki et al., 2025). Consequently, researchers generally adopt learning-augmented heuristic and metaheuristic methods. Uzunoglu et al. (2023) combine learning-augmented mechanisms with local search for parallel serial batch processing machine scheduling; Amirteimoori et al. (2023) propose parallel heuristic methods for hybrid job shop scheduling with conflict-free AGV path planning; Pérez et al. (2023) fuse GRASP, genetic algorithms, and learning mechanisms for supply chain scheduling; Karimi and Alinia (2025) introduce energy consumption factors into multi-factory supply chain scheduling, expanding modeling dimensions. However, most of these studies are still centered on single factories or limited collaborative settings, and research on integrated scheduling across order assignment, manufacturer selection, and vehicle dispatching in dynamic supply chain networks remains relatively scarce.

2.2 Dynamic-scheduling methods based on deep reinforcement learning

In recent years, deep reinforcement learning (DRL) has emerged as an important direction in dynamic-scheduling research because it learns sequential decision policies end to end in high-dimensional state spaces. Liu et al. (2022) formulate dynamic flexible job shop scheduling as Markov decision processes, demonstrating DRL's real-time response advantages under random order arrivals. Subsequent work extends from single-agent methods to graph neural networks, multi-agent reinforcement learning, and hierarchical reinforcement learning. Ngwu et al. (2026) and Hamou et al. (2025) review DRL applications for dynamic job shop scheduling and supply chain production scheduling, respectively; Liu and Huang (2023) combine graph neural networks with DRL for dynamic job shop scheduling; Wang et al. (2025a) propose attention-enhanced reinforcement learning methods for flexible job shop scheduling with transportation constraints; Li et al. (2025b) propose an end-to-end decentralized scheduling framework for dynamic distributed heterogeneous flow shops; Wang et al. (2025b) extend hierarchical multi-agent deep reinforcement learning to flexible job shops with transportation constraints. Meanwhile, multi-agent deep reinforcement learning (MADRL) extends from workshop scheduling to broader collaborative decision problems such as online scheduling in assembly systems, distributed hybrid flow shops, and multi-echelon inventory management (Kaven et al., 2024; Di et al., 2024; Xu et al., 2025; Liu et al., 2025). These studies demonstrate that DRL gradually evolves from single-workshop internal scheduling toward complex scenarios involving cross-resource, cross-node, and multi-agent collaboration.

However, existing DRL research exhibits two limitations in full-chain collaborative scenarios within complex supply chains. First, most methods address single workshops, single logistics systems, or local resource allocation, lacking integrated modeling of order decomposition, heterogeneous manufacturer collaboration, and logistics vehicle scheduling. Second, as supply chain scale and constraint complexity increase, the joint state and action space expands rapidly, causing cold-start difficulties, sparse rewards, inefficient exploration, and multi-agent training non-stationarity. To alleviate these difficulties, CL is introduced into RL training processes. Its fundamental idea improves sample utilization efficiency and policy transfer capability through task organization from easy to difficult (Lin et al., 2025). Applications span flexible job shop scheduling (Lu et al., 2025) and hierarchical reinforcement learning for dynamic AGV scheduling, automated terminal task allocation, and port equipment coordination (Hu et al., 2025a; Chang et al., 2025; Hu et al., 2025b; Yang et al., 2026). However, existing CL methods remain limited to single-dimensional difficulty progression or empirical stage switching without systematic coupling of the synchronous expansion of order scale, manufacturing nodes, logistics capacity, and hierarchical multi-agent decision making with dynamic action space constraints. Based on this, this paper proposes a hierarchical MADRL framework incorporating constraint-progressive curriculum evolution to address cross-level joint decision problems in complex supply chain collaborative scheduling.

3 Problem formulation

This paper investigates a multi-stage decision optimization problem in collaborative supply chain logistics, covering three interrelated decision dimensions: order task decomposition, manufacturing resource allocation, and transportation resource scheduling. Unlike the classical flexible job shop scheduling problem (FJSP), limited to single-facility operations, our collaborative setting introduces strict cross-node logistics constraints and spatiotemporal coupling. This integration exponentially expands the joint action space, making the proposed adaptive curriculum mechanism computationally essential to overcome the resulting high-dimensional exploration challenges. For formal modeling, the problem is represented as a standardized supply chain instance with scale $n\times m\times l$, where n, m, and l denote the number of orders, heterogeneous manufacturers, and logistics vehicles, respectively. Let $\mathcal{O}=\mathit{\{}{O}_{\mathrm{1}},\mathrm{\dots },O_n\mathit{\}}$ represent the order set, $\mathcal{M}=\mathit{\{}{M}_{\mathrm{1}},\mathrm{\dots },M_m\mathit{\}}$ represent the manufacturer set, and $\mathcal{L}=\mathit{\{}{L}_{\mathrm{1}},\mathrm{\dots },L_l\mathit{\}}$ represent the vehicle set. Each order O_i contains H_i operations with process precedence, denoted as $K_i=\mathit{\{}{o}_{i,\mathrm{1}},\mathrm{\dots },o_{i,H_i}\mathit{\}}$, where operation o_i,k must be completed before $o_{i,k+\mathrm{1}}$. Each operation o_i,k can only be assigned to a manufacturer j∈ℳ with the required processing capability, and its processing time is denoted as $p_{i,k,j}$. When adjacent operations are executed by different manufacturers, vehicle v∈ℒ is required to complete inter-node transportation. The transportation time consists of empty vehicle dispatch time from the current position to the pickup point and loaded transportation time from the pickup point to the target manufacturing node. An operation can only start processing when its predecessor operation is completed, materials arrive via transportation, and the corresponding manufacturer resource becomes available. This paper takes minimizing the system makespan as the optimization objective:

$$\begin{array}{}\text{(1)}& min{C}_{max}=min\left(\underset{i\in \mathcal{O}}{max}{C}_{i,H_i}\right),\end{array}$$

where $C_{i,H_i}$ is determined by the manufacturing and transportation processes at each stage of the order. For order i, the completion time of the final operation is as follows:

$$\begin{array}{}\text{(2)}& \begin{array}{rl}{C}_{i,H_i}=& max\left[T_j^{\text{avail}}\right(max(C_{i,H_i-\mathrm{1}},T_{\mathrm{v}}^{\text{free}}+T_{\mathrm{v},i^{\ast }}^{\text{empty}})\\ & +T_{i,j}^{\text{trans}}\left)\right]+p_{i,H_i,j}.\end{array}\end{array}$$

For each operation, the processing start time is constrained by the completion of its predecessor, the arrival of transported materials, and the availability of the assigned manufacturer. The completion time is determined by the start time and the corresponding processing duration. Accordingly, the final order completion time is jointly affected by manufacturing capacity constraints, process precedence constraints, and transportation time constraints. For modeling purposes, the following assumptions are adopted:

• 1.

  Manufacturer resources are exclusive and non-pre-emptive, meaning that each manufacturer can process only one operation at a time and processing cannot be interrupted.

• 2.

  Tasks follow a strict serial flow of processing, transportation, and reprocessing, and all operations must be executed in the prescribed order.

• 3.

  When two adjacent operations are assigned to different manufacturers, inter-node transportation must be completed by a logistics vehicle, and the corresponding transportation time is nonzero.

• 4.

  Logistics vehicles are subject to exclusivity constraints, meaning that each vehicle can execute only one transportation task at a time.

• 5.

  Orders arrive randomly over time, and fluctuations in actual operational efficiency are determined by manufacturer fulfillment reputation q_i and logistics vehicle service quality q_v.

Figure 1 Architecture of the proposed CH-MADRL framework.

4 CH-MADRL framework

To address high-dimensional decision complexity in large-scale supply chain collaborative scheduling, together with cold-start difficulty and inefficient exploration under complex spatiotemporal constraints, this paper proposes the CH-MADRL framework. As shown in Fig. 1, the framework combines hierarchical task decomposition, dynamic masking, and adaptive curriculum learning to reduce decision complexity, incorporate multi-stage dependencies and spatiotemporal constraints, and improve convergence efficiency and generalization performance.

4.1 Hierarchical multi-agent MDP model construction

Considering the dynamic, partially observable, and multi-resource-coupled nature of supply chain networks, the scheduling problem is formulated as a hierarchical decentralized partially observable Markov decision process. The model is defined by the tuple $\langle \mathcal{F},S,A,P,R,\mathrm{\Omega },\mathit{\gamma }\rangle$, where $\mathcal{F}=\mathit{\{}{F}_{\text{ord}},F_{\text{mfg}},F_{\text{log}}\mathit{\}}$ denotes the heterogeneous agent set for order assignment, manufacturer selection, and logistics scheduling; S denotes the global state space describing real-time information on resource nodes and order flows; Ω denotes the joint observation space composed of local observations $\mathit{\{}{o}_{\text{job}},o_{\text{mfg}},o_{\text{log}}\mathit{\}}$ at different decision layers; A denotes the joint action space composed of discrete action subspaces at three hierarchies: order assignment, manufacturer selection, and logistics scheduling; P denotes the state transition function governed by operation precedence, transportation delays, and other physical process constraints; R denotes the reward function designed to promote global collaborative optimization; and $\mathit{\gamma }\in [\mathrm{0},\mathrm{1})$ denotes the discount factor.

Table 1 Definitions of state space features.

4.1.1 State space

Following the partially observable modeling framework, the state space is decomposed into three hierarchical local subspaces: the order agent state $S_{\mathrm{t}}^{\mathrm{O}}$, the manufacturer agent state $S_{\mathrm{t}}^{\mathrm{M}}$, and the logistics agent state $S_{\mathrm{t}}^{\mathrm{L}}$. The state features of the three agent layers are listed in Table 1, where all temporal variables are quantified in minutes to ensure dimensional consistency.

4.1.2 Action space

The joint action space is decomposed as $\mathcal{A}={\mathcal{A}}^{\mathrm{O}}\times {\mathcal{A}}^{\mathrm{M}}\times {\mathcal{A}}^{\mathrm{L}}$. A dynamic mask M_t is introduced to project action probability distributions onto feasible domains and enforce physical feasibility constraints.

Order assignment subspace (𝒜^O). The action $a_{\mathrm{t}}^{\mathrm{O}}\in \mathit{\{}\mathrm{1},\mathrm{\dots },n\mathit{\}}$ selects a high-priority order for processing. The mask $M_{\mathrm{t}}^{\mathrm{O}}\in \mathit{\{}\mathrm{0},\mathrm{1}{\mathit{\}}}^n$ is activated only when an order has not been delivered and satisfies the prerequisite readiness conditions:

$$\begin{array}{}\text{(3)}& M_{\mathrm{t}}^{\mathrm{O}}\left[i\right]=\mathbb{I}({\mathit{\delta }}_i^{\text{done}}=\mathrm{0})\cdot \mathbb{I}\left(\text{Ready}\right(i,\mathrm{t}\left)\right),\end{array}$$

where 𝕀(⋅) is the indicator function, and Ready(i,t) indicates material and preceding-operation readiness for order i.

Manufacturer selection subspace (𝒜^M). The action $a_{\mathrm{t}}^{\mathrm{M}}\in \mathit{\{}\mathrm{1},\mathrm{\dots },m\mathit{\}}$ selects a manufacturer for the current operation. The mask $M_{\mathrm{t}}^{\mathrm{M}}\in \mathit{\{}\mathrm{0},\mathrm{1}{\mathit{\}}}^m$ enforces process qualification constraints based on the capability matrix T_cap, retaining only manufacturers with the required capability for operation k:

$$\begin{array}{}\text{(4)}& M_{\mathrm{t}}^{\mathrm{M}}\left[j\right]=\mathbb{I}\left({\mathbf{T}}_{\text{cap}}\right[k,j]\ne \mathrm{\varnothing }).\end{array}$$

Logistics scheduling subspace (𝒜^L). The action $a_{\mathrm{t}}^{\mathrm{L}}\in \mathit{\{}\mathrm{1},\mathrm{\dots },l\mathit{\}}$ dispatches a logistics vehicle. The mask $M_{\mathrm{t}}^{\mathrm{L}}\in \mathit{\{}\mathrm{0},\mathrm{1}{\mathit{\}}}^l$ reflects resource availability and excludes vehicles under failure or maintenance:

$$\begin{array}{}\text{(5)}& M_{\mathrm{t}}^{\mathrm{L}}\left[v\right]=\mathbb{I}\left({\text{Status}}_{\mathrm{v}}\right(t)\in \mathit{\{}\text{Idle},\text{Active}\mathit{\left\}}\right).\end{array}$$

4.1.3 Reward function

For makespan minimization, a terminal sparse reward provides only limited learning signals, which often leads to temporal credit assignment difficulties and slow convergence in long-horizon scheduling tasks. To improve training efficiency, a potential-based dense reward is adopted, where the potential function is defined as $\mathrm{\Phi }\left(S_{\mathrm{t}}\right)=-C_{\text{max}}\left(S_{\mathrm{t}}\right)$. The immediate reward at step t is therefore defined as follows:

$$\begin{array}{}\text{(6)}& r_{\mathrm{t}}=\mathrm{\Phi }\left(S_{\mathrm{t}}\right)-\mathrm{\Phi }\left(S_{t-\mathrm{1}}\right)=C_{\text{max}}\left(S_{t-\mathrm{1}}\right)-C_{\text{max}}\left(S_{\mathrm{t}}\right).\end{array}$$

Since the makespan is monotonically non-decreasing during scheduling and satisfies $C_{\text{max}}\left(S_{\mathrm{t}}\right)\ge C_{\text{max}}\left(S_{t-\mathrm{1}}\right)$, the immediate reward always satisfies r_t≤0. Actions that increase the critical path receive an immediate penalty, while actions that leave it unchanged receive zero reward. For an episode of length T, the cumulative return can be written in telescoping form:

$$\begin{array}{}\text{(7)}& \begin{array}{rl}R& =\sum _{t=\mathrm{1}}^T{r}_{\mathrm{t}}=\sum _{t=\mathrm{1}}^T\left[C_{\text{max}}\right(S_{t-\mathrm{1}})-C_{\text{max}}(S_{\mathrm{t}}\left)\right]\\ & =C_{\text{max}}\left(S_{\mathrm{0}}\right)-C_{\text{max}}\left(S_T\right)=-C_{\text{max}}^{\text{final}}.\end{array}\end{array}$$

Figure 2 Constraint-progressive adaptive curriculum learning mechanism.

4.2 Constraint-progressive adaptive curriculum learning mechanism

To improve learning in large-scale supply chain scheduling with dynamic task arrivals, sparse rewards, and high-dimensional constraints, a constraint-progressive adaptive curriculum is introduced. As shown in Fig. 2, training starts with simple supply chain instances and gradually progresses to more complex ones by expanding network topology and task scale. Dynamic masking then releases feasible actions stage by stage, reducing invalid exploration in large decision spaces. Promotion to the next stage is allowed only after policy stabilization, which helps maintain stable training and reliable convergence.

4.2.1 Supply chain complexity manifold evolution

To reduce exploration difficulty in high-dimensional environments, the curriculum-learning process is defined as a discrete sequence of environment sets, $\mathcal{C}=\mathit{\{}{\mathcal{E}}_{\mathrm{1}},{\mathcal{E}}_{\mathrm{2}},\mathrm{\dots },{\mathcal{E}}_k\mathit{\}}$, where k=6. At each stage ℰ_k, the supply chain network is specified by the tuple ${\mathrm{\Omega }}_k=\langle N_k^{\text{ord}},N_k^{\text{mfg}},N_k^{\text{log}}\rangle$, which denotes the numbers of orders, manufacturers, and logistics vehicles, respectively. The complexity progression is organized into three stages.

Foundation logic construction stage (ℰ₁). The initial scale is set to ${\mathrm{\Omega }}_{\mathrm{1}}=\langle \mathrm{10},\mathrm{5},\mathrm{3}\rangle$. This stage is used to support rapid policy initialization in a low-dimensional setting and to learn basic coordination rules, including order assignment, resource availability, and task dependencies.

Scale continuous increment stage (ℰ₂∼ℰ₅). The network scale is increased step by step without changing the underlying scheduling mechanism so as to strengthen policy robustness and generalization under growing resource competition and spatiotemporal conflicts.

Target scale collaboration stage (ℰ₆). The environment reaches the target scale ${\mathrm{\Omega }}_{\mathrm{6}}=\langle \mathrm{20},\mathrm{10},\mathrm{5}\rangle$. At this stage, the policy is trained under a larger state space and stronger spatiotemporal coupling to form a stable collaborative scheduling strategy.

4.2.2 Dynamic masking-driven progressive unlocking of action space

In multi-agent systems, directly facing target-scale configurations with preset global maximum action space 𝒜_max (whose dimensions are determined by $N_{max}^{\text{ord}}$, $N_{max}^{\text{mfg}}$, and $N_{max}^{\text{log}}$) causes an extremely severe curse of dimensionality. To this end, this paper decouples the curriculum evolution mechanism from underlying control logic, formalizing it as a dimensionality release process of decision feasible domains: at the early curriculum stage ℰ_k, the system utilizes the dynamic masking mechanism M_t to construct hard constraint boundaries, forcibly masking all high-dimensional action indices exceeding the current network scale, that is, strictly ensuring $M_{\mathrm{t}}\left[a\right]\equiv \mathrm{0},\forall a>N_k$. As the curriculum advances toward higher-stage manifolds, masking constraints are gradually released, and effective decision boundaries of agents are progressively unlocked.

This mechanism reduces the global optimization problem into nested subspace searches, mitigating exploration disorientation and improving training stability.

4.2.3 Adaptive promotion mechanism based on dual-gating

This paper designs an adaptive promotion mechanism based on dual-gating, where promotion is triggered only when both indicators simultaneously satisfy the required conditions within sliding window W. The sliding-window size was set to W=20 episodes, determined through preliminary sensitivity experiments balancing promotion responsiveness and policy convergence stability.

1. Performance lower-bound gating: The average return μ_W within the current window must exceed the preset baseline threshold ${\mathit{\delta }}_{\text{perf}}^{\left(k\right)}$:

   $$\begin{array}{}\text{(8)}& {\mathit{\mu }}_{\mathrm{W}}=\mathbb{E}\left[G_{t-W:t}\right]>{\mathit{\delta }}_{\text{perf}}^{\left(k\right)}.\end{array}$$

   The threshold ${\mathit{\delta }}_{\text{perf}}^{\left(k\right)}$ is determined from the optimal solutions of heuristic algorithms at the corresponding scale. This criterion requires the DRL policy to reach a scheduling level beyond that of traditional rules before entering the next stage.

2. Policy stationarity detection: Due to the high variance of DRL exploration, mean-based indicators can be biased by outliers or occasional high returns. Therefore, a relative standard deviation criterion is introduced:

   $$\begin{array}{}\text{(9)}& {\text{RSD}}_{\mathrm{W}}={\displaystyle \frac{\mathit{\sigma }\left(G_{t-W:t}\right)}{\left|{\mathit{\mu }}_{\mathrm{W}}\right|}}<{\mathit{\delta }}_{\text{stab}},\end{array}$$

   where σ(⋅) is the standard deviation function, and δ_stab is the preset stationarity tolerance factor, which is set to 0.05 through preliminary tuning experiments to balance sensitivity to policy fluctuations and training efficiency. The dual-gating mechanism effectively avoids misjudgment risks from single indicators through joint constraints of performance and stationarity, ensuring robustness of curriculum stage transitions. The complete adaptive curriculum evolution process is detailed in Algorithm

Algorithm 1 Adaptive curriculum evolution algorithm based on stationarity detection.

1:

Input: Initial policy network parameters θ,ϕ; curriculum environment set $\mathcal{C}=\mathit{\{}{E}_{\mathrm{1}},E_{\mathrm{2}},\mathrm{\dots },E_K\mathit{\}}$

2:

Output: Converged optimal parameters ${\mathit{\theta }}_k^{\ast }$ for each stage; final global optimal policy parameters ${\mathit{\theta }}_K^{\ast }$

3:

Initialization: K – total number of curriculum stages; W – sliding-window size; δ_perf(k) – performance promotion threshold for stage k; δ_stab – stationarity (RSD) threshold; G – sliding-window return sequence.

4:

for curriculum stage k=1 to K do

5:

Initialize environment configuration Ω_k←Config(E_k); If k>1 then $\mathit{\theta }\leftarrow {\mathit{\theta }}_{k-\mathrm{1}}^{\ast }$

6:

while True do

7:

Perform one PPO training iteration; obtain current episode total return G_t

8:

Update sliding-window return sequence G; compute current window mean return ${\mathit{\mu }}_{\mathrm{W}}=\mathbb{E}\left[G_{t-W:t}\right]$

9:

Compute current policy relative standard deviation ${\text{RSD}}_{\mathrm{W}}=\mathit{\sigma }\left(G_{t-W:t}\right)/\left|{\mathit{\mu }}_{\mathrm{W}}\right|$

10:

if μ_W>δ_perf(k) And RSD_W<δ_stab then

11:

Save current converged optimal policy parameters ${\mathit{\theta }}_k^{\ast }\leftarrow \mathit{\theta }$

12:

Break (promotion condition satisfied; proceed to next stage E_k+1)

13:

end if

14:

end while

15:

end for

4.3 Hierarchical proximal policy optimization algorithm

This paper adopts a hierarchical proximal policy optimization (H-PPO) algorithm based on the actor–critic architecture, with feasible-domain hard-constraint projection and a global collaborative loss design. At the network level, agents at each hierarchy use structurally symmetric actor–critic networks. The actor network employs a multilayer perceptron (MLP) to extract state features and introduces a hard-constraint projection layer to ensure physical feasibility. Using the dynamic mask M_t defined in Sect. 4.1.2, log-domain renormalization is applied to the output logits z_t, with $z_{\mathrm{t}}^{\prime }=z_{\mathrm{t}}+\log (M_{\mathrm{t}}+\mathit{ϵ})$. This operation assigns zero probability to invalid actions and blocks gradient propagation along illegal paths, thereby improving search efficiency in constrained spaces.

The training process follows a decentralized execution and centralized evaluation paradigm. All agents share the unified differential reward r_t in Sect. 4.1.3, and local policies are jointly optimized for makespan minimization. Parameter updates use the following composite loss function:

$$\begin{array}{}\text{(10)}& \begin{array}{rl}{L}_{\text{Total}}=& -{\mathbb{E}}_{\mathrm{t}}[min({\mathit{\rho }}_{\mathrm{t}}{A}_{\mathrm{t}},\text{clip}({\mathit{\rho }}_{\mathrm{t}},\mathrm{1}-\mathit{ϵ},\mathrm{1}+\mathit{ϵ})A_{\mathrm{t}}\left)\right]\\ & +c_{\mathrm{1}}{L}_{\text{VF}}\left(\mathit{\varphi }\right)-c_{\mathrm{2}}S\left[{\mathit{\pi }}_{\mathit{\theta }}\right],\end{array}\end{array}$$

where ρ_t denotes the ratio between new and old policies, A_t denotes the advantage function, and S[π_θ] denotes policy entropy.

5 Experimental results and analysis

The proposed method is implemented with PyTorch and runs on a workstation equipped with an AMD Ryzen 7 8845H CPU, 16 GB RAM, and an NVIDIA GeForce RTX 4060 Laptop GPU with 8 GB VRAM. All random seeds are fixed to ensure reproducibility, and each reported result is averaged over 100 independently sampled test instances per scale configuration.

5.1 Training efficiency and curriculum efficacy validation

To evaluate the effect of the curriculum mechanism, two training settings are considered: progressive training with CH-MADRL and direct training with standard DRL. CH-MADRL is trained from ℰ₁ (10 × 5 × 3) and promoted stage by stage under the dual-gating criteria defined by δ_perf(k) and δ_stab, with parameter warm start used at each stage transition. Standard DRL is trained end to end on the target scale 20 × 10 × 5.

Figure 3 Training efficiency and curriculum efficacy validation: (a) convergence curve comparison between CH-MADRL and standard DRL; (b) training convergence comparison between mixed-dimension expansion and single-dimension expansion strategies.

Supply chain networks involve coupled order, resource, and logistics flows, making curriculum design an important factor in training efficiency. Two curriculum expansion strategies are compared: single-dimension expansion, which increases only the number of orders while keeping the manufacturer scale and logistics capacity fixed, and mixed-dimension expansion, which jointly increases order quantity, manufacturer scale, and logistics capacity.

As illustrated in Fig. 3a, standard DRL shows pronounced oscillation in the early training stage and converges to a suboptimal makespan of about 1500, reflecting cold-start difficulty. CH-MADRL exhibits brief performance drops near stage transition points but recovers quickly after each transition and converges to a lower makespan. Figure 3b shows that mixed-dimension expansion leads to smoother stage transitions and faster convergence, while single-dimension expansion produces noticeable performance drops when later stages introduce abrupt parameter changes.

5.2 Overall scheduling performance comparison

To evaluate the scalability of the framework, experiments are conducted across six gradient scales from 10 × 10 × 5 to 20 × 10 × 5, with 100 independent random test instances generated for each configuration. To simulate dynamic disturbances and resource heterogeneity in supply chains, critical environment parameters are independently sampled from uniform distributions:

• 1.

  Dynamic order arrivals: Release times $A_i\sim \mathcal{U}[\mathrm{30},\mathrm{300}]$.

• 2.

  Resource capability fluctuations: Manufacturer processing quality q_j and logistics transportation quality $q_{\mathrm{v}}\sim \mathcal{U}[\mathrm{0.1},\mathrm{1.0}]$.

• 3.

  Logistics time-lag distribution: Inter-node transportation times $t_{\text{trans}}\sim \mathcal{U}[\mathrm{1},\mathrm{10}]$.

The average makespan comparison of various algorithms across different scales is presented in Fig. 4. Results demonstrate that CH-MADRL maintains performance advantages across all test scales. Particularly in the highest-complexity 20 × 10 × 5 large-scale scenario, CH-MADRL converges to an average makespan of 1764.82, representing a 2.3 % reduction compared with standard DRL (1806.67), validating the scalability of the proposed method in high-dimensional state spaces.

Figure 4 Performance trend analysis of CH-MADRL versus standard DRL in multi-scale scalability tests.

The reliability of supply chain scheduling solutions is typically measured through statistical dispersion of the results. The boxplot distribution in Fig. 5 further reveals that CH-MADRL solutions exhibit more compact distribution patterns with fewer outliers. Specifically, at the 20 × 10 × 5 scale, CH-MADRL achieves a makespan standard deviation of 257.36, lower than that of the baseline (272.58), representing a 5.6 % reduction.

Figure 5 Boxplot comparison of makespan distributions across different scales for CH-MADRL versus standard DRL.

5.3 Horizontal comparison with baseline algorithms

To comprehensively evaluate the performance advantages of CH-MADRL, this subsection conducts comparative experiments with three representative algorithm categories:

• 1.

  Composite heuristic rules (rule 1–4): Four composite greedy strategies are constructed for multi-stage collaborative characteristics of supply chains. Four classical order priority rules (EST, MOPNR, SPT, MTWR) are combined with, respectively, manufacturer and logistics assignment rules based on shortest processing time (SPT), serving as local-optimal baselines.

• 2.

  Metaheuristic algorithm (GA): Standard GA is introduced as a classical global search baseline to evaluate whether deep reinforcement learning methods can approach or surpass traditional evolutionary computation methods in solution quality.

• 3.

  Deep reinforcement learning baselines (A2C, DRL + IL): The on-policy algorithm A2C is introduced to validate the advantages of the PPO architecture adopted in this paper regarding update stability. DRL + IL incorporating expert demonstration is introduced to validate whether the proposed curriculum-learning mechanism better facilitates agents in breaking through expert experience limitations and exploring globally superior strategies compared to passive imitation learning.

The three-dimensional bar chart in Fig. 6 intuitively demonstrates the scheduling results of all algorithms for the first 20 instances at the 20 × 10 × 5 scale. CH-MADRL achieves the lowest makespan in the vast majority of instances, outperforming rule algorithms based on local greedy strategies.

Figure 6 Makespan comparison of all methods for 20 test instances at 20 × 10 × 5 scale.

Figure 7 demonstrates solution quality distributions of various algorithms across different test sets through boxplots. Results indicate that CH-MADRL exhibits comprehensive advantages in the vast majority of test scenarios: lowest makespan mean, most compact box range, and minimal outliers.

Figure 7 Statistical boxplot analysis of different algorithms at 20 × 10 × 5 scale

5.4 Zero-shot generalization capability testing

To evaluate the model's zero-shot cross-domain generalization, the converged models from Sect. 5.1, trained at the 20 × 10 × 5 scale, are directly transferred to four unseen instances ranging from 13 × 6 × 4 to 30 × 15 × 7 without fine-tuning. The tests include two scenarios: a fixed-order scenario, where the order quantity remains constant to assess scalability under expanded dimensions, and a random-order scenario, where the order quantity fluctuates to assess robustness under dynamic loads.

Figure 8 Zero-shot generalization performance evaluation at different unseen problem scales: (a) fixed-order-quantity scenario; (b) random-order-quantity scenario.

Table 2 Zero-shot generalization performance comparison at different unseen problem scales.

As illustrated in Fig. 8 and Table 2, CH-MADRL consistently outperforms standard DRL in terms of makespan and variance as problem dimensionality expands stepwise. Specifically, in in-distribution interpolation generalization tests (exemplified by 13 × 6 × 4 and other scales), the model achieves performance improvements of 9.76 % and 9.23 % under fixed- and random-order conditions, respectively, indicating that curriculum learning effectively drives agents to extract general representations of underlying collaborative logic rather than overfitting to specific training distributions. Furthermore, in out-of-distribution extrapolation generalization tests (facing the 30 × 15 × 7 scenario exceeding the training scale), CH-MADRL maintains performance advantages of 1.62 % and 2.25 %, with standard deviations consistently below the control group. These results comprehensively demonstrate that the proposed framework possesses high policy stability and structural generalization capability when facing cross-domain and out-of-bound scales.

Figure 9 Microscopic scheduling behavior visualization comparison between CH-MADRL and rule methods at 20 × 10 × 5 scale: (a) rule-based method; (b) CH-MADRL.

5.5 Interpretability and microscopic behavior analysis

To examine the decision behavior of the agents at a finer scale, Fig. 9 compares the Gantt-chart schedules produced by the two methods for a 20 × 10 × 5 instance. The traditional rule-based method, shown in Fig. 9a, is limited by locally greedy decisions and does not account for the cumulative effects of cross-node logistics delays. As a result, manufacturing and logistics flows become temporally misaligned, leading to task fragmentation and resource idleness. By contrast, CH-MADRL, shown in Fig. 9b, produces a more compact spatiotemporal schedule. Through time window coordination and dynamic load balancing, the agents improve the alignment between processing, transportation, and subsequent processing stages, avoid node congestion, and reduce the overall makespan.

6 Conclusions

This paper proposes CH-MADRL for collaborative scheduling in complex supply chain networks. A hierarchical multi-agent Markov decision process is constructed for the joint optimization of order assignment, heterogeneous manufacturer selection, and logistics vehicle scheduling. To improve training under dynamic constraints, a constraint-progressive adaptive curriculum is introduced, together with a stage-conditioned dynamic masking mechanism and a dual-gated promotion strategy. Experimental results show that CH-MADRL achieves better convergence, lower makespan, and stronger zero-shot generalization across different problem scales.

The current work still has several limitations. Experimental evaluation is conducted in simulated environments with predefined scale progression. Although the framework incorporates dynamic order arrivals and resource quality fluctuations, it does not fully capture all disruption patterns encountered in real supply chain systems, such as supplier defaults and logistics network interruptions. Extending the framework to handle such real-world disruptions remains an important direction for future work. In addition, the current framework considers only makespan minimization and does not address other objectives, such as operational cost, carbon emissions, and delivery reliability. Future research will extend CH-MADRL to multi-objective scheduling and explore graph neural-network-based representations to improve scalability in larger and more complex supply chain scenarios.