A Cooperative Hierarchical Deep Reinforcement Learning Based Multi-Agent Method for Distributed Job Shop Scheduling Problem with Random Job Arrivals

Distributed manufacturing has been an important trend in the industrial field, in which the production cost can be reduced through the cooperation among factories. In the real production, the random job arrivals are regular for the enterprises with daily delivered production tasks. In the paper, Distributed Job-shop Scheduling Problem (DJSP) with random job arrivals is studied. The distributed characteristics and the uncertain disturbance raise higher demands on the responsiveness and the self-adaptiveness of the scheduling method. To meet the scheduling requirements, a hierarchical Deep Reinforcement Learning (DRL) based multi-agent method Agentin is presented where the assigning agent (Agenta) and the sequencing agent (Agents) are respectively designed for job allocation and job sequencing, and they share the system information and extract the features they need independently. Agenta and Agents are both based on the specially-designed DQN framework, which has a variable threshold probability in the training stage, and it can balance the exploitation and exploration in the model training. For Agenta and Agents, two Markov Decision Process (MDP) formulations are established with elaborately-explored state features, rules-based action spaces and objective-oriented reward functions. Based on 1350 different production instances, the independent utility tests prove the effectiveness of the independent agents and the importance of the cooperation among the agents. The comparison test with the related algorithms validates the effectiveness of the integrated multi-agent method