Deep reinforcement learning for stochastic last-mile delivery with crowdshipping

We study a setting in which a company not only has a fleet of capacitated vehicles and drivers available to make deliveries but may also use the services of occasional drivers (ODs) willing to make deliveries using their own vehicles in return for a small fee. Under such a business model, a.k.a crowdshipping, the company seeks to make all the deliveries at the minimum total cost, i.e., the cost associated with their vehicles plus the compensation paid to the ODs. We consider a stochastic and dynamic last-mile delivery environment in which customer delivery orders, as well as ODs available for deliveries, arrive randomly throughout the day, within fixed time windows. We present a novel deep reinforcement learning (DRL) approach to the problem that can deal with large problem instances. We formulate the action selection problem as a mixed-integer optimization program. The DRL approach is compared against other optimization under uncertainty approaches, namely, sample-average approximation (SAA) and distributionally robust optimization (DRO). The results show the effectiveness of the DRL approach by examining out-of-sample performance.