Generating Merging Strategies for Connected Autonomous Vehicles Based on Spatiotemporal Information Extraction Module and Deep Reinforcement Learning

A major challenge concerning a mixed traffic flow system, composed of connected autonomous vehicles (CAVs) and human driven vehicles (HDVs), is how to improve overall efficiency and safety by assigning appropriate control strategies to CAVs. Multi-agent deep reinforcement learning (MADRL) is a promising approach to address this challenge. It enables the joint training of multiple CAVs by fusing CAV sensing information and does not need the compliance of HDVs. However, the fusion of CAV sensing information is non-trivial. Traditional approaches usually fail to take advantage of connectivity among CAVs and time series characteristics of vehicle sensing information, leading to insufficient awareness of the traffic environment. Aimed at tackling these issues, this study proposes a MADRL framework integrating the double deep Q network (DDQN) and a spatiotemporal information extraction module, named spatiotemporal deep Q network (STDQN). A long-short term memory neural network with attention mechanism (AttenLSTMNN) is leveraged to extract temporal dependencies from vehicle perceptive information. Besides, a graph convolution network (GCN) is employed to model the spatial correlations among vehicles in a local range, as well as connectivity of multiple CAVs in a global range. Simulation experiments are conducted in an onramp merging scenario, which is one of the most important and commonly seen scenarios in highway or city expressway systems. Experiment results prove that as compared to baseline MADRL and rule-based methods, the proposed STDQN can improve the overall traffic efficiency, safety, and driving comfort. The proposed framework is promised to be deployed into real CAVs, to realize cooperative, safe, and efficient autonomous driving