AdaDFKD : Exploring Adaptive Inter-Sample Relationship in Data-Free Knowledge Distillation

In scenarios like privacy protection or large-scale data transmission, Data-free knowledge distillation (DFKD) methods are proposed to learn Knowledge Distillation (KD) when the data is not accessible. They generate pseudo samples that distill the knowledge from a pretrained teacher model and then use them to train a small student model. Current approaches have limitations in effectively addressing the learning capacity of the student model and the inter-sample relationship among the generated pseudo samples during DFKD, leading to the bottleneck of efficiency and performance. In this research, we propose AdaDFKD to construct adaptive targets for the inter-sample relationship. We achieve this by generating targets from "easy-to-discriminate" samples to "hard-to-discriminate" samples as human does. AdaDFKD designs a relationship refinement module (R$^2$M) to optimize the generation process, wherein we learn a progressive conditional distribution of negative samples and maximize the log-likelihood of inter-sample similarity of pseduosamples. Theoretically, we discover that such design of AdaDFKD both minimize the divergence and maximize the mutual information between the distribution of teacher and student models. Experimental results demonstrate the superiority of our approach over state-of-the-art (SOTA) DFKD methods across various benchmarks, teacher-student pairs, and evaluation metrics, as well as well robustness and fast convergence.​