Knowledge distillation, which transfers knowledge from a large model (the teacher) to a small model (the student), is a promising way in the field of lightweight model design. Existing distillation methods based on intermediate features mainly focus on knowledge transfer between the same stages of teacher and student networks, which may lead to student networks receiving semantically mismatched knowledge and missing contextual knowledge. To solve this problem, a cross-stage distillation with inverted bottleneck projectors is proposed. A cross-stage connection structure is designed that enables the student network to access the teacher network’s most matched stages in semantics and further obtain rich contextual knowledge from multiple stages of the teacher. After establishing the cross-stage connection between the teacher network and the student network, inverted bottleneck projectors are employed to extract useful knowledge from multiple stages of the teacher. A stack structure with three-layer convolutions is proposed in every projector, which can make the knowledge of the teacher network more easily understood by the student network. The inverted bottleneck structure can reduce information loss, ensuring the integrity of knowledge transmission. Further, ReLU activations are incorporated into the projectors to remove features with lower response values, thereby filtering redundant elements. Extensive experiments on CIFAR100, ImageNet, Tiny-ImageNet, and STL-10 datasets demonstrate the effectiveness of the proposed approach.