Pavements need frequent maintenance and management, for which one needs detailed information related to the road condition. At present, much of the condition monitoring around the world is done manually in situ or using high-end equipment. This is quite impractical, especially on a national or global scale because it takes time and involves high cost and effort. Pothole recognition using deep-learning-inspired image classification is currently being researched. Convolutional neural networks (CNNs) have drastically enhanced the techniques of image classification. We discuss an improved model based on CNN for pothole recognition. It introduces a neural architecture, namely layer permutation scheme for CNN (LPS-CNN). There is always a trade-off between network complexity and processing time efficiency in neural network architectures. The specific importance of the proposed neural network is that it learns a sparse architecture of the receptive neurons, which turns out to be fruitful in attaining maximum efficiency (for pothole recognition) with a reasonable processing time. The practical applicability of the proposed model is tested on the publicly available DSH 2017 pothole dataset. The proposed LPS-CNN outperforms all other existing architectures for pothole recognition and achieves an exceptional average accuracy of 99.6% on the benchmark pothole dataset.