Pests and diseases have posed a serious threat to the growth and storage of crops in recent years, leading to major grain losses worldwide. Various chemical substances (such as insecticides) have been widely used to control pests, where some adverse effects have occurred in the agricultural ecosystem. Timely and accurate identification of pests can be one of the most important steps for effective control. However, traditional pest monitoring relies mainly on experts or technicians to manually identify pests, indicating the subjective, labor-intensive, costly, and difficult translation on a large scale. Alternatively, the convolutional neural networks (CNN) in the field of deep learning can be widely expected as one of the best ways to promote conventional identification. However, the current CNNs suffer from low real-time performance, identification accuracy, and complex structures for easy use during pest identification. In this study, an identification model was proposed for the pests of lightweight crops using an improved ShuffleNet V2 CNN. Firstly, the Lightweight Multi-scale Feature Fusion (LMFF) module was designed using depth-separable and pointwise convolution. Three branches were contained in the LMFF module, including the 3×3 depth-separable convolution, 1×1 pointwise convolution, and residual structure. The basic unit structure was introduced into the ShuffleNet V2 to obtain the receptive fields of different scales, particularly for better feature extraction for the pests. Secondly, the parallel paths were added to the Efficient Channel Attention (ECA) mechanism, in order to adaptively update the weights of different paths by learnable parameters. The Adaptive and Efficient Channel Attention (AECA) attention mechanism was then obtained to introduce into the 1×1 point-wise convolution of the ShuffleNet V2 basic unit and 1×1 convolution of each branch of the sub-sampling unit. As such, the cross-channel interaction ability of the model was improved significantly. A Sigmoid Weighted Liner Unit (SiLU) activation function was also established to avoid neuron necrosis in the ReLU activation function. A better performance was achieved in the smooth and non-monotonic SiLU on deep networks, compared with the ReLU. Therefore, the SiLU activation function replaced the ReLu one in the basic and downsampling unit of ShuffleNet V2 for a better overall performance of the model. Finally, the overall architecture of the model was redesigned to reduce the parameters and floating point operations per second (FLOPs). A ShuffleNet for Pest Field (SNPF) model was then constructed for pest identification. Specifically, the number of stacks in the core module was reduced by more than half. The numbers of output channels in the first and last convolutional layers were adjusted from 24 to 32, and from 2048 to 1024, respectively. The experimental results showed that the average identification accuracy and the F1 Score of SNPF on the self-built pest dataset were 79.49% and 78.54%, which were 4.00 percentage points and 3.09 percentage points higher than before, respectively. The parameters and FLOPs of the SNPF model were about 3.74 M and 0.48 G, which were 30.60% and 18.60% lower than the original, respectively. The average identification time of SNPF was 11.9 ms for a single pest image, which was reduced by 57.04%, 50.21%, and 40.50%, respectively, compared with the ResNet 50, GoogLeNet, and EfficientNet B1 models, respectively. The SNPF model can be expected to rapidly and accurately identify crop pests for better pest control. The finding can also greatly contribute to the automatic identification and efficient control of pests during lightweight crop production.