Abstract: The grape industry has been one of the most important strategies in the rural revitalization and precise poverty alleviation. A serious disturbance to grape planting can be the pests and diseases in recent years. The refined measures of agricultural and biological control are highly required to effectively reduce the occurrence of pests and diseases for economic and safety concerns. Accurate identification and diagnosis of grape diseases and insect pests can be the premise of fine control actions. The traditional identification of pests and diseases relies mainly on the experts or technical personnel for the visual sightings, particularly with the time-consuming, laborious, high cost, low timeliness, and difficult to be widely used. Alternatively, machine learning and computer vision can be expected to serve the promising application of efficient image identification. There is also a great potential to accelerate the identification efficiency of pests and diseases, cost saving, and high accuracy. Machine learning classification includes the feature representation and classifier. However, the manual feature-based identification can only be suitable for the small data sets at present. Most machine learning algorithms also depend on complex image processing and hand-designed features, resulting in low robustness in pests and disease identification, especially in a complex environment. Fortunately, deep learning can be widely expected in the agricultural field, due to the most cutting-edge, modern, and promising technology in recent years. This study aims to treat the small data set and a large number of model parameters in the existing deep learning for higher identification accuracy. A data set of grape pests and diseases was also constructed, containing the healthy leaves, three types of diseased leaves, and 16 types of insect pests. An identification model was then proposed for the grape pests and diseases using the improved MobileNet V2. Firstly, a Coordinate Attention (CA) was embedded in the reverse residual module of MobileNet V2 to enhance the information representation of the model. Secondly, a two-branch feature fusion module was designed using depthwise separable convolution for the better identification performance of the model. Finally, the number of channels was adjusted to streamline the structure of MobileNet V2. As such, a lightweight identification model (named MobileNet_Vitis) was proposed for the grape pests and diseases. The results show that the better performance of the model with the CA module was achieved after 1×1 pointwise convolution. Specifically, the accuracy of the improved model was enhanced by 1.61 percentage points, while the parameter size was reduced by 1.62 MB, compared with the introducing CA module after 3×3 depthwise convolution. The two-branch feature fusion also greatly contributed to the higher identification accuracy, which was improved by 1.4 percentage points than before. In addition, a tradeoff of identification accuracy and speed was obtained to adjust the channels in the model. The identification accuracy and F1 score of MobileNet_Vitis on the dataset were 89.16% and 80.44%, respectively, which was 1.83 percentage points and 9.31 percentage points higher than that of MobileNet V2 before improvement, respectively. More importantly, the parameter size of MobileNet_Vitis was 7.85 MB, which was 8.5% less than that of MobileNet V2. Consequently, the MobileNet_Vitis presented a higher identification accuracy and F1 score with a smaller size of parameters, compared with the ResNet 101, ShuffleNet V2, MobileNet V3, and GhostNet. The inference time of MobileNet_Vitis for a single pest image was 17.53 ms, fully meeting the requirement of fast identification. The improved model can be widely expected to better identify the grape pests and diseases with less complexity. Therefore, the MobileNet_Vitis model can be further deployed to a mobile applet for the grape pests and disease control.