Apple diseases have frequently threatened the yield and quality of fruit production, leading to irreversible damage to the plant growth. Meanwhile, the treatments vary significantly in the types of diseases. Therefore, timely and accurate identification can be highly urgent to prevent the spread of the diseases. Traditional and manual identification is often time-consuming and laborious depending mainly on experienced experts. Fortunately, deep learning can open up a new way to control diseases. However, the general model of disease identification is still lacking, due to the complex real environment in the field and few large datasets of apple fruit diseases. In this study, a lightweight GHTR2-YOLOv5s (YOLOv5s with Ghost structure and TR2 module) and transfer learning-based model was proposed to identify the apple fruit diseases, particularly with the high accuracy and less complex architecture. The specific procedure was given as followed. Firstly, an image dataset of apple fruit diseases was collected in the field, including the bitter pit, anthracnose, ring disease, and fruit rust. A series of data enhancement operations were then adopted on the image dataset to avoid over-fitting for the convergence. Secondly, The Ghost structures (Ghost Conv and Ghost Bottleneck) were added to the backbone and neck network of YOLOv5s. A small baseline model was then obtained to adjust the width of the feature map, in order to reduce the storage occupation of the model for the high detection speed. The Convolutional Block Attention Module (CBAM) was adopted to assign the feature weights to the baseline model, where the effective features were selected after evaluation. The Bidirectional Feature Pyramid Network (BIFPN) was used to enhance the robustness and generalization ability. Two Transformer (TR2) encoder modules were stacked for the detection head to enhance the global information of the model. Thirdly, the transfer learning was obtained to improve the convergence speed and generalization ability, where the knowledge was firstly learned from the image dataset of apple leaves diseases, and then transferred to the GHTR2-YOLOv5s model in the disease identification of apple fruits. Finally, the performance of the model was verified by various experiments under different conditions. The experimental results demonstrated that: 1) The proposed GHTR2-YOLOv5s model reduced the model size for high accuracy. The model size and average identification speed of the GHTR2-YOLOv5s model were 2.06 MB and 0.065 s/sheet, which were 1/6 and 2.5 times that of the original YOLOv5s. The mean Average Precision under Intersection over Union (IoU)=0.5 (mAP0.5) reached 0.909, which was higher than that of the original. 2) Transfer learning improved the model accuracy, while shortening the convergence time of the model. The mAP0.5 of the model reached 0.916 by combining the online data enhancement and the secondary transfer learning, which was 8.5% higher than that of the original version. Consequently, the improved model can be expected to identify the apple fruit diseases rapidly and accurately, indicating higher accuracy and faster convergence with a less model size than before. The finding can provide a feasible and promising reference for the intelligent diagnosis of apple fruit diseases in apple production.