Rosa roxburghii is widely distributed in warm temperate zone and subtropical zone, mainly in Guizhou, Yunnan, Sichuan and other places in China. Panxian and Longli are the most abundant the most varieties and the highest yield Rosa roxburghii resources in Guizhou. The harvesting of Rosa roxburghii fruit is the most time-consuming and labor-consuming work in Rosa roxburghii production, and its labor input accounts for 50%-70% of the production process. Hand-picking of Rosa roxburghii fruit is of high cost, high labor intensity and low picking efficiency. In recent years, convolutional neural network has been widely used in target recognition and detection. However, there is no relevant literature on the application of neural network in Rosa roxburghii fruit recognition. In this paper, in order to realize rapid and accurate identification of Rosa roxburghii fruits in natural environment, according to the characteristics of Rosa roxburghii fruits, the structure and parameters of VGG16, VGG_CNN_M1024 and ZF network models under the framework of Faster RCNN were optimized by comparing them. The convolutional neural network adopted bilinear interpolation method and selected alternating optimization training method of Faster RCNN. ROI Pooling in convolutional neural network is improved to ROI Align regional feature aggregation. Finally, VGG16 network model is selected to make the target rectangular box in the detection result more accurate. 6 540 (80%) of 8 175 samples were selected randomly as training validation set (trainval), the remaining 20% as test set, 80% as training set, the remaining 20% as validation set, and the remaining 300 samples that were not trained were used to test the final model. The recognition accuracy of the network model for 11 Rosa roxburghii fruits was 94.00%, 90.85%, 83.74%, 98.55%, 96.42%, 98.43%, 89.18%, 90.61%, 100.00%, 88.47% and 90.91%, respectively. The average recognition accuracy was 92.01%. The results showed that the recognition model trained by the improved algorithm had the lowest recall rate of 81.40%, the highest recall rate of 96.93%, the lowest accuracy rate of 85.63%, the highest 95.53%, and the lowest F1 value of 87.50%, the highest 94.99%. Faster RCNN (VGG16 network) has high recognition accuracy for Rosa roxburghii fruit, reaching 95.16%. The recognition speed of single fruit is faster, and the average recognition time of each Rosa roxburghii fruit is about 0.2 seconds. The average time has some advantages, which is 0.07 s faster than the methods of Fu Longsheng. In this paper, a Faster RCNN Rosa roxburghii fruit recognition network model based on improved VGG16 is proposed, which is suitable for Rosa roxburghii fruit recognition model training. The algorithm proposed in this paper has good recognition effect for Rosa roxburghii fruit under weak and strong illumination conditions, and is suitable for effective recognition and detection of Rosa roxburghii fruit in complex rural environment. This paper is the first study on the depth extraction of Rosa roxburghii fruit image features by using convolution neural network. This research has high recognition rate and good real-time performance under natural conditions, and can meet the requirements of automatic identification and positioning picking of Rosa roxburghii fruit. It lays a certain foundation for intelligent identification and picking of Rosa roxburghii fruit, and opens a new journey for the research of automatic picking technology of Rosa roxburghii fruit.