A planting area of fruit trees has accounted for up to 13% of the national planting area in Xinjiang, China. Local specialty fruits (e.g. walnut, jujube, apricot, fragrant pear, and apple) have been produced in four prefectures of Southern Xinjiang around the Tarim River Basin, accounting for more than 80% of the total fruit planting area in Xinjiang. It is very necessary to real-time and accurately acquire the fruit tree type and area information under the pattern of forest and crops interplanting, further to improve the quality and efficiency of the specialty fruit industry. Taking the continuous area of forest and crops interplanting in Hotan oasis of southern Xinjiang in China as an example, this study aims to extract the structure information of fruit trees from the satellite images. The high-resolution remote sensing image data with abundant texture and spectral characteristics was also integrated with the medium-resolution sensing image data with multi-temporal characteristics. Firstly, an object-oriented programming was used to extract the high-precision boundary of fruit trees parcel using GF-2 (PMS) image data. The GF-2 image data was divided into the winter (February) and summer-autumn (July-September). The Normalized Difference Vegetation Index (NDVI) was determined to identify the spectral and texture features between target objects and other ground objects. Four plots were obtained to gradually eliminate the non-target ground categories. Then, the NDVI time series products were constructed using multi-temporal Sentinel-2 image data. A decision tree model was established to extract the interplanting walnut, pure walnut, jujube, and grape orchard, according to the characteristics of phenology. There were many peaks and troughs in the NDVI time series curve of fruit trees in one year. The peak represented the flourishing period of fruit trees’ growth, and the trough reflected the orchard management (such as irrigation and pruning branches). There were some outstanding and different time windows for the NDVI timing series of pure walnut, interplanting walnut, jujubes, and grapes, particularly for the rarely staggered overlapping. These differences were greatly contributed to the classification of fruit trees. Finally, the multi-phase orchard classification was overlaid the high-resolution of fruit trees parcel, further to obtain the distribution of fruit crops in the study area. The research results show that the major area of fruit was 4.28×105 hm2 here, with 3.31×105 hm2 of the walnut orchard (including interplanting and pure walnut), 8.29×104 hm2 of jujube, and 1.40×104 hm2 of grapes. The area of interplanting walnuts accounted for 63.8% of the total fruit area, The user accuracy and overall classification accuracy were both exceed 90%, and the Kappa coefficient was 0.95, fully meeting the accuracy requirements of agroforestry classification at the county and city level. The relative accuracies of walnuts, jujube, and grapes were 62.1%, 97.8%, and 85.2%, respectively, using remote sensing extraction, compared with the forestry survey in 2019. The areas of jujube and grape using remote sensing extraction were close to the forestry survey data. The walnut planting areas in the Hotan area were mainly distributed in the upper reaches of Yurunkax and Karakax River with suitable soil and water conditions. The jujube trees were mostly distributed in the downstream oasis-desert ecotone, and the grapes were mostly distributed in the sandy desert of the lower reaches of the Karakax River. This finding can provide a strong reference for the fruit tree extraction under the pattern of forest and crop intercropping.