Hyperspectral remote sensing technology has been widely used in vegetation type mapping, and the integrated use of spectral and texture features of remote sensing data has become an effective way to improve the accuracy of vegetation classification, but the Hughes effect generated by the high-dimensional feature space composed of multi-scale texture features and hyperspectral features reduced the image classification accuracy, and the randomness of feature optimization algorithms such as genetic algorithm (GA) can also cause uncertainty in image classification accuracy. In order to solve those problems, a high-dimensional spectral-texture feature space was established by incorporating spectral features of ZY1 02D Advanced HyperSpectral Imager(AHSI) hyperspectral data and multi-scale texture features extracted from ZY1 02D Visible and Near Infrared Camera(VNIC) data at various window sizes and movement directions, covering the vicinity of Zongjia Town in Dulan County, Qinghai Province, China. Jeffries-Matusita(J-M) distance is employed to assess the sample separability between vegetation types of texture features derived from varying window sizes and movement directions. Based on a combination of feature optimization algorithms (GA algorithm, particle swarm optimization (PSO), generalized normal distribution optimization (GNDO), atom search algorithm (ASO) and marine predators algorithm (MPA)) and classification algorithm (random forest (RF)), GA-RF, PSO-RF, GNDO-RF, ASO-RF, and MPA-RF algorithms were proposed and applied to the vegetation type classification of the high-dimensional spectral-texture feature space. The results show that the texture features extracted from different window sizes and window movement directions showed maximum J-M distance between distinct vegetations. For instance, the J-M distance between wolfberry (old) and wolfberry (new) is highest in the 3×3 window size texture images, while the J-M distance between wolfberry (old) and haloxylon is highest in the 11×11 window size texture images. In the 0° window movement direction texture images, poplar and haloxylon had the best separability to other vegetation types with J-M distances greater than 1.87 and 1.86, respectively. Grass and other vegetation types achieved the best separability with J-M distances greater than 1.18 in the 90° window movement direction texture images. The inclusion of multi-scale texture features improved the overall classification accuracy (OA) by 8.02 percentage. The proposed GA-RF, PSO-RF, GNDO-RF, ASO-RF, and MPA-RF algorithms, when compared to the conventional random forest algorithm, led to an improvement in OA of vegetation classification by 1.32 to 2.41 percentage. Among them, the MPA-RF method exhibited the highest accuracy, achieving an OA and Kappa coefficient of 88.92% and 0.86, respectively. The results of this study demonstrate that texture images extracted from different window sizes and window movement directions are useful in distinguishing between different types of vegetation. The accuracy of vegetation recognition is significantly increased by adding multi-scale texture features to the spectral feature set. The combination of feature optimization algorithm and image classification algorithm by iterative optimization alleviates the randomness of the optimization algorithm results, overcomes the Hughes effect of high-dimensional features, and successfully improves the vegetation classification accuracy. This study proposes innovative ideas for feature extraction, feature optimization, and classification algorithm selection for hyperspectral vegetation classification studies.