Rapid and accurate estimation of crude protein (CP) content in pasture plays an important role in the monitoring and management of the forage growth on a wide range of grassland. Crude protein contents of pastures are ideal for characterization using hyperspectral data. In view of the limitation of ground and satellite remote sensing, in this paper, we attempted to achieve an accurate estimation of CP content in forage by unmanned aerial vehicle (UAV)-based hyperspectral remote sensing images with high spatial resolution of the pasture canopy. Although the hyperspectral data of the forage have a large number of bands, the reflectance of the canopy spectrum at each band contains information of various parameters which are from atomic level to plant community level. So, when estimating a physicochemical parameter using the spectral data, we may achieve low prediction accuracy because the spectra are affected by other parameters. Compared with the original spectra, multi-granularity spectral features can provide more sensitive features for inversion of chemical parameters. More importantly, multi-granularity spectral features can extract some hidden weak spectral information, which is of great significance for inversion of low-content physical and chemical indicators. However, in current inversion methods of pasture CP content, there is a lack of effective utilization of spectral multi-granularity information. In view of this, we first proposed a novel multi-granularity spectral feature extraction approach named multi-granularity spectral segmentation (MGSS) to segment each canopy spectrum into multiple spectral features. Second, by using the sequential forward selection method, sensitive components of each feature under a granularity can be selected. Finally, based on these selected components of different features, the inversion models of pasture CP content can be established using two regression methods, i.e., the stepwise multivariate linear regression (SMLR) and the partial least squares regression (PLSR). Taking a typical meadowland in Qinghai Plateau as an example, the detailed experimental analyses have been conducted. Results showed that under the same quantity sensitive components for MGSS and sensitive bands for the raw spectra, on the estimation accuracy of pasture CP content, MGSS was superior to the raw spectra. So the validity of MGSS in improving the accuracy of hyperspectral estimation of CP content in forage was verified. Specifically, under Granularity 23 (G23) of MGSS, the PLSR model achieved the best performance. Its determining coefficient (R2) was 0.937 which was 0.06 higher than that of the optimal model of the raw spectra. And the root mean square error (RMSE) and the mean relative error (MRE) were 1.906 (g/m2) and 8.82%, respectively, which were 0.75 (g/m2) and 1.37 percentage points lower than those of the optimal model of raw spectra. Moreover, on the single and combined components sensitive to CP content in forage, there were three characteristics among the selected components of MGSS, i.e., the agglomeration within the Red Edge range, the dispersion of non-Red Edge range, and the sparsity of strongly sensitive components, which can be helpful for selecting sensitive components. In conclusion, the proposed MGSS achieved the high performance estimation of CP content in forage by UAV hyperspectral imagery. And compared with the raw spectra, MGSS had better performance. This study provides a new technical means for the accurate estimation of CP content in grasslands in large areas.