Alfalfa has been one of the most widely grown forage crops around the world. The "king of forage", alfalfa can be known as the high production of grass rich in nutrients. Timely and accurate monitoring of alfalfa growth and yield can be a high demand for large-scale agricultural production in recent years. In this study, a series of yield prediction models were established for the alfalfa using Unmanned Aerial Vehicle (UAV) remote sensing and machine learning. A field experiment was also performed in Zhuozhou City, Hebei Province, China. Five irrigation modes and a rainfed treatment were set for different alfalfa growth statuses and yield prediction. The UAV multispectral platform was used to monitor the alfalfa during the growing stage of branching, budding, and early blooming. A correlation analysis was then made to determine the alfalfa yields and 11 spectral parameters (vegetation indices). The top five correlation indexes were picked out for each growth stage. Then, a Structure from Motion (SfM) imaging technology was used to reconstruct the plant height of alfalfa. The high accuracy of prediction was achieved with the determination coefficient (R2) of 0.86, the Root Mean Square Error (RMSE) of 6.8 cm, and the Normalized Root Mean Square Error (NRMSE) of 14.4%, compared with the measurement. Eventually, three yield prediction models were established by Support Vector Regression (SVR). Specifically, the measured yield was always used as the output, but the input was used the vegetation indices, the plant height, as well as the combination of vegetation indices and plant height. Soil noise was also removed from the vegetation indices for the best performance. The results showed that there was the most significant correlation of vegetation indices with the measured alfalfa yield during the branching, budding, and early blooming stage. In the branching stage, the R2 values of the three models were between 0.4 and 0.6. The NRMSEs of the models were greater than 30% with the vegetation indices and plant height as the input variables. In the budding stage, the R2 values were above 0.8, and the NRMSEs were less than 20% for all three models. The highest accuracy was achieved in the model with the combination of vegetation indices and plant height as the input variables, with the R2 of 0.87, the RMSE of 564 kg/hm2, and the NRMSE of 16.1%. The R2 value was 0.72 in the yield prediction model with the vegetation indices only as the input variable in the early blooming stage, indicating a lower accuracy than that in the budding stage. However, the R2 values were 0.89 and 0.90 in the models with the plant height only, and the combination of vegetation indices and plant height as the input variables, respectively, while the NRMSEs were lower than 15%, indicating a higher accuracy than that in the budding stage. A higher accuracy of the yield prediction model was found, as the alfalfa grew. The combination of plant height and vegetation indices as the input variables can also be expected to improve the accuracy of the yield prediction model. The best yield prediction was achieved in the model with the combination of five vegetation indices and plant height as the inputs at an early blooming stage, with the R2 of 0.90, the RMSE of 500 kg/hm2, and the NRMSE of 14.3%. The optimal model can be strongly recommended for the rapid and accurate prediction of alfalfa yield. The finding can provide technical support to the large-scale production and precision management of alfalfa.