Abstract: Serious vibration in most mechanical equipment has posed a great impact on the production efficiency of agricultural goods and the service life of key parts. Many efforts have been made on the influence of external excitation on the vibration generation, transmission, and even reduction at present. However, there are only a few studies on the influence of internal excitation on vibration in the ring mold granulator, such as the ring mold and pressing roller. Taking the gear reducer of the 420 ring mold granulator as the research object, this study aims to analyze the dynamic characteristics of the complex transmission, large vibration, and short service life of key parts. The time-varying meshing stiffness and meshing error of helical gears were attributed to the internal excitation of transmission system. A cumulative integral potential energy was used to calculate the time-varying meshing stiffness of gear reducer in a ring mold granulator, thereby determineing the variation of meshing stiffness. Then, a six degree of freedom (DOF) dynamic model was established to determine some parameters for the helical gear transmission system in a ring mold granulator. Since the dynamic response clearly reflected the gear vibration, a modal analysis of transmission was made to effectively avoid the nonlinear dynamic resonance. At the same time, a systematic evaluation was carried out to explore the effects of meshing stiffness and comprehensive meshing error on the dynamic response of helical gear transmission. More importantly, the obtained meshing stiffness of helical gear was utilized to optimize the dynamic response of system for a higher installation accuracy in regular maintenance. Furthermore, a finite element (FE) software was used to verify the accuracy of natural frequency in the dynamic model. A spindle torque test of granulator was also carried out to measure the torque time-domain curve. Fourier transform (FT) was then utilized to transformed into the power spectrum curve. A spectrum analysis demonstrated that there was no resonance of main motor with the transmission system, although the complex internal excitation of a granulator from the torque fluctuation. Additionally, an acceleration vibration test was performed on the whole machine and helical gear transmission. The curve of vibration time domain was measured to transform into the vibration frequency domain. The vibration characteristics of helical gear transmission were analyzed to verify the internal excitation in the whole granulator. Finally, the curve of vibration time-domain was integrated to obtain the displacement fluctuation. The test and theoretical calculation of dynamic response indicated that the internal excitation was an important reason for the vibration of the whole machine.