Labor transfer is regarded as an crucial way to increase family income and ameliorate poverty for poverty-stricken families in mountainous areas in the southwest China. The negative consequence is that it causes labor loss, slowing or damaging local social-economic development. It is hence important to implement national strategies to alleviate poverty while in the meantime revitalizing local economy, in which understanding spatial pattern of the migration workers and the underlying factors plays a key role. Taking Changshou District at Chongqing as an example, this paper surveyed and analyzed migrating labors from poverty-stricken families in the villages using a comprehensive evaluation method. Using the geographic detecting model and the OLS regression model, we diagnosed the main factors underpinning the difference in labor transfer between the villages in these areas. The main results were: 1) There was a regional difference in labor transfer from poverty-stricken families between the villages, characterized by "two-highs and one-low", with the "two-highs" representing Mingyue Mountainous Area in the West and Huangcao Mountainous area in the Southeast and the "one-low" representing the southwest-central low hilly area. 2) The main factors underlying the difference in labor transfer from the poverty-stricken families between the villages were age, cultivable land per capita, land slope and labor training, and the distribution of the decisive power of these four factors was 0.410, 0.396, 0.363 and 0.301, respectively. 3) Labor transfer was a result of the interaction of multiple factors, and the interaction of any pair of factors was greater than that of any single factor. 4) Based on the factors underlying the difference in labor transfer from poverty-stricken families between the villages, we recommend that each village should take strategies based on its location, economy and society to orderly implement poverty-alleviation strategies and revitalize local economy so as to reduce labor transfer.