Diesel engines are widely used in vehicles, construction machinery and generator sets because of their reliability and fuel economy. However, the diffused combustion present in diesel engines increases emissions. The combustion performance and emission characteristics are directly affected by the fuel/air mixing process, by designing a special structure on the combustion chamber wall to guide the distribution of fuel spray in the combustion chamber, the quality of fuel/air mixing in the combustion chamber can be improved, and the process of fuel/air mixing and combustion can be improved, and the significant research focuses on improving the efficiency and fuel/air mixing process of diesel combustion system. To improve air efficiency in the center and squish areas of the combustion chamber, a new separated swirl combustion system (SSCS) was developed in this study. The SSCS chamber consists of the inner chamber, the outer chamber and the separated chamber, and there are two circular ridges. The injector used in the SSCS has two types of holes: upper and lower, and they are arranged in alternating order, and the angles of these two kinds of holes are different, which renders two distinct sprays. The different sprays collide with the different circular ridges in the chamber. When the spray collides with the circular ridges, swirls form, which improves air utilization in the chamber and accelerates the fuel/air mixture. As a new combustion system, the fuel/air mixture formation in the chamber is different from that of traditional combustion system. To make a better understanding of the mechanism of fuel/air mixture formation in SSCS, a single-cylinder diesel engine test system and a simulation method were used to analyze the combustion and emission performance of the SSCS under different conditions. The combustion and emission performance of the SSCS under different speeds, loads and excess air coefficients was tested and compared with a DSCS in a single-cylinder engine. While soot emissions from the SSCS can be tested in a real-world single-cylinder engine, the soot formation characteristics cannot be tested. Therefore, to understand the mechanisms behind soot formation in the SSCS, soot evolution must be investigated using a simulation model. Then a new phenomenological soot model using KIVA-3V R2 code and integrated with a reduced n-heptane/methane/PAH mechanism was developed and used to simulate soot behaviors in the SSCS. Incipient soot particles are fewer and soot mass is lower in the SSCS than that in the DSCS at the same cases. The mechanisms that reduce the soot emissions in the SSCS were revealed by comparing the equivalence ratio distribution and fuel distribution in the cylinder. The mechanism of soot formation in SSCS was also analyzed using the KIVA-3V Release 2 code. The experiment results show that the SSCS effectively reduces fuel consumption and soot emission, with a maximum decrease in fuel consumption of approximately 5.41% (when the power was 17 kW) and a maximum decrease in soot emission of approximately 20.48% (when the power was 43 kW). The simulation results show that the percentage of fuel with an equivalence ratio between 0.66 to 2 is higher in the SSCS than that in the DSCS, while the percentage of fuel with an equivalence ratio more than 2 is lower in the SSCS. So the equivalence ratio is more uniform in the SSCS, and less fuel is consumed, thermal efficiency is improved and soot emission is reduced. The SSCS is helpful to reduce emissions and fuel consumption in DI diesel engines.