Fruit surface defect inspection is always a challenging project for computer vision automated fruit grading. On the basis of HSL color model, a defect segmentation method based on fuzzy color clustering is proposed. By converting pixel value from RGB color model to HSL color model; defining H, S, L fuzzy sets using triangle membership function, and constructing a fuzzy color set, a fuzzy color similarity measure to calculate similarity between two fuzzy colors was created. This algorithm was applied to tomato defect segmentation, and experimental results show that the accuracy is 96%.