A convolutional neural network (CNN) can be expected to grade the mango appearance quality for high accuracy. However, it is still challenging to implement CNN on low-cost and performance-constrained edge devices, due to the computation- and storage-intensive CNN algorithm. This has greatly limited the application and promotion of CNN for mango appearance quality grading. In this study, a low-cost grading device was proposed to balance the grading accuracy, speed, power consumption, and detection cost of mango appearance quality using FPGA-accelerated CNN. Firstly, a lightweight network called “Compact MobileNet” (CMNet) was designed with a simple and efficient structure. The parameter and computational complexity of CMNet were significantly reduced to compress the network structure suitable for the deployment of edge devices with acceptable accuracy. Secondly, Batch normalization (BN) layer fusion and model quantization were used to further reduce the storage requirements and computational complexity of the model, in order to accelerate the execution speed of CMNet on cost- and performance-constrained edge devices. Meanwhile, an FPGA-based hardware accelerator was designed for CMNet. Since the basic function of the accelerator was achieved, the high-level synthesis (HLS) optimization, including “unrolling the for-loop”, “pipelining the for-loop”, and “array partitioning”, were used to optimize the parallelism of the hardware accelerator. Finally, a grading device was developed for the appearance quality of mangoes using an FPGA development board, specifically the “ZYNQ Z7 Lite 7020” model. An OV5640 camera module, the CMNet network acceleration circuit, and an HDMI interface were integrated to enable the mango image collection, real-time detection, and display of appearance quality. A series of experiments were conducted in a laboratory environment, in order to verify the performance of CMNet and the grading device. The Dosehri mango dataset downloaded from the internet was used to train and validate CMNet. The mango appearance quality was classified into three grades, according to China's agricultural industry standard for mango (NY/T 492-2002). The model comparison demonstrated that the CMNet shared a very lightweight model scale, only with 0.015M parameters and 7.2MFlops computations, with a high grading accuracy of 96.4%. Therefore, the CMNet was suitable for deployment on a resource-limited FPGA terminal. The assessment experiments on the FPGA-based accelerator of CMNet demonstrated that the accelerator significantly expedited the quality grading, with a speed of 0.11 s per frame. The accelerator was capable of detecting the mango appearance quality in real time. The performance comparison among various computing devices revealed that the power consumption of the FPGA-based grading device was 2.6 W, which was the lowest among the tested devices. The FPGA-based detection device offered substantial advantages, in terms of power consumption and portability, compared with the CPUs and GPUs. The FPGA excelled in the best unit cost performance, compared with the portable Raspberry Pi devices and Android smartphones. In summary, all experimental results indicate that a low-cost, low-power, high-accuracy, and high-speed grading device was suitable for assessing mango appearance quality. This device can also be effectively utilized for the real-time grading of mango quality in the field. Furthermore, this finding can provide a strong reference for similar agricultural applications, where artificial intelligence can be employed to rapidly extract information from video data.