An improved lightweight YOLOv4 model was proposed to realize the accurate, real-time, and robust automatic sorting of bare and shelled shrimp in the shrimp mechanical shelling process under complex scenarios. The CSP-Darknet53 network was replaced by the GhostNet in the YOLOv4 structure. The ability of the model was then improved to extract the features adaptively. The calculation of model parameters was also simplified after improvement. The GhostNet network was used for the YOLOv4 backbone feature extraction, in order to reduce the network model complexity, and the model parameters for better storage capacity and detection efficiency. A lightweight attention mechanism was introduced into the Resblock module of the YOLOv4 backbone feature extraction network, in order to enhance the feature extraction capability of the backbone feature extraction network. The SE attention mechanism module was used to enhance the attention between feature channels. The attention of the network model was improved to the shrimp shell by fitting the relevant feature information to the target channel and suppressing invalid information. The model recognition accuracy was improved to reduce background interference. The original GIoU loss function was replaced with a CIoU loss function to improve the regression effect of the prediction frame. The CIoU loss function made the data obtained from non-maximal suppression more reasonable and efficient. Furthermore, the prediction frame was more accurate to minimize the distance between the centroids of the detection frame and the labelled frame. The lightweight GhostNet-YOLOv4 model was compared with the YOLOv7, EfficientNet Lite3-YOLOv4, ShuffleNetV2-YOLOv4, and MobilenetV3-YOLOv4 models. The results showed that the GhostNet-YOLOv4 model shared the lowest number of parameters and computational effort. An ablation comparison experiment was designed to verify that replacing the backbone feature extraction network and embedding the SE attention mechanism optimized for the module. The replacement of the CSP-Darknet53 backbone feature extraction network with the GhostNet resulted in a 2.9 percentage point improvement in the mAP and a significant reduction in the number of model parameters and output weights, compared with the original model. The addition of the SE attention mechanism improved the anti-interference and feature extraction ability, whereas the mAP was improved by 1.8 percentage points. After replacing the GIoU loss function with the CIoU one, the shrimp recognition accuracy was further improved, where the mAP was improved by 1.4 percentage points. According to the actual operating environment of the shrimp shell inspection test bed, two types of image datasets were produced, namely bare flesh shrimp and shelled shrimp. The GhostNet-YOLOv4, YOLOv3, YOLOv4, and MobilenetV3-YOLOv4 models were used for testing. The results show that the GhostNet-YOLOv4 model achieved detection accuracy and speed of 95.9% and 25 frames/s, respectively. The GhostNet-YOLOv4 model outperformed all other models in terms of detection speed under the condition of guaranteed detection accuracy. The performance of the GhostNet-YOLOv4 network model was evaluated to identify the shrimp shells for four treatments with the changes in light brightness, speed, shrimp posture, and shrimp species. The shrimp shell detection test showed that the average accuracy of shrimp shell recognition reached 90.4%, fully meeting the operational requirements. It indicates that the test bench was suitable for installation on mobile-embedded devices. The GhostNet-YOLOv4 network model still shared excellent generalization performance, when identifying other species of shrimp shells outside the sample set, with an average accuracy of 87.2%.