Aiming at the problems of low efficiency of manual grading and inaccurate mechanical grading of peanut pods, a convolutional neural network peanut pod grades image recognition method based on transfer learning was proposed. By using the operations of the flip, rotation, translation, contrast transformation, and brightness transformation, the obtained five grades (first-grade pod, second-grade pod, third-grade pod, fourth-grade abnormal pod, and fifth-grade damaged pod) of peanut pod images were expanded and preprocessed, thus the peanut pod grades image data set was established. The 60% of data was randomly selected as the training set, 20% of data was randomly selected as the validation set, and the remaining 20% as the test set. The performance of peanut pod image classification based on the GoogLeNet, ResNet18, and AlexNet was compared and analyzed. The peanut pod grades recognition model was improved by transferring the AlexNet convolution layers. The local response normalization was replaced by batch normalization, and the activation function was placed in different positions before and after the batch normalization layer, so that four different recognition-training models were designed, including the PA-I model, PA-II model, PA-III model, and PA-IV model. The transfer learning contrast experiments and the hyperparameter optimization experiments of the learning rate carried out for the four improved AlexNet models proposed above. The effects of the unsaturated activation function (ReLU) and improved unsaturated activation function (LReLU) on the performance of the model were studied. The experimental results showed that the training time of the AlexNet model was the least on the basis of satisfying the test accuracy and the learning rate of transfer learning based on the improved AlexNet model was a very important hyperparameter that needed to be optimized. If the learning rate is chosen too high, the model training oscillates seriously and even can’t train normally; if the learning rate too small, the model training slow. An appropriate learning rate can speed up the training and improve the recognition ability of the model. When the learning rate was updated automatically, the model with batch normalization had better performance than local response normalization, which could make the model get higher accuracy and lower loss value. When the coefficient of activation function LReLU was 0.000 1, the performance of the LReLU used in the model was equivalent to that of the ReLU used in the model, therefore LReLU had no substantial impact on the training results of the model. The addition of batch normalization and reduction of parameters in the model reduced 220 s training time and improved the model’s performance. The classification accuracy of the proposed peanut pod grades recognition model for the first-grade pod, second-grade pod, third-grade pod, fourth-grade abnormal pod, and fifth-grade damaged pod was 93.57%, 97.14%, 99.29%, 87.14%, and 100% respectively and the average classification accuracy reached 95.43%, and F1-scores achieved 96.32%, 97.49%, 99.64%, 92.42%, and 94.50% respectively. The model proposed in this study had high classification accuracy for peanut pod grades and could provide a reference for the precise classification of other agricultural products.