The predictive performance of supervised learning algorithms depends on the quality of labels. In a typical label collection process, multiple annotators provide subjective noisy estimates of the “truth” under the influence of their varying skill‐levels and biases. Blindly treating these noisy labels as the ground truth limits the accuracy of learning algorithms in the presence of strong disagreement. This problem is critical for applications in domains where the annotation cost is high. Such problems are extremely serious in the domain of agricultural imaging in leaf disease classification. Due to the limitation of acquisition methods, noisy examples are often not only the normal mislabelled ones, but also the samples which contain more than one instance. To cope with the combination of the two noises, label smoothing is blended in the point of increasing entropy for uncertain labels, with Taylor cross entropy loss, which is proved to be efficient to solve the problem of artificial noisy labels on public datasets. And the proposed method is called smooth‐Taylor cross entropy loss, which can deal with the real‐world noises in vision leaf disease dataset. Extensive experimental results on cassava leaf disease dataset demonstrate that our proposed approach significantly outperforms the state‐of‐the‐art counterparts.