YOLOv8-Coal: a coal-rock image recognition method based on improved YOLOv8

DCNv3 module

This study introduces the DCNv3 module as a means of enhancing the accuracy of coal rock image recognition in YOLOv8. The conventional convolution technique is constrained by the fixed sample position and exhibits limited generalization capability. DCNv3 is a network architecture that utilizes deformable convolution to allow the network to flexibly modify the shape and size of the convolution kernel. This adaptation helps the network better accommodate the specific characteristics of the image’s local features. The introduction of dynamically deformable convolution kernels enables this functionality, as illustrated in Fig. 2.

DCNv3 introduces an offset to the standard convolution’s sampling position, allowing the grid to deform freely. This offset can be adjusted through backpropagation, resulting in a sensing field that closely matches the object’s shape. This enables more accurate feature extraction, as demonstrated in Eq. (1).

(1) $$\mathit{y}\left(p_0\right)=\sum _{g=1}^G\sum _{k=1}^K{\mathit{w}}_g{\mathit{m}}_{gk}{\mathit{x}}_g\left(p_0+p_k+\mathrm{\Delta }{p}_{gk}\right)$$where $G$ is defined as the total number of aggregated groups, and $K$ represents the number of sampling points within each aggregation group. Specifically, for the $g$-th group, the dimension of the group is denoted as $C^{\mathrm{\prime }}=C/G$, and the location-irrelevant projection weights of the group is denoted as ${\mathit{w}}_g\in R^{C\times C^{\mathrm{\prime }}}$. The modulation scalar of the $\mathrm{k}$-th sampling point in the $\mathrm{g}$-th group is ${\mathit{m}}_{gk}\in R$, which is normalized by a Softmax function along dimension $\mathrm{K}$. The sliced input feature map is denoted as ${\mathit{x}}_g\in R^{C^{\mathrm{\prime }}\times H\times W}$. $p_0$ represents the current pixel provided, while $\mathrm{\Delta }{p}_{gk}$ represents the offset that corresponds to the grid sampling position $p_k$ of the $g$th group.

DCNv3 features weight sharing among convolutional neurons, a multi-grouping mechanism, and a modulation factor generating approach. Sharing weights among convolutional neurons draws on the idea of separable convolutions, separating the original convolutional weights into two parts: channel-by-channel convolution and point-by-point convolution, so that its parameter and storage complexity are no longer linearly related to the total number of sampling points, which improves the efficiency of the model; the multi-grouping mechanism is to divide the spatial aggregation process into multiple groups, each of which has independent sampling offsets and modulation scales, so that different groups can have different spatial aggregation patterns on the convolution layer, which can better capture the changing patterns of the target at different locations and scales; and the modulation factor generation method is to replace the Sigmoid normalization based on bit-by-bit operation with Softmax normalization based on sample points, so that the sum of modulation scalars is restricted to 1, which makes the training process of the model more stable at different scales.

The DCNv3 module is well-suited for addressing the intricate texture and irregular shape issues that are frequently encountered in coal rock photos. This module increases the model’s flexibility and recognition accuracy for coal-rock-specific textures by using a dynamically changeable convolutional kernel. Additionally, it enhances the model’s generalization capacity in a variable environment. DCNv3 is very good in accurately distinguishing between coal and rock. It can effectively address the challenges of classifying these materials, which often have similar textures. As a result, DCNv3 greatly enhances the accuracy and efficiency of recognition. Hence, the implementation of DCNv3 not only improves the efficiency of the current model architecture, but also introduces a novel approach for the targeted job of recognizing coal-rock images, thereby enhancing the model’s performance in real-world scenarios.

PSA module

To enhance the accuracy of identifying and locating the important characteristics in the image of the coal rock, we have introduced the PSA module. The structure of this module is illustrated in Fig. 3. The PSA is a sophisticated attention mechanism that improves the model’s ability to concentrate on important aspects by using polarization operations. It also reduces the impact of redundant information, allowing the model to better acquire and utilize crucial information.

The PSA is bifurcated into two branches: the channel dimension and the spatial dimension. Equation (2) displays the formula used to assign weight to the channel dimension.

(2) $${\mathit{A}}^{ch}\left(\mathit{X}\right)=F_{SG}\left[{\mathit{W}}_{z|{\theta }_1}\left(\left({\sigma }_1\left({\mathit{W}}_v\left(\mathit{X}\right)\right)\times F_{SM}\left({\sigma }_2\left({\mathit{W}}_q\left(\mathit{X}\right)\right)\right)\right)\right)\right]$$where ${\mathit{A}}^{ch}\left(\mathit{X}\right)\in {\mathrm{R}}^{C\times 1\times 1}$, $\mathit{X}$ represents the input feature tensor, ${\mathit{W}}_q$, ${\mathit{W}}_v$ and ${\mathit{W}}_z$ denote 1 × 1 convolution layers, ${\theta }_1$ is an intermediate parameter for the convolution of these channels, ${\sigma }_1$ and ${\sigma }_2$ are two dimensionality reduction operations, $F_{SM}(\cdot )$ represents the Softmax function, $F_{SM}\left(\mathit{X}\right)={\sum }_{j=1}^{N_p}\frac{e^{x_j}}{{\sum }_{m=1}^{N_p}{e}^{x_m}}{x}_j$, $F_{SG}(\cdot )$ represents the Sigmoid function, “ $\times$” means matrix dot product operation, and the number of internal channels between ${\mathit{W}}_{\mathit{v}}\mid {\mathit{W}}_{\mathit{q}}$ and ${\mathit{W}}_z$ is $C/2$. Then the output of the channel dimension is ${\mathit{Z}}^{ch}={\mathit{A}}^{ch}\left(\mathit{X}\right){\odot }^{ch}\mathit{X}\in R^{C\times H\times W}$, where ${\odot }^{ch}$ denotes a channel-wise multiplication operator.

Equation (3) displays the formula used to assign weight to the spatial dimension.

(3) $${\mathbf{A}}^{sp}\left(\mathit{X}\right)=F_{SG}\left[{\sigma }_3\left(F_{SM}\left({\sigma }_1\left(F_{GP}\left({\mathit{W}}_q\left(\mathit{X}\right)\right)\right)\right)\times {\sigma }_2\left({\mathit{W}}_v\left(\mathit{X}\right)\right)\right)\right]$$where ${\mathit{A}}^{sp}\left(\mathit{X}\right)\in {\mathrm{R}}^{1\times H\times W}$, $\mathit{X}$ represents the input feature tensor, ${\mathit{W}}_q$ and ${\mathit{W}}_v$ denote 1 × 1 convolution layers, ${\sigma }_1$ and ${\sigma }_2$ are two dimensionality reduction operations, ${\sigma }_3$ is an ascending operation, $F_{SG}(\cdot )$, $F_{SM}(\cdot )$ and $F_{GP}(\cdot )$ stand for the Softmax function, the Sigmoid function, and the global pooling respectively, $F_{GP}\left(\mathit{X}\right)=\frac{1}{H\times W}{\sum }_{i=1}^H{\sum }_{j=1}^{W}X\left(:,i,j\right)$, “ $\times$” means matrix dot product operation. Then the output of the spatial dimension is ${\mathit{Z}}^{sp}={\mathit{A}}^{sp}\left(\mathit{X}\right){\odot }^{sp}\mathit{X}\in R^{C\times H\times W}$, where ${\odot }^{sp}$ denotes a spatial-wise multiplication operator.

The outputs of these two branches are fused in parallel to obtain the output of the structure as shown in Eq. (4).

(4) $$PS{A}_p\left(\mathit{X}\right)={\mathit{Z}}^{ch}+{\mathit{Z}}^{sp}={\mathit{A}}^{ch}\left(\mathit{X}\right){\odot }^{ch}\mathit{X}+{\mathit{A}}^{sp}\left(\mathit{X}\right){\odot }^{sp}\mathit{X}$$where “ $+$” is the element-by-element addition operator.

The PSA mechanism integrates spatial and channel dimensions to achieve precise pixel-by-pixel regression using polarization filtering and High Dynamic Range (HDR), while preserving high resolution in both dimensions. Polarization filtering compresses features in one dimension while preserving high resolution in the perpendicular dimension, minimizing information loss from downscaling. HDR enhances the dynamic range of attention by applying Softmax normalization to the minimum tensor in the attention module, followed by projection mapping using a Sigmoid function to ensure all parameters are within the range of 0 to 1.

By incorporating the PSA module into the feature fusion network, it becomes possible to assign different weights to features and merge them at various levels. This mechanism effectively mitigates the impact of background noise and irrelevant information by enhancing the focus on crucial characteristics in coal rock images. As a result, the model’s recognition efficiency and accuracy are significantly improved. The PSA module is particularly effective when working with coal rock images that have intricate textures and irregular shapes. It ensures that the network utilizes the feature information from each layer to its fullest extent, thereby optimizing the overall process of extracting and recognizing features. The implementation of this focused feature processing strategy allows the model to achieve greater accuracy and resilience in recognizing coal rock images, particularly in situations with uneven lighting and visual obstruction.

C2fGhost module

C2fGhost is employed as an enhanced lightweight module to minimise model computation without compromising accuracy. Deep neural networks typically include feature maps that contain abundant information, but often exhibit redundancy due to the presence of numerous pairs of comparable feature maps. The Ghost module utilises cost-effective linear operations and produces intrinsic feature maps to enhance the generation of feature maps. This approach allows for the creation of feature maps with reduced computational requirements and fewer parameters, resulting in lower computational costs without compromising performance.

The distinction between regular convolution and the Ghost module is illustrated in Fig. 4. During the process of forward propagation, the Ghost module initially creates an output feature map by regular convolution. Subsequently, this feature map is utilized to construct a second feature map through cost-effective linear operations. Ultimately, the two feature maps are combined in the channel dimension to provide the ultimate result. The Ghost module is more efficient than regular convolution as it decreases the number of parameters and computations while maintaining the same size of the output feature maps.

Assume that the input data is $h\times w\times c$, the output data is $h^{\prime }\times w^{\prime }\times n$, and the size of the convolution kernel is $k\times k$, where $h$ and $w$ are the height and width of the input data, $h^{\prime }$ and $w^{\prime }$ are the height and width of the output data, and $c$ and $n$ are the number of input and output channels, respectively. Then the computation of ordinary convolution is $n\times {\mathrm{h}}^{\prime }\times w^{\prime }\times c\times k\times k$. The Ghost module is able to generate the same number of feature maps as the ordinary convolution using fewer operations. Assuming that $d\times d$ is the kernel size of the linear operation and it is of similar order of magnitude to $k\times k$, and $s$ denotes the number of cheap linear operations and $s\ll c$, the computational amount of the Ghost module is ${\displaystyle \frac{n}{s}\times h^{\mathrm{\prime }}\times w^{\mathrm{\prime }}\times c\times k\times k+\left(s-1\right)\times {\displaystyle \frac{n}{s}\times h^{\mathrm{\prime }}\times w^{\mathrm{\prime }}\times d\times d}}$. Thus, the conventional convolution is substituted with the Ghost module, which exhibits a theoretical speed-up ratio, as depicted in Eq. (5).

(5) $$\begin{array}{rl}{r}_s& ={\displaystyle \frac{n\cdot h^{\mathrm{\prime }}\cdot w^{\mathrm{\prime }}\cdot c\cdot k\cdot k}{\frac{n}{s}\cdot h^{\mathrm{\prime }}\cdot w^{\mathrm{\prime }}\cdot c\cdot k\cdot k+\left(s-1\right)\cdot \frac{n}{s}\cdot h^{\mathrm{\prime }}\cdot w^{\mathrm{\prime }}\cdot d\cdot d}}\\ & ={\displaystyle \frac{c\cdot k\cdot k}{\frac{1}{s}\cdot c\cdot k\cdot k+\frac{s-1}{s}\cdot d\cdot d}\approx {\displaystyle \frac{s\cdot c}{s+c-1}\approx s}}\end{array}.$$

As can be seen from Eq. (5), the computation of the ordinary convolution is approximately $s$ times that of the Ghost module. The parametric quantity is computed by a similar process, again approximated by $s$.

GhostBottleneck was introduced as an extension of Ghost modules, incorporating a shortcut connection and two stacked Ghost modules. The initial Ghost module is employed to augment the quantity of channels, whilst the subsequent module is utilized to diminish the amount of channels in order to align with the shortcuts. Shortcuts are employed to connect the inputs and outputs of the two modules in order to generate the output of the GhostBottleneck. The C2fGhost module is utilized as a replacement for the Bottleneck in the C2f module. The C2fGhost module incorporates the benefits of the Ghost module while reducing the number of parameters and processing requirements. This module offers an efficient solution for coal-rock identification, particularly in situations with limited computational resources, as depicted in Fig. 5.

The incorporation of the C2fGhost module into the backbone network significantly decreases the parameters and performance demands of the model, allowing the deep learning model to efficiently operate on resource-limited end devices. The module employs a lightweight feature extraction strategy while analysing photos of coal rock. This approach not only preserves the capacity to extract features with high precision, but also enhances the model’s suitability and usefulness in real-world application scenarios.

Coal-rock dataset

The dataset utilized in this study is sourced from the School of Mining Engineering at Heilongjiang University of Science and Technology. It comprises 70 images obtained from two distinct coal mines, as depicted in Fig. 6. The two scenes were captured using an Olympus TG-320 and a Huawei PCT-AL10 with resolutions of 4,288 × 3,216 and 4,000 × 3,000, respectively. Coals typically have a darker color and sparse texture, while rocks have a lighter color and dense texture, making them easily distinguishable. Recognizing coal in practice can be challenging due to various factors, such as image defocus from camera limitations, coal being partially obscured by workers, similar colors between coal and rock in dim mine lighting, and color changes in coal and rock from water exposure. Several image examples are displayed in Fig. 7. To enhance the precision of coal mining operations and reduce the chances of errors, a prudent approach is taken by deliberately disregarding areas that are challenging to identify due to image quality issues like poor lighting and image blurriness during image annotation. This action aims to guarantee the dataset’s quality, thereby enhancing the reliability of the coal identification process.

Step 1: Image acquisition and pre-processing. A double-three interpolation algorithm was utilized to adjust the resolution of an image from 4,288 × 3,216 to 4,000 × 3,000, addressing the issue of inconsistent resolution from various acquisition devices. Afterwards, each image was divided into 12 non-overlapping 1,000 × 1,000 blocks. This was achieved by cropping the image four times horizontally and three times vertically, creating a total of 840 blocks. In 527 blocks containing coal with specific targets were chosen through manual filtering. The coals in the images were labeled using the LabelImg tool, and a total of 813 coal targets were identified. The study exclusively focused on annotating images of a specific target category, namely “coal”, and did not include any other categories. By focusing on a single target category, this method simplifies the training and validation processes of the model.

Step 2: Dataset segmentation. The blocks labeled above are divided into three sets for different purposes. The training set consists of 70% (368 images) and is used to train the deep learning model. This set provides enough samples for the model to learn the complex features in the data. The validation set, which comprises 20% (106 images), is used for model tuning and to prevent overfitting. Finally, the test set consists of the remaining 10% (53 images) and is used to independently evaluate the performance of the final model. The data partitioning strategy described here has been extensively utilized in numerous deep learning research studies and has demonstrated its efficacy in achieving a balance between model training and evaluation requirements (Faghani et al., 2022; Humayun, Bhatti & Khurshid, 2023). Furthermore, this allocation adheres to commonly employed methodologies in numerous studies, facilitating straightforward comparisons with other research.

Step 3: Image enhancements. To ensure accurate coal-rock identification, a significant amount of data is required. Offline data enhancement through image panning was utilized to transform the images into multiple 640 × 640 images with overlapping sections. Beginning from the upper left corner pixel of every 1,000 × 1,000 block, cropping was done horizontally at intervals of 180 pixels, allowing for a total of three cropping actions. Next, 180 pixels are scrolled vertically and cropping proceeds horizontally, repeating the process. Ultimately, nine 640 × 640 images were acquired from each 1,000 × 1,000 block following data enhancement. By enhancing the images, it can simulate the distribution of coal from various shooting angles, hence improving the model’s recognition performance in different situations and strengthening its robustness.

The final enhanced dataset comprises 4,743 images, consisting of 3,312 training images, 954 validation images, and 477 test images. Figure 8 illustrates the sequence of steps involved in processing the dataset, while Table 1 provides the specific quantities of images and samples at each stage of the dataset.

Experimental environment and parameter configuration

The study’s model training and evaluation were conducted on a high-performance server featuring an Intel(R) Xeon(R) Platinum 8358P @ 2.60 GHz processor, 80 GB of RAM, and an NVIDIA GeForce RTX 3090 24 GB graphics card. Furthermore, the server operates on the Ubuntu 20.04 LTS operating system, equipped with CUDA 11.8 and CuDNN 8.4.0. The software utilizes Python 3.8 as the runtime environment and depends on the PyTorch 2.0.0 deep learning framework, along with torchvision 0.15.2 for image processing. It also incorporates the Ultralytics 8.0.195, mmcv 2.0.0, pillow 10.2.0, thop 0.1.1, and pycocotools 2.0.7 libraries to enhance its capabilities in image processing and performance analysis.

The configuration of parameters for this experiment is as follows: The epochs is set to 300 rounds, based on an analysis of experiment outcomes ranging from 100 to 500 rounds. This selection ensures that the model learns enough and prevents overfitting. The patience was chosen at 50 rounds so as to maximise the efficiency of computational resources and avoid overfitting. The batch was determined as 32 in order to consider the specific attributes of the dataset and the constraints of the computational resources. According to Masters & Luschi (2018), this enhances the model’s performance and ensures the training process remains stable. Imgsz was determined to be 640 pixels, taking into account the characteristics of the dataset and the dimensions of the data-enhanced image (640 × 640). Workers was set to 15 in order to minimise data loading time and maximise the utilisation of parallel processing capabilities. The optimizer SGD was selected with an initial learning rate (lr0) and final learning rate (lrf) of 0.01, momentum of 0.937, and weight_decay of 0.0005. These parameter values adhere to the recommendations of Ultralytics, which have been proven to yield outstanding training outcomes and ensure model stability.

Evaluation metrics

In order to provide a comprehensive measure of the performance of the coal-rock recognition model, the following metrics are used to evaluate it: precision, recall, average precision (AP₅₀, AP_50:95), average recall (AR_50:95), optimal localization recall precision (oLRP), the predicted bounding box count (BBox Count), number of parameters (Params), number of floating point operations (FLOPs), and model weight size (Weight).

Assuming that TP represent the number of correctly classified positive cases, FP represent the number of misclassified negative cases, FN represent the number of misclassified positive cases, and TN represent the number of correctly classified negative cases. Precision refers to the probability of truly being a positive class among all the samples predicted to be positive classes, which is defined as shown in Eq. (6):

(6) $$Precision={\displaystyle \frac{TP}{TP+FP}.}$$

Recall refers to the probability of being predicted as a positive class in a sample that is actually a positive class and is defined as shown in Eq. (7):

(7) $$Recall={\displaystyle \frac{TP}{TP+FN}.}$$

Given that this study focuses on only one specific category (coal), we computed the Average Precision (AP) for this category to assess the model’s detection performance, rather than using the mean Average Precision (mAP). Equation (8) provides the precise definition of average precision.

(8) $$AP={\int }_0^{1}P\left(R\right)dR$$where the notation $P\left(R\right)$ represents the precision when the recall ratio is $R$. AP₅₀ refers to the average precision calculated when the Intersection over Union (IoU) ratio is set to 0.5. AP_50:95 represents the average precision calculated over various IoU thresholds ranging from 0.50 to 0.95, with a step size of 0.05.

The Average Recall (AR) is a metric that computes the mean value of the model’s recall at various IoU thresholds. It is mathematically defined by Eq. (9).

(9) $$AR={\displaystyle \frac{1}{N_{\mathrm{I}\mathrm{o}\mathrm{U}}}\sum _{t=Io{U}_{\mathrm{m}\mathrm{i}\mathrm{n}}}^{Io{U}_{\mathrm{m}\mathrm{a}\mathrm{x}}}{R}_t}$$where the IoU threshold ranges from $Io{U}_{\mathrm{m}\mathrm{i}\mathrm{n}}$ to $Io{U}_{\mathrm{m}\mathrm{a}\mathrm{x}}$ with an increment of $\mathrm{\Delta }\mathrm{I}\mathrm{o}\mathrm{U}$. $N_{\mathrm{I}\mathrm{o}\mathrm{U}}$ represents the number of IoU thresholds used to calculate the recall rate. It is calculated using the formula $N_{\mathrm{I}\mathrm{o}\mathrm{U}}=\frac{\mathrm{I}\mathrm{o}{\mathrm{U}}_{\mathrm{m}\mathrm{a}\mathrm{x}}-\mathrm{I}\mathrm{o}{\mathrm{U}}_{\mathrm{m}\mathrm{i}\mathrm{n}}}{\mathrm{\Delta }\mathrm{I}\mathrm{o}\mathrm{U}}+1$. $R_t$ represents the recall at a specific IoU threshold $t$. The notation AR_50:95 represents the mean average precision across various IoU thresholds, ranging from 0.50 to 0.95 with a step size of 0.05.

Localization-Recall-Precision (LRP) (Oksuz et al., 2021) is a comprehensive error metric that considers both the accuracy of localization and classification. The LRP error is defined according to Eq. (10):

(10) $$LRP\left(G,D\right)={\displaystyle \frac{1}{Z}\left(\sum _{i=1}^{N_{TP}}{\displaystyle \frac{1-lq\left(g_i,d_i\right)}{1-\tau }}+N_{FP}+N_{FN}\right)}$$where $D$ represents a collection of detection sets that include information about the category and location, while $G$ represents a collection of real labeling sets. Prior to computation, the detection item $d_i\in D$ is allocated to the appropriate true marker item $g_i\in G$ based on the matching criteria of the IoU. $N_{TP}$, $N_{FP}$, and $N_{FN}$ denote the quantities of true positive examples, false positive examples, and false negative examples, respectively. The function $lq\left(g_i,d_i\right)$ denotes the locational quality of the genuine instances, where $d_i$ is the actual marker that corresponds to the true marker $g_i$ in the genuine instances. $Z=N_{TP}+N_{FP}+N_{FN}$ is the normalization constant. The true example validation threshold, denoted as $\tau$, is set to 0.50.

The Optimal Localization Recall Precision (oLRP) is the minimum possible error in LRP that can be achieved within the detection threshold, or in other words, the confidence score. The confidence score is defined by Eq. (11).

(11) $$\mathrm{o}\mathrm{L}\mathrm{R}\mathrm{P}:={min}_{s\in \mathcal{S}}LRP\left(\mathcal{G},{\mathcal{D}}_s\right)$$where the set ${\mathcal{D}}_{\mathrm{\&}}$ consists of detections that meet or exceed a confidence score threshold $\mathrm{\&}$. Detections with confidence scores below $\mathrm{\&}$ are discarded. The formula involves searching through a set of confidence scores $\mathcal{S}$ to determine the optimal trade-off between precision, recall, and localization error. A lower value of oLRP indicates superior performance of the model.

The predicted bounding box count (BBox Count) refers to the overall number of bounding boxes that the model has identified as containing the target object. Params represent the total number of trainable parameters in the model, used to quantify the spatial intricacy of the model. FLOPs are utilized for determining the time complexity, enabling the measurement of the model’s complexity. The term “model weight size” denotes the size of the weight file generated and saved upon completion of the final training process.