Complexity reduction algorithm for intra mode selection in H.264/AVC video coding

The emerging H.264/AVC video coding standard improves coding performance significantly by adopting many advanced techniques. This is achieved at the expense of great increase of encoder complexity. Specifically, the intra prediction using RDO examines all possible combinations of coding modes, which depend on spatial directional correlation with adjacent blocks. There are 9 modes for a 4×4 luma block, and 4 modes for a 16×16 luma block and an 8×8 chroma block, respectively. Therefore the number of mode combinations for each MB is 592. This paper proposes a complexity reduction algorithm using simple directional masks and neighboring modes. The proposed method reduces the number of mode combinations into 132 at the most. Simulation results show the proposed method reduces the encoding time up to 70% with negligible loss of PSNR and bit-rate increase compared with the H.264/AVC exhaustive search.