Rate distortion optimization (RDO) is an important approach for ensuring the coding efficiency of the encoder by selecting the optimal combination of multiple candidate coding parameters in video coding. Discrete sine transform (DST) has high accuracy but causes high complexity in RDO, whereas Hadamard transform is simple but unsuitable for estimating the transform coefficients of DST in RDO. In this study, we initially proposed a new simplified DST transformation matrix to estimate the rate distortion (RD) cost accurately for intra prediction and then analyzed the decorrelated performance and complexity. Moreover, we designed a high-throughput hardware architecture based on the FPGA platform. Experimental results show that in comparison with the high efficiency video coding reference software HM16.7, the proposed simplified DST matrix-based RD cost estimation model improves the RD performance. Moreover, the proposed architecture achieves higher throughput with pipeline structure compared with the existing hardware structures. The implementation in Altera’s Arria 10 FPGA can operate at 275 MHz and supports real-time processing of 4096 × 2160 ultra-high-definition at a minimum of 75 fps.