Abstract
Click-Through Rate (CTR) prediction models have achieved huge success mainly due to the ability to model arbitrary-order feature interactions. Recently, Self-Attention Network (SAN) has achieved significant success in CTR prediction. However, most of the existing SAN-based methods directly perform feature interaction operations on raw features. We argue that such operations, which ignore the intra-field information and inter-field affinity, are not designed to model richer feature interactions. In this paper, we propose an Intra and Inter-field Self-Attentive Network (IISAN) model for CTR prediction. Specifically, we first design an effective embedding block named Gated Fusion Layer (GFL) to refine raw features. Then, we utilize self-attention to model the feature interactions for each item to form meaningful high-order features via a multi-head attention mechanism. Next, we use the attention mechanism to aggregate all interactive embeddings. Finally, we assign DNNs in the prediction layer to generate the final output. Extensive experiments on three real public datasets show that IISAN achieves better performance than existing state-of-the-art approaches for CTR prediction.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The field embedding \(\mathbf {e}_i\) is also known as the feature embedding. If the field is multivalent, the sum of feature embedding is used as the field embedding.
- 2.
- 3.
- 4.
- 5.
References
Blondel, M., Fujino, A., Ueda, N., Ishihata, M.: Higher-order factorization machines. In: NIPS, pp. 3351–3359 (2016)
Cheng, C., Xia, F., Zhang, T., King, I., Lyu, M.R.: Gradient boosting factorization machines. In: RecSys, pp. 265–272. ACM (2014)
Cheng, H., et al.: Wide & deep learning for recommender systems. In: DLRS@RecSys, pp. 7–10. ACM (2016)
Cheng, W., Shen, Y., Huang, L.: Adaptive factorization network: Learning adaptive-order feature interactions. In: AAAI, pp. 3609–3616. AAAI Press (2020)
Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N.: Convolutional sequence to sequence learning. In: ICML, Proceedings of Machine Learning Research, vol. 70, pp. 1243–1252. PMLR (2017)
Guo, H., Tang, R., Ye, Y., Li, Z., He, X.: Deepfm: a factorization-machine based neural network for CTR prediction. In: IJCAI, pp. 1725–1731. ijcai.org (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778. IEEE Computer Society (2016)
He, X., Chua, T.: Neural factorization machines for sparse predictive analytics. In: SIGIR, pp. 355–364. ACM (2017)
Huang, T., Zhang, Z., Zhang, J.: Fibinet: combining feature importance and bilinear feature interaction for click-through rate prediction. In: RecSys, pp. 169–177. ACM (2019)
Juan, Y., Zhuang, Y., Chin, W., Lin, C.: Field-aware factorization machines for CTR prediction. In: RecSys, pp. 43–50. ACM (2016)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (Poster) (2015)
Li, Z., Cheng, W., Chen, Y., Chen, H., Wang, W.: Interpretable click-through rate prediction through hierarchical attention. In: WSDM, pp. 313–321. ACM (2020)
Lian, J., Zhou, X., Zhang, F., Chen, Z., Xie, X., Sun, G.: xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In: KDD, pp. 1754–1763. ACM (2018)
Lu, J., Yang, J., Batra, D., Parikh, D.: Hierarchical question-image co-attention for visual question answering. In: NIPS, pp. 289–297 (2016)
Luo, Y., Zhou, H., Tu, W., Chen, Y., Dai, W., Yang, Q.: Network on network for tabular data classification in real-world applications. In: SIGIR, pp. 2317–2326. ACM (2020)
Rendle, S.: Factorization machines with LIBFM. ACM Trans. Intell. Syst. Technol. 3(3), 1–22 (2012)
Shan, Y., Hoens, T.R., Jiao, J., Wang, H., Yu, D., Mao, J.C.: Deep crossing: Web-scale modeling without manually crafted combinatorial features. In: KDD, pp. 255–262. ACM (2016)
Song, W., et al.: Autoint: automatic feature interaction learning via self-attentive neural networks. In: CIKM, pp. 1161–1170. ACM (2019)
Tan, Z., Wang, M., Xie, J., Chen, Y., Shi, X.: Deep semantic role labeling with self-attention. In: AAAI, pp. 4929–4936. AAAI Press (2018)
Vaswani, A., et al.: Attention is all you need. In: NIPS, pp. 5998–6008 (2017)
Wang, R., Fu, B., Fu, G., Wang, M.: Deep & cross network for ad click predictions. In: ADKDD@KDD, pp. 1–7. ACM (2017)
Xiao, J., Ye, H., He, X., Zhang, H., Wu, F., Chua, T.: Attentional factorization machines: Learning the weight of feature interactions via attention networks. In: IJCAI, pp. 3119–3125. ijcai.org (2017)
Xiao, T., Xu, Y., Yang, K., Zhang, J., Peng, Y., Zhang, Z.: The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In: CVPR, pp. 842–850. IEEE Computer Society (2015)
Xu, C., et al.: Graph contextualized self-attention network for session-based recommendation. In: IJCAI, pp. 3940–3946. ijcai.org (2019)
Zhao, Y., Liang, S., Ren, Z., Ma, J., Yilmaz, E., de Rijke, M.: Explainable user clustering in short text streams. In: SIGIR, pp. 155–164. ACM (2016)
Zhu, J., Liu, J., Yang, S., Zhang, Q., He, X.: Fuxictr: an open benchmark for click-through rate prediction (2020). CoRR abs/2009.05794
Zhu, Y., Xu, Y., Yu, F., Liu, Q., Wu, S., Wang, L.: Disentangled self-attentive neural networks for click-through rate prediction (2021). CoRR abs/2101.03654
Acknowledgements
This research was partially supported by NSFC (No.61876117, 61876217, 61872258, 61728205), Open Program of Key Lab of IIP of CAS (No. IIP2019-1) and the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, S., Xian, X., Hao, Y., Sheng, V.S., Cui, Z., Zhao, P. (2021). Exploiting Intra and Inter-field Feature Interaction with Self-Attentive Network for CTR Prediction. In: Zhang, W., Zou, L., Maamar, Z., Chen, L. (eds) Web Information Systems Engineering – WISE 2021. WISE 2021. Lecture Notes in Computer Science(), vol 13081. Springer, Cham. https://doi.org/10.1007/978-3-030-91560-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-91560-5_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-91559-9
Online ISBN: 978-3-030-91560-5
eBook Packages: Computer ScienceComputer Science (R0)