Abstract
Multi-target regression (MTR) regards predictive problems with multiple numerical targets. To solve this, machine learning techniques can model solutions treating each target as a separated problem based only on the input features. Nonetheless, modelling inter-target correlation can improve predictive performance. When performing MTR tasks using the statistical dependencies of targets, several approaches put aside the evaluation of each pair-wise correlation between those targets, which may differ for each problem. Besides that, one of the main drawbacks of the current leading MTR method is its high memory cost. In this paper, we propose a novel MTR method called Multi-output Tree Chaining (MOTC) to overcome the mentioned disadvantages. Our method provides an interpretative internal tree-based structure which represents the relationships between targets denominated Chaining Trees (CT). Different from the current techniques, we compute the outputs dependencies, one-by-one, based on the Random Forest importance metric. Furthermore, we proposed a memory friendly approach which reduces the number of required regression models when compared to a leading method, reducing computational cost. We compared the proposed algorithm against three MTR methods (Single-target - ST; Multi-Target Regressor Stacking - MTRS; and Ensemble of Regressor Chains - ERC) on 18 benchmark datasets with two base regression algorithms (Random Forest and Support Vector Regression). The obtained results show that our method is superior to the ST approach regarding predictive performance, whereas, having no significant difference from ERC and MTRS. Moreover, the interpretative tree-based structures built by MOTC pose as great insight on the relationships among targets. Lastly, the proposed solution used significantly less memory than ERC being very similar in predictive performance.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
The source codes for MOTC and the other evaluated MTR methods are disponible in http://www.uel.br/grupo-pesquisa/remid/?page_id=145.
References
Aho, T., Zenko, B., Dzeroski, S., Elomaa, T. (2012). Multi-target regression with rule ensembles. Journal of Machine Learning Research, 13, 2367–2407.
Borchani, H., Varando, G., Bielza, C., Larrañaga, P. (2015). A survey on multi-output regression. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(5), 216–233.
Breiman, L. (2001). Random forests. Machine learning, 45.1, 5–32. https://doi.org/10.1017/CBO9781107415324.004.
Brugger, D., Rosenstiel, W., Bogdan, M. (2011). Online SVR training by solving the primal optimization problem. Journal of Signal Processing Systems, 65(3), 391–402.
Chen, H., & Ser, W. (2011). Sound source DOA estimation and localization in noisy reverberant environments using least-squares support vector machines. Journal of Signal Processing Systems, 63(3), 287–300.
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1023/A:1022627411411.
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1–30.
Di Persio, L., & Honchar, O. (2016). Artificial neural networks architectures for stock price prediction: comparisons and applications. International Journal of Circuits, Systems and Signal Processing, 10, 403–413.
Drucker, H., Burges, C.J.C., Kaufman, L., Smola, A.J., Vapnik, V. (1997). Support vector regression machines. In Mozer, M.C., Jordan, M.I., Petsche, T. (Eds.) Advances in neural information processing systems (Vol. 9, pp. 155–161). MIT Press. http://papers.nips.cc/paper/1238-support-vector-regression-machines.pdf.
Evgeniou, T., Figueiras-Vidal, A.R., Theodoridis, S. (2008). Emerging machine learning techniques in signal processing.
Gama, J., & Brazdil, P. (2000). Cascade generalization. Machine Learning, 41(3), 315–343. https://doi.org/10.1023/A:1007652114878.
Genuer, R., Poggi, J.M., Tuleau-Malot, C. (2010). Variable selection using random forests. Pattern Recognition Letters, 31(14), 2225–2236. https://doi.org/10.1016/j.patrec.2010.03.014. http://www.sciencedirect.com/science/article/pii/S0167865510000954.
Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301), 13–30. https://doi.org/10.1080/01621459.1963.10500830. http://amstat.tandfonline.com/doi/abs/10.1080/01621459.1963.10500830.
Katagiri, S., Nakamura, A., Adali, T., Tao, J., Larsen, J., Tan, T. (2014). Guest editorial: Machine learning for signal processing. Journal of Signal Processing Systems, 74(3), 281–283. https://doi.org/10.1007/s11265-014-0871-6.
Kocev, D., Džeroski, S., White, M.D., Newell, G.R., Griffioen, P. (2009). Using single- and multi-target regression trees and ensembles to model a compound index of vegetation condition. Ecological Modelling, 220(8), 1159–1168.
Kocev, D., Vens, C., Struyf, J., Džeroski, S. (2007). Ensembles of multi-objective decision trees. In European conference on machine learning (pp. 624–631). Springer.
Kocev, D., Vens, C., Struyf, J., Džeroski, S. (2013). Tree ensembles for predicting structured outputs. Pattern Recognition, 46(3), 817–833.
Li, X., & Zheng, J. (2016). Active learning for regression with correlation matching and labeling error suppression. IEEE Signal Processing Letters, 23(8), 1081–1085.
Lichman, M. (2013). UCI machine learning repository. http://archive.ics.uci.edu/ml.
Mastelini, S.M., Santana, E.J., Cerri, R., Barbon, S. Jr. (2017). DSTARS: a multi-target deep structure for tracking asynchronous regressor stack. In Brazilian conference on intelligent systems. BRACIS 2017.
Melki, G., Cano, A., Kecman, V., Ventura, S. (2017). Multi-target support vector regression via correlation regressor chains. Information Sciences, 415, 53–69.
Moyano, J.M., Gibaja, E.L., Ventura, S. (2017). An evolutionary algorithm for optimizing the target ordering in ensemble of regressor chains. In 2017 IEEE congress on evolutionary computation (CEC) (pp. 2015–2021). IEEE.
Santana, E.J., Mastelini, S.M., Barbon, S. Jr. (2017). Deep regressor stacking for air ticket prices prediction. In Brazilian symposium of information systems (pp. 216–233). SBSI 2017.
Sidike, P., Krieger, E., Alom, M.Z., Asari, V.K., Taha, T. (2017). A fast single-image super-resolution via directional edge-guided regularized extreme learning regression. In Signal, image and video processing (pp. 1–8).
Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W., Vlahavas, I. (2016). Multi-target regression via input space expansion: treating targets as inputs. Machine Learning, 104(1), 55–98.
Tsoumakas, G., Spyromitros-Xioufis, E., Vrekou, A., Vlahavas, I. (2014). Multi-target regression via random linear target combinations. In Joint european conference on machine learning and knowledge discovery in databases (pp. 225–240). Springer.
Wang, Q., Wu, Y., Shen, Y., Liu, Y., Lei, Y. (2015). Supervised sparse manifold regression for head pose estimation in 3d space. Signal Processing, 112, 34–42.
Watanabe, S., Nakamura, A., Juang, B.H.F. (2014). Structural bayesian linear regression for hidden Markov models. Journal of Signal Processing Systems, 74(3), 341–358.
Zhang, W., Liu, X., Ding, Y., Shi, D. (2012). Multi-output LS-SVR machine in extended feature space. In CIMSA 2012 - 2012 IEEE Int. Conf. Comput. Int.ll. Meas. Syst. Appl. Proc. (pp. 130–144). https://doi.org/10.1109/CIMSA.2012.6269600.
Acknowledgements
The authors would like to thank CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico) and FAPESP (Fundação de Amparo à Pesquisa do Estado de São Paulo) for financial support.
Author information
Authors and Affiliations
Corresponding author
Appendices
Appendix A: Datasets Used in the Experiments
Appendix B: Obtained Condensed Chaining Tree Graphs and Target Labels
Rights and permissions
About this article
Cite this article
Mastelini, S.M., da Costa, V.G.T., Santana, E.J. et al. Multi-Output Tree Chaining: An Interpretative Modelling and Lightweight Multi-Target Approach. J Sign Process Syst 91, 191–215 (2019). https://doi.org/10.1007/s11265-018-1376-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11265-018-1376-5