Article

Kernel regression with order preferences

Authors:

Andrew B. GoldbergAuthors Info & Claims

AAAI'07: Proceedings of the 22nd national conference on Artificial intelligence - Volume 1

Pages 681 - 686

Published: 22 July 2007 Publication History

Abstract

We propose a novel kernel regression algorithm which takes into account order preferences on unlabeled data. Such preferences have the form that point x₁ has a larger target value than that of x₂, although the target values for x₁, x₂ are unknown. The order preferences can be viewed as side information or a form of weak labels, and our algorithm can be related to semi-supervised learning. Learning consists of formulating the order preferences as additional regularization in a risk minimization framework. We define a linear program to effectively solve the optimization problem. Experiments on benchmark datasets, sentiment analysis, and housing price problems show that the proposed algorithm outperforms standard regression, even when the order preferences are noisy.

References

[1]

Belkin, M.; Niyogi, P.; and Sindhwani, V. 2004. Manifold regularization: A geometric framework for learning from examples. Technical Report TR-2004-06, University of Chicago.

[2]

Bi, J.; Bennett, K.; Embrechts, M.; Breneman, C.; and Song, M. 2003. Dimensionality reduction via sparse support vector machines. JMLR 3:1229-1243.

[3]

Bradley, P., and Mangasarian, O. 1998. Feature selection via concave minimization and support vector machines. In ICML98.

[4]

Brefeld, U.; Gaertner, T.; Scheffer, T.; and Wrobel, S. 2006. Efficient co-regularized least squares regression. In ICML06.

[5]

Burges, C.; Shaked, T.; Renshaw, E.; Lazier, A.; Deeds, M.; Hamilton, N.; and Hullender, G. 2005. Learning to rank using gradient descent. In ICML05.

[6]

Chu, W., and Ghahramani, Z. 2005. Gaussian processes for ordinal regression. JMLR 6(July): 1019-1041.

[7]

Chu, W., and Keerthi, S. S. 2005. New approaches to support vector ordinal regression. In ICML05, 145-152.

[8]

Collobert, R.; Sinz, F.; Weston, J.; and Bottou, L. 2006. Large scale transductive SVMs. JMLR 7(Aug):1687-1712.

[9]

Dekel, O.; Manning, C.; and Singer, Y. 2004. Loglinear models for label-ranking. In NIPS 16.

[10]

Herbrich, R.; Obermayer, K.; and Graepel, T. 2000. Large margin rank boundaries for ordinal regression. In Smola, A. J.;. Bartlett, P.; Schölkopf, B.; and Schuurmans, D., eds., Advances in Large Margin Classifiers. MIT Press. 115-132.

[11]

Joachims, T. 1999a. Making large-scale svm learning practical. In Schölkopf, B.; Burges, C.; and Smola, A., eds., Advances in Kernel Methods - Support Vector Learning. MIT Press.

[12]

Joachims, T. 1999b. Transductive inference for text classification using support vector machines. In ICML99, 200-209. Morgan Kaufmann, San Francisco, CA.

[13]

Joachims, T. 2002. Optimizing search engines using clickthrough data. In KDD02. ACM Press.

[14]

Kimeldorf, G., and Wahba, G. 1971. Some results on Tchebychean spline functions. Journal of Mathematics Analysis and Applications 33:82-95.

[15]

Mangasarian, O. L.; Shavlik, J. W.; and Wild, E. W. 2004. Knowledge-based kernel approximation. JMLR 5:1127-1141.

[16]

Mangasarian, O. 2000. Generalized support vector machines. In Smola, A. J.; Bartlett, P.; Schölkopf, B.; and Schuurmans, D., eds., Advances in Large Margin Classifiers. MIT Press. 135-146.

[17]

Mizra, M.; Sommers, J.; Barford, P.; and Zhu, X. 2007. A machine learning approach to TCP throughput prediction. In ACM SIGMETRICS.

[18]

Pang, B., and Lee, L. 2005. Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the Association for Computational Linguistics.

[19]

Schölkopf, B.; Herbrich, R.; and Smola, A. J. 2001. A generalized representer theorem. In COLT.

[20]

Sindhwani, V.; Niyogi, P.; and Belkin, M. 2005. A coregularized approach to semi-supervised learning with multiple views. In Proc. of the 22nd ICML Workshop on Learning with Multiple Views.

[21]

Smola, A., and Schölkopf, B. 2004. A tutorial on support vector regression. Statistics and Computing 14:199-222.

[22]

Yu, S.; Yu, K.; Tresp, V.; and Kriegel, H.-P. 2006. Collaborative ordinal regression. In ICML06.

[23]

Zhu, J.; Rosset, S.; Hastie, T.; and Tibshirani, R. 2004. 1-norm support vector machines. In NIPS 16.

Cited By

Silva NColetta LHruschka E(2016)A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised LearningACM Computing Surveys10.1145/293270849:1(1-26)Online publication date: 29-Jun-2016
https://dl.acm.org/doi/10.1145/2932708
Yu HKim SNa SHuang JKoudas NJones GWu XCollins-Thompson KAn A(2010)RankSVRProceedings of the 19th ACM international conference on Information and knowledge management10.1145/1871437.1871550(879-888)Online publication date: 26-Oct-2010
https://dl.acm.org/doi/10.1145/1871437.1871550
Qian MChen BXu HQi HHuang JKoudas NJones GWu XCollins-Thompson KAn A(2010)How about utilizing ordinal information from the distribution of unlabeled dataProceedings of the 19th ACM international conference on Information and knowledge management10.1145/1871437.1871477(289-298)Online publication date: 26-Oct-2010
https://dl.acm.org/doi/10.1145/1871437.1871477
Show More Cited By

Recommendations

Semi-Supervised Kernel Regression
ICDM '06: Proceedings of the Sixth International Conference on Data Mining

Insufficiency of training data is a major obstacle in machine learning and data mining applications. Many different semi-supervised learning algorithms have been proposed to tackle this difficulty by leveraging a large amount of unlabeled data. However, ...
Kernel continuum regression

The continuum regression technique provides an appealing regression framework connecting ordinary least squares, partial least squares and principal component regression in one family. It offers some insight on the underlying regression model for a ...
Probability kernel regression for WiFi localisation
Indoor Positioning and Navigation. Part I: Fingerprinting

Various methods have been developed for indoor localisation using WLAN signals. Algorithms that fingerprint the received signal strength indicators RSSI of WiFi for different locations can achieve tracking accuracies of the order of a few metres. RSSI ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

AAAI'07: Proceedings of the 22nd national conference on Artificial intelligence - Volume 1

July 2007

942 pages

ISBN:9781577353232

Editor:
Anthony Cohn
University of Leeds

Sponsors

Association for the Advancement of Artificial Intelligence

Publisher

AAAI Press

Publication History

Published: 22 July 2007

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Silva NColetta LHruschka E(2016)A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised LearningACM Computing Surveys10.1145/293270849:1(1-26)Online publication date: 29-Jun-2016
https://dl.acm.org/doi/10.1145/2932708
Yu HKim SNa SHuang JKoudas NJones GWu XCollins-Thompson KAn A(2010)RankSVRProceedings of the 19th ACM international conference on Information and knowledge management10.1145/1871437.1871550(879-888)Online publication date: 26-Oct-2010
https://dl.acm.org/doi/10.1145/1871437.1871550
Qian MChen BXu HQi HHuang JKoudas NJones GWu XCollins-Thompson KAn A(2010)How about utilizing ordinal information from the distribution of unlabeled dataProceedings of the 19th ACM international conference on Information and knowledge management10.1145/1871437.1871477(289-298)Online publication date: 26-Oct-2010
https://dl.acm.org/doi/10.1145/1871437.1871477
Sun JShen ZLi HShen Y(2008)Clustering via local regressionProceedings of the 2008th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part II10.5555/3121525.3121556(456-471)Online publication date: 15-Sep-2008
https://dl.acm.org/doi/10.5555/3121525.3121556
Wang FLi TWang GZhang C(2008)Semi-supervised classification using local and global regularizationProceedings of the 23rd national conference on Artificial intelligence - Volume 210.5555/1620163.1620185(726-731)Online publication date: 13-Jul-2008
https://dl.acm.org/doi/10.5555/1620163.1620185
Cheng HTan PLi YLiu BSarawagi S(2008)Semi-supervised learning with data calibration for long-term time series forecastingProceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining10.1145/1401890.1401911(133-141)Online publication date: 24-Aug-2008
https://dl.acm.org/doi/10.1145/1401890.1401911

View Options

View options

Figures

Tables

Media

View Table of Conten