research-article

Active Learning Query Strategies for Classification, Regression, and Clustering: A Survey

Authors:

Atul GuptaAuthors Info & Claims

Journal of Computer Science and Technology, Volume 35, Issue 4

Pages 913 - 945

https://doi.org/10.1007/s11390-020-9487-4

Published: 01 July 2020 Publication History

Abstract

Generally, data is available abundantly in unlabeled form, and its annotation requires some cost. The labeling, as well as learning cost, can be minimized by learning with the minimum labeled data instances. Active learning (AL), learns from a few labeled data instances with the additional facility of querying the labels of instances from an expert annotator or oracle. The active learner uses an instance selection strategy for selecting those critical query instances, which reduce the generalization error as fast as possible. This process results in a refined training dataset, which helps in minimizing the overall cost. The key to the success of AL is query strategies that select the candidate query instances and help the learner in learning a valid hypothesis. This survey reviews AL query strategies for classification, regression, and clustering under the pool-based AL scenario. The query strategies under classification are further divided into: informative-based, representative-based, informative- and representative-based, and others. Also, more advanced query strategies based on reinforcement learning and deep learning, along with query strategies under the realistic environment setting, are presented. After a rigorous mathematical analysis of AL strategies, this work presents a comparative analysis of these strategies. Finally, implementation guide, applications, and challenges of AL are discussed.

References

[1]

Mitchell T. Machine Learning (1st edition). MacGraw-Hill Education, 1997.

[2]

Hu R. Active learning for text classification [Ph.D. Thesis]. Dublin Institute of Technology, 2011.

[3]

Tuia D, Ratle F, Pacifici F, Kanevski MF, and Emery WJ Active learning methods for remote sensing image classification IEEE Trans. Geoscience and Remote Sensing 2009 47 7-2 2218-2232

[4]

Guo J, Chen H, Sun Z, and Lin Y A novel method for protein secondary structure prediction using dual-layer SVM and profiles PROTEINS: Structure, Function, and Bioinformatics 2004 54 4 738-743

[5]

Zhu X. Semi-supervised learning literature survey. Technical Report, University of Wisconsin-Madison, 2008. http://pages.cs.wisc.edu/∼jerryzhu/pub/ssl survey.pdf, Nov. 2019.

[6]

Settles B. Active learning literature survey. Technical Report, University of Wisconsin-Madison, 2009. http://apophenia.wdfiles.com/local–files/start/settles active.learning.pdf, Nov. 2019.

[7]

Cohn D, Atlas L, and Ladner R Improving generalization with active learning Machine Learning 1994 15 2 201-221

[8]

Wang M, Hua X S. Active learning in multimedia annotation and retrieval: A survey. ACM Trans. Intelligent Systems and Technology, 2011, 2(2): Article No. 10.

[9]

Lewis D D, Catlett J. Heterogeneous uncertainty sampling for supervised learning. In Proc. the 11th Int. Conference on Machine Learning, July 1994, pp.148-156.

[10]

Zhu X, Zhang P, Lin X, Shi Y. Active learning from data streams. In Proc. the 7th IEEE Int. Conference on Data Mining, October 2007, pp.757-762.

[11]

Zhu X, Zhang P, Lin X, and Shi Y Active learning from stream data using optimal weight classifier ensemble. IEEE Trans Systems, Man, and Cybernetics, Part B 2010 40 6 1607-1621

[12]

Zliobaite I and Bifet A Pfahringer, Holmes G. Active learning with drifting streaming data. IEEE Trans Neural Networks and Learning Systems 2014 25 1 27-39

[13]

Wang P, Zhang P, Guo L. Mining multi-label data streams using ensemble-based active learning. In Proc. the 12th SIAM International Conference on Data Mining, April 2012, pp.1131-1140.

[14]

Angluin DQueries and concept learningMachine Learning198824319-3423363446

[15]

Wang L, Hu X, Yuan B, and Lu J Active learning via query synthesis and nearest neighbour search Neurocomputing 2015 147 426-434

[16]

Sun L L, Wang X Z. A survey on active learning strategy. In Proc. the Int. Conference on Machine Learning and Cybernetics, July 2010, pp.161-166.

[17]

Fu Y, Zhu X, and Li B A survey on instance selection for active learning Knowledge and Information Systems 2012 35 2 249-283

[18]

Aggarwal C, Kong X, Gu Q, Han J, Yu P. Active learning: A survey. In Data Classification: Algorithms and Applications, Aggarwal C C (ed.), CRC Press, 2014, pp.571-605.

[19]

Lewis D D, Gale W A. A sequential algorithm for training text classifiers. In Proc. the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, July 1994, pp.3-12.

[20]

Atlas L, Cohn D A, Ladner R E. Training connectionist networks with queries and selective sampling. In Proc. the 3rd Annual Conference on Neural Information Processing Systems, November 1989, pp.566-573.

[21]

Culotta A, Mccallum A. Reducing labeling effort for structured prediction tasks. In Proc. the 20th National Conference on Artificial Intelligence, July 2005, pp.746-751.

[22]

Shannon CEA mathematical theory of communicationBell System Technical Journal1948273379-423262861154.94303

[23]

Scheffer T, Decomain C, Wrobel S. Active hidden Markov models for information extraction. In Proc. the 4th International Conference on Advances in Intelligent Data Analysis, September 2001, pp.309-318.

[24]

Seung H, Opper M, Sompolinsky H. Query by committee. In Proc. the 5th Annual Conference on Computational Learning Theory, July 1992, pp.287-294.

[25]

Abe N, Mamitsuka H. Query learning strategies using boosting and bagging. In Proc. the 15th International Conference on Machine Learning, July 1998, pp.1-9.

[26]

Melville P, Mooney R J. Diverse ensembles for active learning. In Proc. the 21st Int. Conference on Machine learning, July 2004, Article No. 56.

[27]

Muslea I, Minton S, Knoblock C A. Selective sampling with redundant views. In Proc. the 17th National Conference on Artificial Intelligence, July 2000, pp.621-626.

[28]

Cortes C and Vapnik VSupport-vector networksMachine Learning1995203273-2970831.68098

[29]

Kremer J, Pedersen KS, and Igel C Active learning with support vector machines Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 2014 4 4 313-326

[30]

Tong S and Koller DSupport vector machine active learning with applications to text classificationJournal of Machine Learning Research2002245-661009.68131

[31]

Vapnik V An overview of statistical learning theory IEEE Trans. Neural Networks 1999 10 5 988-999

[32]

Schohn G, Cohn D. Less is more: Active learning with support vector machines. In Proc. the 17th Int. Conference on Machine Learning, June 2000, pp.839-846.

[33]

Campbell C, Cristianini N, Smola A. Query learning with large margin classifiers. In Proc. the 17th Int. Conference on Machine Learning, June 2000, pp.111-118.

[34]

Indyk P, Motwani R. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proc. the 30th Annual ACM Symposium on Theory of Computing, May 1998, pp.604-613.

[35]

Gionis A, Indyk P, Motwani R. Similarity search in high dimension via hashing. In Proc. the 25th Int. Conference on Very Large Data Bases, September 1999, pp.518-529.

[36]

Jain P, Vijayanarasimhan S, Grauman K. Hashing hyperplane queries to near points with applications to large-scale active learning. In Proc. the 24th Annual Conference on Neural Information Processing Systems, December 2010, pp.928-936.

[37]

Vijayanarasimhan S, Jain P, and Grauman K Hashing hyperplane queries to near points with applications to large-scale active learning. IEEE Trans Pattern Analysis and Machine Intelligence 2014 36 2 276-288

[38]

Basri R, Hassner T, and Zelnik-Manor L Approximate nearest subspace search. IEEE Trans Pattern Analysis and Machine Intelligence 2011 33 2 266-278

[39]

Basri R, Hassner T, Zelnik-Manor L. Approximate nearest subspace search with applications to pattern recognition. In Proc. the 2017 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2007.

[40]

Wang J, Shen H, Song J, Ji J. Hashing for similarity search: A survey. arXiv:1408.2927, 2014. http://arxiv.org/abs/1408.2927, Nov. 2019.

[41]

Settles B, Craven M. An analysis of active learning strategies for sequence labeling tasks. In Proc. the 2008 Conference on Empirical Methods in Natural Language Processing, October 2008, pp.1070-1079.

[42]

Wu Y, Kozintsev I, Bouguet J Y, Dulong C. Sampling strategies for active learning in personal photo retrieval. In Proc. the IEEE International Conference on Multimedia and Expo, July 2006, pp.529-532.

[43]

Ienco D, Bifet A, Zliobaite I et al. Clustering based active learning for evolving data streams. In Proc. the 16th Int. Conference on Discovery Science, October 2013, pp.79-93.

[44]

Brinker K. Incorporating diversity in active learning with support vector machines. In Proc. the 20th Int. Conference on Machine Learning, August 2003, pp.59-66.

[45]

Hoi S C H, Jin R, Lyu M R. Large-scale text categorization by batch mode active learning. In Proc. the 15th Int. Conference on World Wide Web, May 2006, pp.633-642.

[46]

Hoi S C H, Jin R, Zhu J, Lyu M R. Batch mode active learning and its application to medical image classification. In Proc. the 23rd Int. Conference on Machine Learning, June 2006, pp.417-424.

[47]

Xu Z, Akella R, Zhang Y. Incorporating diversity and density in active learning for relevance feedback. In Proc. the 29th Eur. Conf. Inf. Retrieval Research, April 2007, pp.246-257.

[48]

Sen P, Namata G, Bilgic M, Getoor L, and Galligher B Eliassi- Rad T. Collective classification in network data AI Magazine 2008 29 3 93-106

[49]

Neville J, Jensen D. Iterative classification in relational data. In Proc. the AAAI 2000 Workshop on Learning Statistical Models from Relational Data, July 2000, pp.42-49.

[50]

Richardson M and Domingos P Markov logic networks Machine Learning 2006 62 1/2 107-136

[51]

Bilgic M, Mihalkova L, Getoor L. Active learning for networked data. In Proc. the 27th Int. Conference on Machine Learning, June 2010, pp.79-86.

[52]

Wang Z, Ye J. Querying discriminative and representative samples for batch mode active learning. In Proc. the 19th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, August 2013, pp.158-166.

[53]

Nguyen H T, Smeulders A. Active learning using preclustering. In Proc. the 21st Int. Conference on Machine Learning, July 2004, Article No. 19.

[54]

Huang SJ, Jin R, and Zhou ZH Active learning by querying informative and representative examples IEEE Trans. Pattern Analysis and Machine Intelligence 2014 36 10 1936-1949

[55]

Hoi S C, Jin R, Zhu J, Lyu M R. Semi-supervised SVM batch mode active learning for image retrieval. In Proc. the 2008 IEEE Conference on Computer Vision and Pattern Recognition, June 2008, Article No. 10.

[56]

Belkin M, Niyogi P, and Sindhwani VManifold regularization: A geometric framework for learning from labeled and unlabeled examplesJournal of Machine Learning Research200672399-243422744441222.68144

[57]

Du B, Wang Z, Zhang L, Zhang L, Liu W, Shen J, and Tao D Exploring representativeness and informativeness for active learning IEEE Trans. Cybernetics 2017 47 1 14-26

[58]

Gretton A, Borgwardt KM, RaschM J, Schölkopf B, and Smola AA kernel two-sample testJournal of Machine Learning Research201213723-77329137161283.62095

[59]

Luo W, Schwing A, Urtasun R. Latent structured active learning. In Proc. the 27th Annual Conference on Neural Information Processing Systems, December 2013, pp.728-736.

[60]

Anderson N, Hall P, and Titterington DTwo-sample test statistics for measuring discrepancies between two multivariate probability density functions using kernel-based density estimatesJournal of Multivariate Analysis199450141-5412926070798.62055

[61]

Wang Z, Fang X, Tao X, et al. Multi-class active learning by integrating uncertainty and diversity IEEE Access 2018 6 22794-22803

[62]

Krempl G, Kottke D, Spiliopoulou M. Probabilistic active learning: Towards combining versatility, optimality and efficiency. In Proc. the 17th Int. Conference on Discovery Science, October 2014, pp.168-179.

[63]

Chapelle O, Sch¨olkopf B, Zien A. Semi-Supervised Learning. The MIT Press, 2010.

[64]

Krempl G, Kottke D, and Lemaire VOptimised probabilistic active learning (OPAL) — For fast, non-myopic, costsensitive active classificationMachine Learning20151002-3449-47633839781341.68160

[65]

Settles B, Craven M, Ray S. Multiple-instance active learning. In Proc. the 21st Annual Conference on Neural Information Processing Systems, December 2007, pp.1289-1296.

[66]

Roy N, McCallum A. Toward optimal active learning through sampling estimation of error reduction. In Proc. the 18th Int. Conference on Machine Learning, June 2001, pp441-448.

[67]

Moskovitch R, Nissim N, Stopel D et al. Improving the detection of unknown computer worms activity using active learning. In Proc. the 30th German Conference on AI, September 2007, pp.489-493.

[68]

Fang M, Li Y, Cohn T. Learning how to active learn: A deep reinforcement learning approach. In Proc. the Conference on Empirical Methods in Natural Language Processing, September 2017, pp.595-605.

[69]

Liu M, Buntine W, Haffari G. Learning how to actively learn: A deep imitation learning approach. In Proc. the 56th Annual Meeting of the Association for Computational Linguistics, July 2018, pp.1874-1883.

[70]

Pang K, Dong M, Wu Y et al. Meta-learning transferable active learning policies by deep reinforcement learning. arXiv:1806.04798, 2008. https://arxiv.org/abs/1806.04798, Nov. 2019.

[71]

Bachman P, Sordoni A, Trischler A. Learning algorithms for active learning. In Proc. the 34th Int. Conference on Machine Learning, August 2017, pp.301-310.

[72]

Cohn D, Ghahramani Z, Jordan M. Active learning with statistical models. In Proc. the 1994 Annual Conference on Neural Information Processing Systems, December 1994, pp.705-712.

[73]

Geman S, Bienenstock E, and Doursat R Neural networks and the bias/variance dilemma Neural Computation 1992 4 1 1-58

[74]

Schervish M. Theory of Statistics (1st edition). Springer, 1995.

[75]

Long B, Chapelle O, Zhang Y, Chang Y, Zheng Y, and Tseng B Active learning for ranking through expected loss optimization IEEE Trans. Knowledge and Data Engineering 2015 27 5 1180-1191

[76]

Freund Y, Seung H, Shamir E, and Tishby NSelective sampling using the query by committee algorithmMachine Learning19972823133-1680881.68093

[77]

Krogh A, Vedelsby J. Neural network ensembles, cross validation, and active learning. In Proc. the 8th Annual Conference on Neural Information Processing Systems, November 1995, pp.231-238.

[78]

Burbidge R, Rowland J J, King R D. Active learning for regression based on query by committee. In Proc. the 8th Int. Conference on Intelligent Data Engineering and Automated Learning, December 2007, pp.209-218.

[79]

Cai W, Zhang Y, Zhou J. Maximizing expected model change for active learning in regression. In Proc. the 13th International Conference on Data Mining, December 2013, pp.51-60.

[80]

Bottou L. Large-scale machine learning with stochastic gradient descent. In Proc. the 19th Int. Conference on Computational Statistics, August 2010, pp.177-186.

[81]

Cai W, Zhang Y, Zhou S Y et al. Active learning for support vector machines with maximum model change. In Proc. the 2014 European Conference on Machine Learning and Knowledge Discovery in Databases, September 2014, pp.211-226.

[82]

Dasgupta S. The two faces of active learning. In Proc. the 20th Int. Conference on Algorithmic Learning Theory, October 2009, Article No. 1.

[83]

Dasgupta S, Hsu D. Hierarchical sampling for active learning. In Proc. the 25th Int. Conference on Machine Learning, June 2008, pp.208-215.

[84]

Urner R, Ben-David S. Probabilistic lipschitzness: A niceness assumption for deterministic labels. In Proc. the 27th NIPS Learning Faster from Easy Data Workshop, December 2013.

[85]

Steinwart I and Scovel CFast rates for support vector machines using Gaussian kernelsThe Annals of Statistics2007352575-60723368601127.68091

[86]

Urner R, Shalev-Shwartz S, Ben-David S. Access to unlabeled data can speed up prediction time. In Proc. the 28th Int. Conference on Machine Learning, June 2011, pp.641-648.

[87]

Verma N, Kpotufe S, Dasgupta S. Which spatial partition trees are adaptive to intrinsic dimension? In Proc. the 25th Conference Uncertainty Artif. Intell., June 2009, pp.565-574.

[88]

Urner R, Wulff S, Ben-David S. PlAL: Cluster-based active learning. In Proc. the 26th Conference on Learning Theory, June 2013, pp.376-397.

[89]

Wang M, Min F, Zhang ZH, and Wu YX Active learning through density clustering Expert Systems with Applications 2017 85 305-317

[90]

Rodriguez A and Laio A Clustering by fast search and find of density peaks Science 2014 344 6191 1492-1496

[91]

Yan Y, Rosales R, Fung G, Dy J. Active learning from crowds. In Proc. the 28th Int. Conference on Machine Learning, June 2011, pp.1161-1168.

[92]

Fang M, Zhu X, Li B, Ding W, Wu X. Self-taught active learning from crowds. In Proc. the 12th Int. Conference on Data Mining, December 2012, pp.858-863.

[93]

Shu Z, Sheng VS, and Li J Learning from crowds with active learning and self-healing Neural Computing and Applications 2018 30 9 2883-2894

[94]

Lampert CH, Nickisch H, and Harmeling S Attribute-based classification for zero-shot visual object categorization IEEE Trans. Pattern Analysis and Machine Intelligence 2014 36 3 453-465

[95]

Ertekin S, Huang J, Giles C L. Active learning for class imbalance problem. In Proc. the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 2007, pp.823-824.

[96]

Attenberg J, Ertekin S¸. Class imbalance and active learning. In Imbalanced Learning: Foundations, Algorithms, and Applications, He H B, Ma Y Q (eds.), John Wiley & Sons, Inc., 2013, pp.101-149.

[97]

Tomanek K, Morik K. Inspecting sample reusability for active learning. In Proc. the Workshop on Active Learning and Experimental Design, May 2010, pp.169-181.

[98]

Hu R, Namee BM, and Delany SJ Active learning for text classification with reusability Expert Systems with Applications 2016 45 C 438-449

[99]

Settles B, Craven M, Friedland L. Active learning with real annotation costs. In Proc. the 2008 NIPS Workshop on Cost-Sensitive Learning, December 2008.

[100]

Tomanek K, Hahn U. A comparison of models for costsensitive active learning. In Proc. the 23rd Int. Conference on Computational Linguistics, August 2010, pp.1247-1255.

[101]

Liu A, Jun G, Ghosh J. Active learning of hyperspectral data with spatially dependent label acquisition costs. In Proc. the 2009 IEEE International Geoscience and Remote Sensing Symposium, July 2009, pp.256-259.

[102]

Persello C, Boularias A, Dalponte M, et al. Cost-sensitive active learning with lookahead: Optimizing field surveys for remote sensing data classification IEEE Trans. Geoscience and Remote Sensing 2014 52 10 6652-6664

[103]

Margineantu D. Active cost-sensitive learning. In Proc. the 19th International Joint Conference on Artificial Intelligence, July 2005, pp.1622-1623.

[104]

Krishnamurthy A, Agarwal A, Huang T et al. Active learning for cost-sensitive classification. arXiv:1703.01014, 2017. https://arxiv.org/abs/1703.01014, May 2019.

[105]

Zhang D, Wang F, Shi Z, et al.Interactive localized content based image retrieval with multiple-instance active learningPattern Recognition2010432478-4841182.68079

[106]

Wang R, Wang X, Kwong S, et al. Incorporating diversity and informativeness in multiple-instance active learning IEEE Trans. Fuzzy Systems 2017 25 6 1460-1475

[107]

Wu J, Sheng V S, Zhang J, Zhao P, Cui Z. Multi-label active learning for image classification. In Proc. the 21st IEEE Int. Conference on Image Processing, October 2014, pp.5227-5231.

[108]

Yang B, Sun J T, Wang T, Chen Z. Effective multi-label active learning for text classification. In Proc. the 15th ACM SIGKDD Int. Conference on Knowledge Discovery and Data Mining, June 2009, pp.917-926.

[109]

Pupo O, Morell C, and Ventura S Effective active learning strategy for multi-label learning Neurocomputing 2017 273 494-508

[110]

Cherman EA, Papanikolaou Y, Tsoumakas G, et al. Multilabel active learning: Key issues and a novel query strategy Evolving Systems 2017 10 1 63-78

[111]

Rani M, Dhok S, and Deshmukh R A systematic review of compressive sensing: Concepts, implementations and applications IEEE Access 2018 6 4875-4894

[112]

Som S. Learning label structure for compressed sensing based multilabel classification. In Proc. the 2016 SAI Computing Conference, July 2016, pp.54-60.

[113]

Wu J, Ye C, Sheng V, et al. Active learning with label correlation exploration for multi-label image classification IET Computer Vision 2017 11 7 577-584

[114]

Pupo O, Ventural S. Evolutionary strategy to perform batch-mode active learning on multi-label data. ACM Trans. Intelligent Systems and Technology, 2018, 9(4): Article No. 46.

[115]

Reichart R, Tomanek K, Hahn U, Rappoport A. Multitask active learning for linguistic annotations. In Proc. the 46th Association for Computational Linguistics, June 2008, pp.861-869.

[116]

Zhang Y. Multi-task active learning with output constraints. In Proc. the 24th AAAI Conference on Artificial Intelligence, July 2010, pp.667-672.

[117]

Harpale A. Multi-task active learning [Ph.D. Thesis]. School of Computer Science, Carnegie Mellon University, 2012.

[118]

Gavves E, Mensink T, Tommasi T et al. Active transfer learning with zero-shot priors: Reusing past datasets for future tasks. In Proc. the 2015 IEEE International Conference on Computer Vision, December 2015, pp.2731-2739.

[119]

Wang X, Huang T, Schneider J. Active transfer learning under model shift. In Proc. the 31st Int. Conference on Machine Learning, June 2014, pp.1305-1313.

[120]

Guo Y, Schuurmans D. Discriminative batch mode active learning. In Proc. the 21st Annual Conference on Neural Information Processing Systems, December 2007, pp.593-600.

[121]

Chakraborty S, Balasubramanian V, and Panchanathan SAdaptive batch mode active learning. IEEE TransNeural Networks and Learning Systems20152681747-17603454970

[122]

Shen P, Li C, and Zhang Z Distributed active learning IEEE Access 2016 4 2572-2579

[123]

Hinton GE, Osindero S, and The YA fast learning algorithm for deep belief netsNeural Computing20061871527-155422244851106.68094

[124]

Wang K, Zhang D, Li Y, et al. Cost-effective active learning for deep image classification IEEE Trans. Circuits and Systems for Video Technology 2017 27 12 2591-2600

[125]

Rahhal MMA, Bazi Y, Alhichri H, et al. Deep learning approach for active classification of electrocardiogram signals Information Sciences 2016 345 C 340-354

[126]

Zhou S, Chen Q, and Wang X Active deep learning method for semi-supervised sentiment classification Neurocomputing 2013 120 536-546

[127]

Valiant LGA theory of the learnableCommunications of the ACM198427111134-11420587.68077

[128]

Hanneke S. A bound on the label complexity of agnostic active learning. In Proc. the 24th Int. Conference on Machine Learning, June 2007, pp.353-360.

[129]

Hanneke S. Theoretical foundations of active learning [Ph.D. Thesis]. Machine Learning Department, CMU, 2009.

[130]

Hanneke STheory of disagreement-based active learningFoundations and Trends in Machine Learning201472/3131-3091327.68193

[131]

Dasgupta S. Coarse sample complexity bounds for active learning. In Proc. the 19th Annual Conference on Neural Information Processing Systems, December 2005, pp.235-242.

[132]

Tosh C, Dasgupta S. Diameter-based active learning. In Proc. the 34th International Conference on Machine Learning, August 2017, pp.3444-3452.

[133]

Audibert JY and Tsybakov ABFast learning rates for plug-in classifiersThe Annals of Statistics2005352608-63323368611118.62041

[134]

Minsker SPlug-in approach to active learningJournal of Machine Learning Research20121367-9029136941283.68294

[135]

Locatelli A, Carpentier A, Kpotufe S. Adaptivity to noise parameters in nonparametric active learning. In Proc. the 30th Conference on Learning Theory, July 2017, pp.1383-1416.

[136]

Schein AI and Ungar LH Active learning for logistic regression: An evaluation Machine Learning 2007 68 3 235-265

[137]

Melville P, Mooney R. Diverse ensembles for active learning. In Proc. the 21st Int. Conference on Machine Learning, July 2004, pp.584-591.

[138]

Yang Y and Loog M A benchmark and comparison of active learning for logistic regression Pattern Recognition 2018 83 401-415

[139]

Ramirez-Loaiza ME, Sharma M, Kumar G, et al.Active learning: An empirical study of common baselinesData Mining and Knowledge Discovery2017312287-3133608319

[140]

Pupo O, Altalhi H, and Ventura S Statistical comparisons of active learning strategies over multiple datasets Knowledge-Based Systems 2018 145 1 274-288

[141]

Merz C, Murphy P. UCI repository of machine learning databases. http://www.ics.uci.edu/ mlearn/MLRepository. html, Nov. 2019.

[142]

Frey PW and Slate DJ Letter recognition using Holland-style adaptive classifiers Machine Learning 1991 6 2 161-182

[143]

Xu L, Krzyzak A, and Suen C Methods of combining multiple classifiers and their applications to handwritten recognition IEEE Trans. Systems Man and Cybernetics 1992 22 3 418-435

[144]

Garofolo J, Lamel L, FisherWet al. DARPA TIMIT acoustic phonetic continuous speech corpus CD-ROM. Technical Report, 1993. https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir4930.pdf, Nov. 2019.

[145]

Craven M, DiPasquo D, Freitag D, et al.Learning to construct knowledge bases from theWorldWideWebArtificial Intelligence20001181/269-1130939.68745

[146]

LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition Proceedings of the IEEE 1998 86 11 2278-2324

[147]

Lang K. NewsWeeder: Learning to filter net news. In Proc. the 12th Int. Conference on Machine Learning, July 1995, pp.331-339.

[148]

Deng J, Dong W, Socher R et al. ImageNet: A large-scale hierarchical image database. In Proc. the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2009, pp.248-255.

[149]

Sang E F, de Meulder F. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proc. the 7th Conference on Natural Language Learning, May 2003, pp.142-147.

[150]

Collier N, Kim J. Introduction to the bio-entity recognition task at JNLPBA. In Proc. the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications, August 2004, Article No. 13.

[151]

Yeh A, Morgan A, Colosimo M et al. BioCreAtIvE task 1A: Gene mention finding evaluation. BMC Bioinformatics, 2005, 6(S-1): Article No. 2.

[152]

Vlachos A. Evaluating and combining biomedical named entity recognition systems. In Proc. the Workshop on Biological, Translational, and Clinical Language Processing, June 2007, pp.199-200.

[153]

Peng F and McCallum A Information extraction from research papers using conditional random fields Information Processing and Management 2006 42 4 963-979

[154]

de Carvalho V R, Cohen W. Learning to extract signature and reply lines from email. In Proc. the 1st Conference on Email and Anti-Spam, July 2004.

[155]

Guyon I, Cawley G, Dror G et al. Results of the active learning challenge. In Proc. the Active Learning and Experimental Design Workshop, May 2010, pp.19-45.

[156]

Pace RK and Barry RSparse spatial autoregressionsStat. Probab. Lett.1997333291-2970901.62112

[157]

Bay SD, Kibler D, Pazzani M, et al. The UCI KDD archive of large data sets for data mining research and experimentation SIGKDD Explor. 2000 2 2 81-85

[158]

Tang Y P, Li G X, Huang S J. ALiPy: Active learning in Python. arXiv:1901.03802, 2019. https://arxiv.org/abs/1901.03802, Nov. 2019.

[159]

Yang Y Y, Lee S C, Chung Y A et al. libact: Poolbased active learning in Python. arXiv:1710.00379, 2017. https://arxiv.org/abs/1710.00379, October 2019.

[160]

Tran VC, Nguyen NT, Fujita H, et al. A combination of active learning and self-learning for named entity recognition on Twitter using conditional random fields Knowledge-Based Systems 2017 132 179-187

[161]

Scheffer T, Decomain C, Wrobel S. Active hidden Markov models for information extraction. In Proc. the 4th Int. Conference on Advances in Intelligent Data Analysis, September 2001, pp.309-318.

[162]

Aldogan D and Yaslan Y A comparison study on active learning integrated ensemble approaches in sentiment analysis Computers and Electrical Engineering 2017 57 C 311-323

[163]

Zhang H, Huang M, and Zhu X A unified active learning framework for biomedical relation extraction Journal of Computer Science and Technology 2012 27 6 1302-1313

[164]

Hoi S C H, Jin R, Zhu J et al. Batch mode active learning and its application to medical image classification. In Proc. the 23rd Int. Conference on Machine Learning, June 2006, pp.417-424.

[165]

Wallace B C, Small K, Brodley C et al. Active learning for biomedical citation screening. In Proc. the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 2010, pp.173-182.

[166]

Ma A, Patel N, Li M et al. Confidence based active learning for whole object image segmentation. In Proc. the 2006 Int. Workshop on Multimedia Content Representation, Classification and Security, September 2006, pp.753-760.

[167]

Pavlopoulou, Kak A, Brodley C. Application of semisupervised and active learning to interactive contour delineation. In Proc. the ICML 2003 Workshop on the Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining, August 2003, pp.26-33.

[168]

Boutell MR, Luo J, Shen X, et al. Learning multi-label scene classification Pattern Recognition 2004 37 9 1757-1771

[169]

Zhang B, Wang Y, and Chen FMultilabel image classification via high-order label correlation driven active learningIEEE Trans. Image Processing20142331430-144131729971374.94442

[170]

Top A, Hamarneh G, Abugharbieh R. Active learning for interactive 3D image segmentation. In Proc. the 14th Int. Conference on Medical Image Computing and Computerassisted Intervention, September 2011, pp.603-610.

[171]

Caicedo J C, Lazebnik S. Active object localization with deep reinforcement learning. In Proc. the 2015 IEEE Int. Conference on Computer Vision, December 2015, pp.2488-2496.

[172]

Kim Y, Kim S. Design of aging-resistant Wi-Fi fingerprintbased localization system with continuous active learning. In Proc. the 20th Int. Conference on Advanced Communication Technology, February 2018, pp.s1054-1059.

[173]

Ayache S, Qu´enot G. Video corpus annotation using active learning. In Proc. the 30th European Conference on Information Retrieval Research, March 2008, pp.187-198.

[174]

Reker D and Schneider G Active-learning strategies in computer-assisted drug discovery Drug Discovery Today 2015 20 4 458-465

[175]

Warmuth M K, Rätsch G, Mathieson M et al. Active learning in the drug discovery process. In Proc. the 15th Annual Conference on Neural Information Processing Systems, December 2001, pp.1449-1456.

[176]

Figueroa RL, Zeng-Treitler Q, Ngo L, et al. Active learning for clinical text classification: Is it better than random sampling? Journal of the American Medical Informatics Association 2012 19 5 809-816

[177]

Chen Y, Lasko T, Mei Q, et al. A study of active learning methods for named entity recognition in clinical text Journal of Biomedical Informatics 2015 58 1 11-18

[178]

Gu Y, Zydek D. Active learning for intrusion detection. In Proc. the 2014 National Wireless Research Collaboration Symposium, May 2014, pp.117-122.

[179]

Hossain H M S, Roy N, Khan M. Active learning enabled activity recognition. In Proc. the 2016 IEEE Int. Conference on Pervasive Computing and Communications, March 2016, Article No. 26.

[180]

Reker D, Schneider P, and Schneider G Multi-objective active machine learning rapidly improves structure-activity models and reveals new protein-protein interaction inhibitors Chemical Science 2016 7 6 3919-3927

[181]

Yan S, Chaudhuri K, Javidi T. Active learning with logged data. arXiv:1802.09069, 2018. https://arxiv.org/abs/1802.09069, Nov. 2019.

[182]

Danka T, Horvath P. modAL: A modular active learning framework for Python. arXiv:1805.00979, 2018. https://arxiv.org/abs/1805.00979, Nov. 2019.

[183]

Pedregosa F, Varoquaux G, Gramfort A, et al.Scikit-learn: Machine learning in PythonJournal of Machine Learning Research2011122825-283028543481280.68189

[184]

Atienza R. Advanced Deep Learning with Keras: Apply Deep Learning Techniques, Autoencoders, GANs, Variational Autoencoders, Deep Reinforcement Learning, Policy Gradients, and More. Packt Publishing, 2018.

Cited By

He DTaimoor N(2025)Batch-mode active ordinal classification based on expected model output change and leadership treeApplied Intelligence10.1007/s10489-024-06152-z55:4Online publication date: 4-Jan-2025
https://dl.acm.org/doi/10.1007/s10489-024-06152-z
Lämmle SBogoclu CVoßhall RHaselhoff ARoos DKiyavash NMooij J(2024)Quantifying local model validity using active learningProceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence10.5555/3702676.3702776(2113-2135)Online publication date: 15-Jul-2024
https://dl.acm.org/doi/10.5555/3702676.3702776
Padios JGerardo BMedina R(2024)Active Learning with Adaptive Reinforcement for Informative Sample Selection in Few-Shot Image ClassificationProceeding of the 2024 6th International Conference on Information Technology and Computer Communications10.1145/3704391.3704405(95-100)Online publication date: 25-Oct-2024
https://dl.acm.org/doi/10.1145/3704391.3704405
Show More Cited By

Index Terms

Active Learning Query Strategies for Classification, Regression, and Clustering: A Survey
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification
    2. Machine learning approaches
      1. Classification and regression trees

Index terms have been assigned to the content through auto-classification.

Recommendations

Combining active learning and semi-supervised for improving learning performance
ISABEL '11: Proceedings of the 4th International Symposium on Applied Sciences in Biomedical and Communication Technologies

In many learning tasks, there are abundant unlabeled samples but the number of labeled training samples is limited, because labeling the samples requires the efforts of human annotators and expertise. There are three major techniques for labeling the ...
Active learning with direct query construction
KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining

Active learning may hold the key for solving the data scarcity problem in supervised learning, i.e., the lack of labeled data. Indeed, labeling data is a costly process, yet an active learner may request labels of only selected instances, thus reducing ...
Semi-supervised Dictionary Active Learning for Pattern Classification
Pattern Recognition and Computer Vision
Abstract
Gathering labeled data is one of the most time-consuming and expensive tasks in supervised machine learning. In practical applications, there are usually quite limited labeled training samples but abundant unlabeled data that is easy to collect. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal of Computer Science and Technology

Journal of Computer Science and Technology Volume 35, Issue 4

Jul 2020

242 pages

ISSN:1000-9000

Issue’s Table of Contents

© Institute of Computing Technology, Chinese Academy of Sciences 2020.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 July 2020

Revision received: 13 January 2020

Received: 16 February 2019

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

He DTaimoor N(2025)Batch-mode active ordinal classification based on expected model output change and leadership treeApplied Intelligence10.1007/s10489-024-06152-z55:4Online publication date: 4-Jan-2025
https://dl.acm.org/doi/10.1007/s10489-024-06152-z
Lämmle SBogoclu CVoßhall RHaselhoff ARoos DKiyavash NMooij J(2024)Quantifying local model validity using active learningProceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence10.5555/3702676.3702776(2113-2135)Online publication date: 15-Jul-2024
https://dl.acm.org/doi/10.5555/3702676.3702776
Padios JGerardo BMedina R(2024)Active Learning with Adaptive Reinforcement for Informative Sample Selection in Few-Shot Image ClassificationProceeding of the 2024 6th International Conference on Information Technology and Computer Communications10.1145/3704391.3704405(95-100)Online publication date: 25-Oct-2024
https://dl.acm.org/doi/10.1145/3704391.3704405
Padios JGerardo BMedina R(2024)Reinforcement Learning-Driven Active Few-Shot Learning Framework with Hyperparameter Optimization for Rice Pest ClassificationProceedings of the 2024 2nd International Conference on Communications, Computing and Artificial Intelligence10.1145/3676581.3676591(53-58)Online publication date: 21-Jun-2024
https://dl.acm.org/doi/10.1145/3676581.3676591
Li NQi YLi CZhao Z(2024)Active Learning for Data Quality Control: A SurveyJournal of Data and Information Quality10.1145/366336916:2(1-45)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3663369
Wyss AMorgenshtern GHirsch-Hüsler ABernard J(2024)DaedalusData: Exploration, Knowledge Externalization and Labeling of Particles in Medical Manufacturing — A Design StudyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345632931:1(54-64)Online publication date: 23-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456329
Beigrezaei ACabral HTsiporkova E(2024)Active domain adaptation in support of reliable operating mode detection for wind turbinesProcedia Computer Science10.1016/j.procs.2024.09.545246:C(1191-1200)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.procs.2024.09.545
Gobert MBriffoteaux GGmys JMelab NTuyttens D(2024)Observations in applying Bayesian versus evolutionary approaches and their hybrids in parallel time-constrained optimizationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109075137:PAOnline publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.109075
Cacciarelli DKulahci M(2024)Active learning for data streams: a surveyMachine Language10.1007/s10994-023-06454-2113:1(185-239)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s10994-023-06454-2
Lu SZheng JLi ZDai X(2024)WMBAL: weighted minimum bounds for active learningApplied Intelligence10.1007/s10489-024-05328-x54:3(2551-2563)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s10489-024-05328-x
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents