Elamur Research
☰
home / papers / 2009 / Active Learning Literature Survey

Active Learning Literature Survey

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "author's copy (UW–Madison CS TR 1648) — read only" does not allow redistributing the text itself. Original — burrsettles.com/pub/settles.activelearning.pdf

Summary

Supervised learning is often held back by annotation. Raw instances are usually cheap and plentiful, but having experts label them is slow and costly. In this technical report, Burr Settles surveys active learning, where the learner itself chooses which examples to send to an oracle (usually a human annotator). The goal is to reach good accuracy with considerably fewer labels. The report presents no new experiments. Instead, it organizes the existing literature in several ways:

  • It describes the settings in which a learner can request labels.
  • It groups the proposed query selection methods into major families.
  • It reviews empirical and theoretical evidence on when active learning pays off.

It also covers extended problem settings, practical difficulties in real annotation projects, and connections to neighbouring research fields. Its main contribution is a structured taxonomy and entry point to the area. This makes it useful for planning label-efficient annotation, such as building NLP corpora on a limited budget.

Key points

  • Publication: issued in 2009 as technical report number 1648 by the computer sciences department at the University of Wisconsin–Madison, with a revision dated January 26, 2010.
  • Three querying scenarios: membership query synthesis, stream-based selective sampling and pool-based sampling.
  • Six query strategy frameworks: uncertainty sampling, query-by-committee, expected model change, expected error reduction, variance reduction and density-weighted methods.
  • Problem variants: structured outputs, active feature acquisition and classification, active class selection and active clustering. Seven practical issues: batch-mode selection, noisy oracles, variable labeling costs, alternative query types, multi-task learning, changing or unknown model classes and stopping criteria.
  • Related fields: semi-supervised learning, reinforcement learning, submodular optimization, equivalence query learning, and model parroting/compression.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

361 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 356 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{settles2009active,
  title = {{Active Learning Literature Survey}},
  author = {Settles, Burr},
  booktitle = {Computer Sciences Technical Report 1648, University of Wisconsin–Madison},
  year = {2009},
  url = {https://burrsettles.com/pub/settles.activelearning.pdf}
}

References

160 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. N. Abe and H. Mamitsuka. Query learning strategies using boosting and bagging. In Proceedings of the International Conference on Machine Learning (ICML), pages 1–9. Morgan Kaufmann, 1998. find
  2. S. Andrews, I. Tsochantaridis, and T. Hofmann. Support vector machines for multiple-instance learning. In Advances in Neural Information Processing Systems (NIPS), volume 15, pages 561–568. MIT Press, 2003. find
  3. D. Andrzejewski, X. Zhu, and M. Craven. Incorporating domain knowledge into topic modeling via Dirichlet forest priors. In Proceedings of the International Conference on Machine Learning (ICML), pages 25–32. ACM Press, 2009. find
  4. D. Angluin. Queries and concept learning. Machine Learning, 2:319–342, 1988. find
  5. D. Angluin. Queries revisited. In Proceedings of the International Conference on Algorithmic Learning Theory, pages 12–31. Springer-Verlag, 2001. find
  6. S. Arora, E. Nyberg, and C.P. Rosé. Estimating annotation cost for active learning in a multi-annotator environment. In Proceedings of the NAACL HLT Workshop on Active Learning for Natural Language Processing, pages 18–26. ACL Press, 2009. find
  7. M.F. Balcan, A. Beygelzimer, and J. Langford. Agnostic active learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 65–72. ACM Press, 2006. find
  8. M.F. Balcan, S. Hanneke, and J. Wortman. The true sample complexity of active learning. In Proceedings of the Conference on Learning Theory (COLT), pages 45–56. Springer, 2008. find
  9. J. Baldridge and M. Osborne. Active learning and the total cost of annotation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9–16. ACL Press, 2004. find
  10. J. Baldridge and A. Palmer. How well does active learning actually work? Time-based evaluation of cost-reduction strategies for language documentation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 296–305. ACL Press, 2009. find
  11. J. Baxter, A. Tridgell, and L. Weaver. Reinforcement learning and chess. In J. Furnkranz and M. Kubat, editors, Machines that Learn to Play Games, pages 91–116. Nova Science Publishers, 2001. find
  12. A.L. Berger, V.J. Della Pietra, and S.A. Della Pietra. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39– 71, 1996. find our card
  13. A. Beygelzimer, S. Dasgupta, and J. Langford. Importance-weighted active learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 49–56. ACM Press, 2009. find
  14. M. Bloodgood and V. Shanker. A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. In Proceedings of the Conference on Natural Language Learning (CoNLL), pages 39–47. ACL Press, 2009. find
  15. A. Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the Conference on Learning Theory (COLT), pages 92–100. Morgan Kaufmann, 1998. find our card
  16. A. Blum, A. Frieze, R. Kannan, and S. Vempala. A polynomial-time algorithm for learning noisy linear threshold functions. In Proceedings of the IEEE Symposium on the Foundations of Computer Science, page 330. IEEE Press, 1996. find
  17. C. Bonwell and J. Eison. Active Learning: Creating Excitement in the Classroom. Jossey-Bass, 1991. find
  18. L. Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996. find our card
  19. K. Brinker. Incorporating diversity in active learning with support vector machines. In Proceedings of the International Conference on Machine Learning (ICML), pages 59–66. AAAI Press, 2003. find
  20. C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil. Model compression. In Proceedings of the International Conference on Knowledge Discovery and Data Mining (KDD), pages 535–541. ACM Press, 2006. find
  21. R. Burbidge, J.J. Rowland, and R.D. King. Active learning for regression based on query by committee. In Proceedings of Intelligent Data Engineering and Automated Learning (IDEAL), pages 209–218. Springer, 2007. find
  22. A. Carlson, J. Betteridge, R. Wang, E.R. Hruschka Jr, and T. Mitchell. Coupled semi-supervised learning for information extraction. In Proceedings of the International Conference on Web Search and Data Mining (WSDM). ACM Press, 2010. find
  23. X. Chai, L. Deng, Q. Yang, and C.X. Ling. Test-cost sentistive naive Bayes classification. In Proceedings of the IEEE Conference on Data Mining (ICDM), pages 51–5. IEEE Press, 2004. find
  24. K. Chaloner and I. Verdinelli. Bayesian experimental design: A review. Statistical Science, 10(3):237–304, 1995. find
  25. S.F. Chen and R. Rosenfeld. A survey of smoothing techniques for ME models. IEEE Transactions on Speech and Audio Processing, 8(1):37–50, 2000. find
  26. D. Cohn. Neural network exploration using optimal experiment design. In Advances in Neural Information Processing Systems (NIPS), volume 6, pages 679–686. Morgan Kaufmann, 1994. find
  27. D. Cohn, L. Atlas, R. Ladner, M. El-Sharkawi, R. Marks II, M. Aggoune, and D. Park. Training connectionist networks with queries and selective sampling. In Advances in Neural Information Processing Systems (NIPS). Morgan Kaufmann, 1990. find
  28. D. Cohn, L. Atlas, and R. Ladner. Improving generalization with active learning. Machine Learning, 15(2):201–221, 1994. find
  29. D. Cohn, Z. Ghahramani, and M.I. Jordan. Active learning with statistical models. Journal of Artificial Intelligence Research, 4:129–145, 1996. find
  30. T.M. Cover and J.A. Thomas. Elements of Information Theory. Wiley, 2006. find
  31. M. Craven and J. Shavlik. Extracting tree-structured representations of trained networks. In Advances in Neural Information Processing Systems (NIPS), volume 8, pages 24–30. MIT Press, 1996. find
  32. N. Cressie. Statistics for Spatial Data. Wiley, 1991. find
  33. A. Culotta and A. McCallum. Reducing labeling effort for stuctured prediction tasks. In Proceedings of the National Conference on Artificial Intelligence (AAAI), pages 746–751. AAAI Press, 2005. find
  34. I. Dagan and S. Engelson. Committee-based sampling for training probabilistic classifiers. In Proceedings of the International Conference on Machine Learning (ICML), pages 150–157. Morgan Kaufmann, 1995. find
  35. S. Dasgupta. Analysis of a greedy active learning strategy. In Advances in Neural Information Processing Systems (NIPS), volume 16, pages 337–344. MIT Press, 2004. find
  36. S. Dasgupta and D.J. Hsu. Hierarchical sampling for active learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 208–215. ACM Press, 2008. find
  37. S. Dasgupta, A. Kalai, and C. Monteleoni. Analysis of perceptron-based active learning. In Proceedings of the Conference on Learning Theory (COLT), pages 249–263. Springer, 2005. find
  38. S. Dasgupta, D. Hsu, and C. Monteleoni. A general agnostic active learning algorithm. In Advances in Neural Information Processing Systems (NIPS), volume 20, pages 353–360. MIT Press, 2008. find
  39. V.R. de Sa. Learning classification with unlabeled data. In Advances in Neural Information Processing Systems (NIPS), volume 6, pages 112–119. MIT Press, 1994. find
  40. T. Dietterich, R. Lathrop, and T. Lozano-Perez. Solving the multiple-instance problem with axis-parallel rectangles. Artificial Intelligence, 89:31–71, 1997. find
  41. P. Donmez, J. Carbonell, and J. Schneider. Efficiently learning the accuracy of labeling sources for selective sampling. In Proceedings of the International Conference on Knowledge Discovery and Data Mining (KDD), pages 259–268. ACM Press, 2009. find
  42. G. Druck, G. Mann, and A. McCallum. Learning from labeled features using generalized expectation criteria. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, pages 595–602. ACM Press, 2008. find
  43. G. Druck, B. Settles, and A. McCallum. Active learning by labeling features. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 81–90. ACL Press, 2009. find
  44. R. Duda, P. Hart, and D. Stork. Pattern Classification. Wiley-Interscience, 2001. find
  45. S. Esmeir and S. Markovitch. Anytime induction of cost-sensitive trees. In Advances in Neural Information Processing Systems (NIPS), volume 20, pages 425–432. MIT Press, 2008. find
  46. V. Federov. Theory of Optimal Experiments. Academic Press, 1972. find
  47. P. Flaherty, M. Jordan, and A. Arkin. Robust design of biological experiments. In Advances in Neural Information Processing Systems (NIPS), volume 18, pages 363–370. MIT Press, 2006. find
  48. Y. Freund and R.E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119–139, 1997. find
  49. Y. Freund, H.S. Seung, E. Shamir, and N. Tishby. Selective samping using the query by committee algorithm. Machine Learning, 28:133–168, 1997. find
  50. A. Fujii, T. Tokunaga, K. Inui, and H. Tanaka. Selective sampling for example-based word sense disambiguation. Computational Linguistics, 24(4):573–597, 1998. find
  51. C. Gasperin. Active learning for anaphora resolution. In Proceedings of the NAACL HLT Workshop on Active Learning for Natural Language Processing, pages 1–8. ACL Press, 2009. find
  52. S. Geman, E. Bienenstock, and R. Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 4:1–58, 1992. find
  53. R. Gilad-Bachrach, A. Navot, and N. Tishby. Query by committee made real. In Advances in Neural Information Processing Systems (NIPS), volume 18, pages 443–450. MIT Press, 2006. find
  54. J. Goodman. Exponential priors for maximum entropy models. In Proceedings of Human Language Technology and the North American Association for Computational Linguistics (HLT-NAACL), pages 305–312. ACL Press, 2004. find
  55. R. Greiner, A. Grove, and D. Roth. Learning cost-sensitive active classifiers. Artificial Intelligence, 139:137–174, 2002. find
  56. N. Grira, M. Crucianu, and N. Boujemaa. Active semi-supervised fuzzy clustering for image database categorization. In Proceedings the ACM Workshop on Multimedia Information Retrieval (MIR), pages 9–16. ACM Press, 2005. find
  57. C. Guestrin, A. Krause, and A.P. Singh. Near-optimal sensor placements in gaussian processes. In Proceedings of the International Conference on Machine Learning (ICML), pages 265–272. ACM Press, 2005. find
  58. Y. Guo and R. Greiner. Optimistic active learning using mutual information. In Proceedings of International Joint Conference on Artificial Intelligence (IJ-CAI), pages 823–829. AAAI Press, 2007. find
  59. Y. Guo and D. Schuurmans. Discriminative batch mode active learning. In Advances in Neural Information Processing Systems (NIPS), number 20, pages 593–600. MIT Press, Cambridge, MA, 2008. find
  60. R. Haertel, K. Seppi, E. Ringger, and J. Carroll. Return on investment for active learning. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning, 2008. find
  61. A. Haghighi and D. Klein. Prototype-driven learning for sequence models. In Proceedings of the North American Association for Computational Linguistics (NAACL), pages 320–327. ACL Press, 2006. find
  62. S. Hanneke. A bound on the label complexity of agnostic active learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 353–360. ACM Press, 2007. find
  63. A. Hauptmann, W. Lin, R. Yan, J. Yang, and M.Y. Chen. Extreme video retrieval: joint maximization of human and computer performance. In Proceedings of the ACM Workshop on Multimedia Image Retrieval, pages 385–394. ACM Press, 2006. find
  64. D. Haussler. Learning conjunctive concepts in structural domains. Machine Learning, 4(1):7–40, 1994. find
  65. T. Hofmann and J.M. Buhmann. Active data clustering. In Advances in Neural Information Processing Systems (NIPS), volume 10, pages 528—534. Morgan Kaufmann, 1998. find
  66. S.C.H. Hoi, R. Jin, and M.R. Lyu. Large-scale text categorization by batch mode active learning. In Proceedings of the International Conference on the World Wide Web, pages 633–642. ACM Press, 2006a. find
  67. S.C.H. Hoi, R. Jin, J. Zhu, and M.R. Lyu. Batch mode active learning and its application to medical image classification. In Proceedings of the International Conference on Machine Learning (ICML), pages 417–424. ACM Press, 2006b. find
  68. Y. Huang and T. Mitchell. Text clustering with extended user feedback. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, pages 413–420. ACM Press, 2006. find
  69. R. Hwa. On minimizing training corpus for parser acquisition. In Proceedings of the Conference on Natural Language Learning (CoNLL), pages 1–6. ACL Press, 2001. find
  70. R. Hwa. Sample selection for statistical parsing. Computational Linguistics, 30 (3):73–77, 2004. find
  71. S. Ji and L. Carin. Cost-sensitive feature acquisition and classification. Pattern Recognition, 40:1474–1485, 2007. find
  72. A. Kapoor, E. Horvitz, and S. Basu. Selective supervision: Guiding supervised learning with decision-theoretic active learning,. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), pages 877–882. AAAI Press, 2007. find
  73. R.D. King, K.E. Whelan, F.M. Jones, P.G. Reiser, C.H. Bryant, S.H. Muggleton, D.B. Kell, and S.G. Oliver. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427(6971):247–52, 2004. find
  74. R.D. King, J. Rowland, S.G. Oliver, M. Young, W. Aubrey, E. Byrne, M. Liakata, M. Markham, P. Pir, L.N. Soldatova, A. Sparkes, K.E. Whelan, and A. Clare. The automation of science. Science, 324(5923):85–89, 2009. find
  75. C. Körner and S. Wrobel. Multi-class ensemble-based active learning. In Proceedings of the European Conference on Machine Learning (ECML), pages 687–694. Springer, 2006. find
  76. A. Krause. Optimizing Sensing: Theory and Applications. PhD thesis, Carnegie Mellon University, 2008. find
  77. V. Krishnamurthy. Algorithms for optimal scheduling and management of hidden markov model sensors. IEEE Transactions on Signal Processing, 50(6):1382– 1397, 2002. find
  78. S. Kullback and R.A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22:79–86, 1951. find
  79. K. Lang. Newsweeder: Learning to filter netnews. In Proceedings of the International Conference on Machine Learning (ICML), pages 331–339. Morgan Kaufmann, 1995. find
  80. K. Lang and E. Baum. Query learning can work poorly when a human oracle is used. In Proceedings of the IEEE International Joint Conference on Neural Networks, pages 335–340. IEEE Press, 1992. find
  81. D. Lewis and J. Catlett. Heterogeneous uncertainty sampling for supervised learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 148–156. Morgan Kaufmann, 1994. find
  82. D. Lewis and W. Gale. A sequential algorithm for training text classifiers. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3–12. ACM/Springer, 1994. find
  83. P. Liang, H. Daume, and D. Klein. Structure compilation: Trading structure for features. In Proceedings of the International Conference on Machine Learning (ICML), pages 592–599. ACM Press, 2008. find
  84. P. Liang, M.I. Jordan, and D. Klein. Learning from measurements in exponential families. In Proceedings of the International Conference on Machine Learning (ICML), pages 641–648. ACM Press, 2009. find
  85. M. Lindenbaum, S. Markovitch, and D. Rusakov. Selective sampling for nearest neighbor classifiers. Machine Learning, 54(2):125–152, 2004. find
  86. C.X. Ling, Q. Yang, J. Wang, and S. Zhang. Decision trees with minimal costs. In Proceedings of the International Conference on Machine Learning (ICML), pages 483–486. ACM Press, 2004. find
  87. Y. Liu. Active learning with support vector machine applied to gene expression data for cancer classification. Journal of Chemical Information and Computer Sciences, 44:1936–1941, 2004. find
  88. R. Lomasky, C.E. Brodley, M. Aernecke, D. Walt, and M. Friedl. Active class selection. In Proceedings of the European Conference on Machine Learning (ECML), pages 640–647. Springer, 2007. find
  89. Z. Lu and J. Bongard. Exploiting multiple classifier types with active learning. In Proceedings of the Conference on Genetic and Evolutionary Computation (GECCO), pages 1905–1906. find
  90. D. MacKay. Information-based objective functions for active data selection. Neural Computation, 4(4):590–604, 1992. find
  91. G. Mann and A. McCallum. Generalized expectation criteria for semi-supervised learning of conditional random fields. In Proceedings of the Association for Computational Linguistics (ACL). ACL Press, 2008. find
  92. D. Margineantu. Active cost-sensitive learning. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), pages 1622–1623. AAAI Press, 2005. find
  93. O. Maron and T. Lozano-Perez. A framework for multiple-instance learning. In Advances in Neural Information Processing Systems (NIPS), volume 10, pages 570–576. MIT Press, 1998. find
  94. A. McCallum and K. Nigam. Employing EM in pool-based active learning for text classification. In Proceedings of the International Conference on Machine Learning (ICML), pages 359–367. Morgan Kaufmann, 1998. find
  95. P. Melville and R. Mooney. Diverse ensembles for active learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 584–591. Morgan Kaufmann, 2004. find
  96. P. Melville, M. Saar-Tsechansky, F. Provost, and R. Mooney. Active feature-value acquisition for classifier induction. In Proceedings of the IEEE Conference on Data Mining (ICDM), pages 483–486. IEEE Press, 2004. find
  97. P. Melville, S.M. Yang, M. Saar-Tsechansky, and R. Mooney. Active learning for probability estimation using Jensen-Shannon divergence. In Proceedings of the European Conference on Machine Learning (ECML), pages 268–279. Springer, 2005. find
  98. L. Mihalkova and R. Mooney. Using active relocation to aid reinforcement learning. In Proccedings of the Florida Artificial Intelligence Research Society (FLAIRS), pages 580–585. AAAI Press, 2006. find
  99. M. Mintz, S. Bills, R. Snow, and D. Jurafsky. Distant supervision for relation extraction without labeled data. In Proceedings of the Association for Computational Linguistics (ACL), 2009. find
  100. T. Mitchell. Generalization as search. Artificial Intelligence, 18:203–226, 1982. find
  101. T. Mitchell. Machine Learning. McGraw-Hill, 1997. find
  102. C. Monteleoni. Learning with Online Constraints: Shifting Concepts and Active Learning. PhD thesis, Massachusetts Institute of Technology, 2006. find
  103. R. Moskovitch, N. Nissim, D. Stopel, C. Feher, R. Englert, and Y. Elovici. Improving the detection of unknown computer worms activity using active learning. In Proceedings of the German Conference on AI, pages 489–493. Springer, 2007. find
  104. I. Muslea, S. Minton, and C.A. Knoblock. Selective sampling with redundant views. In Proceedings of the National Conference on Artificial Intelligence (AAAI), pages 621–626. AAAI Press, 2000. find
  105. G.L. Nemhauser, L.A. Wolsey, and M.L. Fisher. An analysis of approximations for maximizing submodular set functions. Mathematical Programming, 14(1): 265–294, 1978. find
  106. H.T. Nguyen and A. Smeulders. Active learning using pre-clustering. In Proceedings of the International Conference on Machine Learning (ICML), pages 79–86. ACM Press, 2004. find
  107. F. Olsson. Bootstrapping Named Entity Recognition by Means of Active Machine Learning. PhD thesis, University of Gothenburg, 2008. find
  108. F. Olsson. A literature survey of active machine learning in the context of natural language processing. Technical Report T2009:06, Swedish Institute of Computer Science, 2009. find
  109. F. Olsson and K. Tomanek. An intrinsic stopping criterion for committee-based active learning. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL), pages 138–146. ACL Press, 2009. find
  110. G. Paass and J. Kindermann. Bayesian query construction for neural network models. In Advances in Neural Information Processing Systems (NIPS), volume 7, pages 443–450. MIT Press, 1995. find
  111. G.J. Qi, X.S. Hua, Y. Rui, J. Tang, and H.J. Zhang. Two-dimensional active learning for image classification. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2008. find
  112. H. Raghavan, O. Madani, and R. Jones. Active learning with feedback on both features and instances. Journal of Machine Learning Research, 7:1655–1686, 2006. find
  113. R. Rahmani and S.A. Goldman. MISSL: Multiple-instance semi-supervised learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 705–712. ACM Press, 2006. find
  114. S. Ray and M. Craven. Supervised versus multiple instance learning: An empirical comparison. In Proceedings of the International Conference on Machine Learning (ICML), pages 697–704. ACM Press, 2005. find
  115. R. Reichart, K. Tomanek, U. Hahn, and A Rappoport. Multi-task active learning for linguistic annotations. In Proceedings of the Association for Computational Linguistics (ACL), pages 861–869. ACL Press, 2008. find
  116. E. Ringger, M. Carmen, R. Haertel, K. Seppi, D. Lonsdale, P. McClanahan, J. Carroll, and N. Ellison. Assessing the costs of machine-assisted corpus annotation through a user study. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). European Language Resources… find
  117. N. Roy and A. McCallum. Toward optimal active learning through sampling estimation of error reduction. In Proceedings of the International Conference on Machine Learning (ICML), pages 441–448. Morgan Kaufmann, 2001. find
  118. M. Saar-Tsechansky, P. Melville, and F. Provost. Active feature-value acquisition. Management Science, 55(4):664–684, 2009. find
  119. T. Scheffer, C. Decomain, and S. Wrobel. Active hidden Markov models for information extraction. In Proceedings of the International Conference on Advances in Intelligent Data Analysis (CAIDA), pages 309–318. Springer-Verlag, 2001. find
  120. A.I. Schein and L.H. Ungar. Active learning for logistic regression: An evaluation. Machine Learning, 68(3):235–265, 2007. find
  121. M.J. Schervish. Theory of Statistics. Springer, 1995. find
  122. B. Settles. Curious Machines: Active Learning with Structured Instances. PhD thesis, University of Wisconsin–Madison, 2008. find
  123. B. Settles and M. Craven. An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1069–1078. ACL Press, 2008. find
  124. B. Settles, M. Craven, and L. Friedland. Active learning with real annotation costs. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning, pages 1–10, 2008a. find
  125. B. Settles, M. Craven, and S. Ray. Multiple-instance active learning. In Advances in Neural Information Processing Systems (NIPS), volume 20, pages 1289– 1296. MIT Press, 2008b. find
  126. H.S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In Proceedings of the ACM Workshop on Computational Learning Theory, pages 287–294, 1992. find
  127. C.E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27:379–423,623–656, 1948. find
  128. V.S. Sheng and C.X. Ling. Feature value acquisition in testing: A sequential batch test algorithm. In Proceedings of the International Conference on Machine Learning (ICML), pages 809–816. ACM Press, 2006. find
  129. V.S. Sheng, F. Provost, and P.G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the International Conference on Knowledge Discovery and Data Mining (KDD). ACM Press, 2008. find
  130. V. Sindhwani, P. Melville, and R.D. Lawrence. Uncertainty sampling and transductive experimental design for active dual supervision. In Proceedings of the International Conference on Machine Learning (ICML), pages 953–960. ACM Press, 2009. find
  131. R. Snow, B. O’Connor, D. Jurafsky, and A. Ng. Cheap and fast—but is it good? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 254–263. ACM Press, 2008. find
  132. M. Steyvers and T. Griffiths. Probabilistic topic models. In T. Landauer, D.S. McNamara, S. Dennis, and W. Kintsch, editors, Handbook of Latent Semantic Analysis. Erlbaum, 2007. find
  133. M. Sugiyama and N. Rubens. Active learning with model selection in linear regression. In Proceedings of the SIAM International Conference on Data Mining, pages 518–529. SIAM, 2008. find
  134. R. Sutton and A.G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998. find
  135. C.A. Thompson, M.E. Califf, and R.J. Mooney. Active learning for natural language parsing and information extraction. In Proceedings of the International Conference on Machine Learning (ICML), pages 406–414. Morgan Kaufmann, 1999. find
  136. K. Tomanek and U. Hahn. Semi-supervised active learning for sequence labeling. In Proceedings of the Association for Computational Linguistics (ACL), pages 1039–1047. ACL Press, 2009. find
  137. K. Tomanek and F. Olsson. A web survey on the use of active learning to support annotation of text data. In Proceedings of the NAACL HLT Workshop on Active Learning for Natural Language Processing, pages 45–48. ACL Press, 2009. find
  138. K. Tomanek, J. Wermter, and U. Hahn. An approach to text corpus construction which cuts annotation costs and maintains reusability of annotated data. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 486–495. ACL Press, 2007. find
  139. S. Tong. Active Learning: Theory and Applications. PhD thesis, Stanford University, 2001. find
  140. S. Tong and E. Chang. Support vector machine active learning for image retrieval. In Proceedings of the ACM International Conference on Multimedia, pages 107–118. ACM Press, 2001. find
  141. S. Tong and D. Koller. Support vector machine active learning with applications to text classification. In Proceedings of the International Conference on Machine Learning (ICML), pages 999–1006. Morgan Kaufmann, 2000. find
  142. G. Tür, D. Hakkani-Tür, and R.E. Schapire. Combining active and semi-supervised learning for spoken language understanding. Speech Communication, 45(2):171–186, 2005. find
  143. L.G. Valiant. A theory of the learnable. Communications of the ACM, 27(11): 1134–1142, 1984. find
  144. V.N. Vapnik and A. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, 16:264–280, 1971. find
  145. S. Vijayanarasimhan and K. Grauman. What’s it going to cost you? Predicting effort vs. informativeness for multi-label image annotations. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Press, 2009a. find
  146. S. Vijayanarasimhan and K. Grauman. Multi-level active prediction of useful image annotations for recognition. In Advances in Neural Information Processing Systems (NIPS), volume 21, pages 1705–1712. MIT Press, 2009b. find
  147. A. Vlachos. A stopping criterion for active learning. Computer Speech and Language, 22(3):295–312, 2008. find
  148. Z. Xu, R. Akella, and Y. Zhang. Incorporating diversity and density in active learning for relevance feedback. In Proceedings of the European Conference on IR Research (ECIR), pages 246–257. Springer-Verlag, 2007. find
  149. R. Yan, J. Yang, and A. Hauptmann. Automatically labeling video data using multi-class active learning. In Proceedings of the International Conference on Computer Vision, pages 516–523. IEEE Press, 2003. find
  150. D. Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the Association for Computational Linguistics (ACL), pages 189–196. ACL Press, 1995. find our card
  151. H. Yu. SVM selective sampling for ranking with application to data retrieval. In Proceedings of the International Conference on Knowledge Discovery and Data Mining (KDD), pages 354–363. ACM Press, 2005. find
  152. K. Yu, J. Bi, and V. Tresp. Active learning via transductive experimental design. In Proceedings of the International Conference on Machine Learning (ICML), pages 1081–1087. ACM Press, 2006. find
  153. C. Zhang and T. Chen. An active learning framework for content based information retrieval. IEEE Transactions on Multimedia, 4(2):260–268, 2002. find
  154. T. Zhang and F.J. Oles. A probability analysis on the value of unlabeled data for classification problems. In Proceedings of the International Conference on Machine Learning (ICML), pages 1191–1198. Morgan Kaufmann, 2000. find
  155. Z. Zheng and B. Padmanabhan. On active learning for data acquisition. In Proceedings of the IEEE Conference on Data Mining (ICDM), pages 562–569. IEEE Press, 2002. find
  156. Z.H. Zhou, K.J. Chen, and Y. Jiang. Exploiting unlabeled data in content-based image retrieval. In Proceedings of the European Conference on Machine Learning (ECML), pages 425–435. Springer, 2004. find
  157. X. Zhu. Semi-Supervised Learning with Graphs. PhD thesis, Carnegie Mellon University, 2005a. find
  158. X. Zhu. Semi-supervised learning literature survey. Computer Sciences Technical Report 1530, University of Wisconsin–Madison, 2005b. find
  159. X. Zhu, J. Lafferty, and Z. Ghahramani. Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions. In Proceedings of the ICML Workshop on the Continuum from Labeled to Unlabeled Data, pages 58–65, 2003. find
  160. Index 0/1-loss, 19 active class selection, 33 active classification, 32 active clustering, 33 active feature acquisition, 32 active learning, 4 active learning with feature labels, 41 agnostic active learning, 29 alternating selection, 42 batch-mode active learning, 35 classification, 4 cost-sensiti… find