Elamur Research⌕
☰
home / papers / 2017 / HDLTex: Hierarchical Deep Learning for Text Classification

HDLTex: Hierarchical Deep Learning for Text Classification

Paper · 2017 · Kamran Kowsari, Donald E. Brown, Mojtaba Heidarysafa et al.

Summary

Sorting scientific literature automatically gets harder as collections grow, because topics and subtopics multiply too. Flat multi-class classifiers struggle once every field–subfield pair becomes its own label. HDLTex (Hierarchical Deep Learning for Text classification) treats the task as hierarchical. A parent-level deep network assigns a document to a broad field, such as Computer Science or Medical Science. A separate child-level network trained for that field then selects the specialization. Each level can be a DNN, CNN or RNN. CNN and RNN models read 100-dimensional GloVe embeddings, while DNN models use N-gram counts and tf-idf. The authors compare fifteen methods, including non-hierarchical baselines such as SVM variants and naïve Bayes. The references point to Web of Science as the document source. HDLTex is reported to beat traditional classifiers in accuracy. The provided excerpt omits the exact figures and dataset sizes. The open-source code is a reusable template for two-level topic classification and suits any library-style document taxonomy.

Key points

  • Motivation: in 2014, about 28,100 active peer-reviewed English-language journals published roughly 2.5 million articles, and the growth rate appears to be increasing.
  • Architecture: one parent model predicts the top-level field, and a dedicated child model (Ψ_i) for each field predicts the subfield. The deep learning stacks can mix DNN, CNN and RNN.
  • Features: CNN/RNN use 100-dimensional GloVe word vectors. DNN uses count-based N-gram features (e.g., 1- plus 2-grams) and tf-idf weighting.
  • Baselines: term-weighted SVM, multi-word SVM, naïve Bayes classification, stacking SVM, and flat (non-hierarchical) DNN, CNN and RNN. Fifteen methods are compared in total.
  • Resource: the code is released at github.com/kk7nc/HDLTex, and the implementation cites TensorFlow and Keras.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of VirginiaCharlottesville · United States40 works · 10,099 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Text classification · 50 works

Cite

Cite as Kamran Kowsari, Donald E. Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S. Gerber and Laura E. Barnes. 2017. HDLTex: Hierarchical Deep Learning for Text Classification. ICMLA 2017.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1709.08267

@inproceedings{kowsari2017hdltex,
  title = {{HDLTex: Hierarchical Deep Learning for Text Classification}},
  author = {Kamran Kowsari, Donald E. Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S. Gerber and Laura E. Barnes},
  booktitle = {ICMLA 2017},
  year = {2017},
  doi = {10.1109/icmla.2017.0-134},
  eprint = {1709.08267},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/icmla.2017.0-134}
}

References

51 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Ware and M. Mabe, “The stm report: An overview of scientific and scholarly journal publishing,” 2015. find
  2. C. D. Manning, P. Raghavan, H. Schütze et al., Introduction to information retrieval. Cambridge university press Cambridge, 2008, vol. 1, no. 1. find
  3. J. C. French, D. E. Brown, and N.-H. Kim, “A classification approach to boolean query reformulation,” JASIS, vol. 48, no. 8, pp. 694–706, 1997. find
  4. K. Kowsari, M. Yammahi, N. Bari, R. Vichr, F. Alsaby, and S. Y. Berkovich, “Construction of fuzzyfind dictionary using golay coding transformation for searching applications,” International Journal of Advanced Computer Science and Applications(IJACSA), vol. 6, pp. 81– 87, 2015. find
  5. T. Joachims, “Transductive inference for text classification using support vector machines,” in ICML, vol. 99, 1999, pp. 200–209.
  6. S. Tong and D. Koller, “Support vector machine active learning with applications to text classification,” Journal of machine learning research, vol. 2, no. Nov, pp. 45–66, 2001. find
  7. M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim, “Do we need hundreds of classifiers to solve real world classification problems,” J. Mach. Learn. Res, vol. 15, no. 1, pp. 3133–3181, 2014. find
  8. C. S. Leslie, E. Eskin, and W. S. Noble, “The spectrum kernel: A string kernel for svm protein classification.” in Pacific symposium on biocomputing, vol. 7, no. 7, 2002, pp. 566–575. find
  9. E. Eskin, J. Weston, W. S. Noble, and C. S. Leslie, “Mismatch string kernels for svm protein classification,” in Advances in neural information processing systems, 2003, pp. 1441–1448. find
  10. A. McCallum, K. Nigam et al., “A comparison of event models for naive bayes text classification,” in AAAI-98 workshop on learning for text categorization, vol. 752. Citeseer, 1998, pp. 41–48.
  11. S.-B. Kim, K.-S. Han, H.-C. Rim, and S. H. Myaeng, “Some effective techniques for naive bayes text classification,” IEEE transactions on knowledge and data engineering, vol. 18, no. 11, pp. 1457–1466, 2006. find
  12. N. Van Linh, N. K. Anh, K. Than, and C. N. Dang, “An effective and interpretable method for document classification,” Knowledge and Information Systems, vol. 50, no. 3, pp. 763–793, 2017. find
  13. G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” science, vol. 313, no. 5786, pp. 504–507, 2006. find
  14. R. Johnson and T. Zhang, “Effective use of word order for text categorization with convolutional neural networks,” arXiv preprint arXiv:1412.1058, 2014.
  15. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. find
  16. M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and transferring mid-level image representations using convolutional neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 1717–1724. find
  17. J. Y. Lee and F. Dernoncourt, “Sequential short-text classification with recurrent and convolutional neural networks,” arXiv preprint arXiv:1603.03827, 2016.
  18. X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” in Advances in neural information processing systems, 2015, pp. 649–657.
  19. L. Medsker and L. Jain, “Recurrent neural networks,” Design and Applications, vol. 5, 2001. find
  20. R. Salakhutdinov, J. B. Tenenbaum, and A. Torralba, “Learning with hierarchical-deep models,” IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 8, pp. 1958–1971, 2013. find
  21. G. B. Huang, H. Lee, and E. Learned-Miller, “Learning hierarchical representations for face verification with convolutional deep belief networks,” in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012, pp. 2518–2525. find
  22. K. Chen, Z. Zhang, J. Long, and H. Zhang, “Turning from tf-idf to tf-igm for term weighting in text classification,” Expert Systems with Applications, vol. 66, pp. 245–260, 2016. find
  23. W. Zhang, T. Yoshida, and X. Tang, “Text classification based on multiword with support vector machine,” Knowledge-Based Systems, vol. 21, no. 8, pp. 879–886, 2008. find
  24. V. Vapnik and A. Y. Chervonenkis, “A class of algorithms for pattern recognition learning,” Avtomat. i Telemekh, vol. 25, no. 6, pp. 937–945, 1964. find
  25. A. Y. Chervonenkis, “Early history of support vector machines,” in Empirical Inference. Springer, 2013, pp. 13–20. find
  26. B. E. Boser, I. M. Guyon, and V. N. Vapnik, “A training algorithm for optimal margin classifiers,” in Proceedings of the fifth annual workshop on Computational learning theory. ACM, 1992, pp. 144–152. find
  27. R. Singh, A. Sekhon, K. Kowsari, J. Lanchantin, B. Wang, and Y. Qi, “Gakco: a fast gapped k-mer string kernel using counting,” 2017. find
  28. J. Weston and C. Watkins, “Multi-class support vector machines,” Technical Report CSD-TR-98-04, Department of Computer Science, Royal Holloway, University of London, May, Tech. Rep., 1998. find
  29. A. Sun and E.-P. Lim, “Hierarchical text classification and evaluation,” in Data Mining, 2001. ICDM 2001, Proceedings IEEE International Conference on. IEEE, 2001, pp. 521–528. find
  30. F. Sebastiani, “Machine learning in automated text categorization,” ACM computing surveys (CSUR), vol. 34, no. 1, pp. 1–47, 2002.
  31. D. D. Lewis, “Naive (bayes) at forty: The independence assumption in information retrieval,” in European conference on machine learning. Springer, 1998, pp. 4–15. find
  32. J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation.” in EMNLP, vol. 14, 2014, pp. 1532–1543.
  33. W. B. Cavnar, J. M. Trenkle et al., “N-gram-based text categorization,” Ann Arbor MI, vol. 48113, no. 2, pp. 161–175, 1994. find
  34. D. Glickman, “Comparing input words to a word dictionary for correct spelling,” Feb. 5 1985, uS Patent 4,498,148. find
  35. V. Kešelj, F. Peng, N. Cercone, and C. Thomas, “N-gram-based author profiles for authorship attribution,” in Proceedings of the conference pacific association for computational linguistics, PACLING, vol. 3, 2003, pp. 255–264. find
  36. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. find
  37. J. Lanchantin, R. Singh, B. Wang, and Y. Qi, “Deep motif dashboard: Visualizing and understanding genomic sequences using deep neural networks,” arXiv preprint arXiv:1608.03644, 2016. — [arXiv:1608.03644](https://arxiv.org/abs/1608.03644) find
  38. A. Karpathy, “The unreasonable effectiveness of recurrent neural networks,” http://karpathy.github.io/2015/05/21/rnn-effectiveness/, May 2015. ↗
  39. Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE transactions on neural networks, vol. 5, no. 2, pp. 157–166, 1994. find
  40. R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training recurrent neural networks.” ICML (3), vol. 28, pp. 1310–1318, 2013. find
  41. S. Lai, L. Xu, K. Liu, and J. Zhao, “Recurrent convolutional neural networks for text classification.” in AAAI, vol. 333, 2015, pp. 2267– 2273. find
  42. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, 2014.
  43. D. Scherer, A. Müller, and S. Behnke, “Evaluation of pooling operations in convolutional architectures for object recognition,” Artificial Neural Networks–ICANN 2010, pp. 92–101, 2010. find
  44. D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  45. O. Poungpair, A. Pootong, P. Tongtawe, T. Songserm, P. Tapchaisri, and W. Chaicumpa, “Web of science R ,” COMMUNICATIONS, 2013. find
  46. T. Reuters, “Web of science.” 2012. find
  47. S. Munzert, C. Rubba, P. Meißner, and D. Nyhuis, Automated data collection with R: A practical guide to web scraping and text mining. John Wiley & Sons, 2014. find
  48. U. Gundecha, Selenium Testing Tools Cookbook. Packt Publishing Ltd, 2015. find
  49. L. Richardson, “Beautiful soup,” https://www.crummy.com/software/BeautifulSoup/, May 2017. ↗
  50. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin et al., “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” arXiv preprint arXiv:1603.04467, 2016.
  51. F. Chollet et al., “Keras: Deep learning library for theano and tensorflow,” https://keras.io/ , 2015. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kowsari-2017-hdltex-hierarchical-deep-learning-text/