Elamur Research⌕
☰
home / papers / 2015 / Short Text Clustering via Convolutional Neural Networks

Short Text Clustering via Convolutional Neural Networks

Paper · 2015 · Jiaming Xu, Peng Wang, Guanhua Tian et al.

Summary

Clustering short texts such as search snippets or question titles is difficult because their bag-of-words vectors are extremely sparse. TF-IDF and other shallow representations therefore capture little semantic similarity. The authors propose STCC, an unsupervised method that trains a convolutional network without any labels through a self-taught scheme. First, keyword features are compressed into compact binary codes. These codes come from Laplacian Eigenmaps over a heat-kernel k-nearest-neighbour graph, thresholded at the median, so that texts close in the original space stay close. Next, a Dynamic Convolutional Neural Network (DCNN) reads word2vec embeddings and is trained to predict those binary codes. Its hidden layer serves as a dense text representation, which K-means then groups into clusters. The method was tested on SearchSnippets and a StackOverflow title sample. It beat K-means on TF/TF-IDF, spectral clustering and averaged embeddings on both accuracy and NMI, with the largest gain on StackOverflow. The work shows that CNNs, usually applied to supervised tasks, can learn useful features for clustering without annotation or heavy NLP preprocessing.

Key points

  • Datasets: SearchSnippets has 12,340 texts in 8 domains and a vocabulary of 30,642. The StackOverflow sample has 20,000 question titles across 20 tags, drawn from Kaggle challenge data. Stop words and symbols were kept in both.
  • Embeddings: word2vec vectors use 48 dimensions and a minimum count of 5. They were trained on Wikipedia for SearchSnippets and on the full StackOverflow corpus for StackOverflow. These vectors cover 95% and 97% of tokens respectively.
  • Training: the network has 48-dimensional embeddings feeding wide 1-D convolution, folding and dynamic k-max pooling. Logistic outputs fit the pre-trained binary codes. Training uses Adagrad with mini-batches and 50% dropout.
  • Results (ACC/NMI, %): STCC reaches 77.09/63.16 on SearchSnippets, against best baseline scores of 74.76 ACC (Spectral Clustering, tuned dimension) and 58.30 NMI. On StackOverflow it scores 51.13/49.03, compared with roughly 37.2/38.4 for the best baselines (Average Embedding with TF).
  • Visualisation: t-SNE plots show STCC features forming more clearly separated topic groups than the baseline representations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🔬Institute of AutomationBeijing · China87 works · 14,883 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang and Hongwei Hao. 2015. Short Text Clustering via Convolutional Neural Networks. NAACL 2015 Workshop on Vector Space Modeling for NLP.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{xu2015short,
  title = {{Short Text Clustering via Convolutional Neural Networks}},
  author = {Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang and Hongwei Hao},
  booktitle = {NAACL 2015 Workshop on Vector Space Modeling for NLP},
  year = {2015},
  doi = {10.3115/v1/w15-1509},
  url = {https://doi.org/10.3115/v1/w15-1509}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Charu C Aggarwal and ChengXiang Zhai. 2012. A survey of text clustering algorithms. In Mining Text Data, pages 77–128. Springer. find
  2. Somnath Banerjee, Krishnan Ramanathan, and Ajay Gupta. 2007. Clustering short texts using wikipedia. In SIGIR, pages 787–788. ACM. find
  3. Mikhail Belkin and Partha Niyogi. 2001. Laplacian eigenmaps and spectral techniques for embedding and clustering. In NIPS, volume 14, pages 585–591. find
  4. Phil Blunsom, Edward Grefenstette, Nal Kalchbrenner, et al. 2014. A convolutional neural network for modelling sentences. In ACL.
  5. Deng Cai, Xiaofei He, and Jiawei Han. 2005. Document clustering using locality preserving indexing. Knowledge and Data Engineering, IEEE Transactions on, 17(12):1624–1637. find
  6. Wen-Yen Chen, Yangqiu Song, Hongjie Bai, Chih-Jen Lin, and Edward Y Chang. 2011. Parallel spectral clustering in distributed systems. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(3):568–586. find
  7. (b). Spectral Clustering (best) business computers culture education engineering health politics sports (d). STCC Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JMLR, 12:2493–2537.
  8. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121–2159. find
  9. Samah Fodeh, Bill Punch, and Pang-Ning Tan. 2011. On ontology-driven document clustering using core semantic features. Knowledge and information systems, 28(2):395–421. find
  10. Geoffrey E Hinton and Ruslan R Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507. find
  11. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In ACL, pages 873–882. Association for Computational Linguistics.
  12. Peihao Huang, Yan Huang, Wei Wang, and Liang Wang. 2014. Deep embedding network for clustering. In Pattern Recognition (ICPR), 2014 22nd International Conference on, pages 1532–1537. IEEE. find
  13. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.
  14. Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In AAAI. find
  15. (a). K−means (TF−IDF) (c). Average Embedding (TF) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2015-short-text-clustering-convolutional-neural/