Elamur Research⌕
☰
home / papers / 2017 / Does William Shakespeare REALLY Write Hamlet? Knowledge Representation Learning with Confidence

Does William Shakespeare REALLY Write Hamlet? Knowledge Representation Learning with Confidence

Paper · 2017 · Ruobing Xie, Zhiyuan Liu, Fen Lin et al.

Summary

Large knowledge graphs such as Freebase are increasingly filled by automatic extraction and crowdsourcing, so they contain wrong or conflicting facts. Standard embedding methods treat every triple as equally true. This paper proposes CKRL, a confidence-aware extension of TransE. Each training triple gets a confidence score, and that score scales its contribution to the margin-based loss. Confidences are recomputed during training and use only the graph's own structure, with no external text. There are three kinds: a local score based on how well the triple fits the translation assumption, and two global scores derived from relation paths linking the head and tail. The authors build noisy versions of FB15K (FB15K-N1/N2/N3, with increasing noise rates). They evaluate on noise detection, entity prediction and triple classification. CKRL beats TransE on every task, and the margin grows as noise increases. The result matters because it gives a cheap way to keep downstream models robust when a graph cannot be fully verified by people.

Key points

  • The loss is TransE's margin ranking loss, with each positive–negative pair multiplied by the confidence C(h, r, t) of the positive triple. Low-confidence facts therefore barely shape the embeddings.
  • Three variants are reported: LT, LT+PP and LT+PP+AP, which add path-based global confidence to the local triple confidence.
  • Entity prediction, filtered Hits@10:
    • TransE scores 59.8 / 56.3 / 51.8 on N1 / N2 / N3.
    • CKRL (LT) scores 61.8 / 59.5 / 56.9.
    • TransE trained on clean FB15K reaches 64.7, which serves as an upper bound.
  • Triple classification uses a per-relation threshold δr tuned on validation data. TransE reaches 81.3 / 79.4 / 76.9, while CKRL reaches up to 81.9 / 80.2 / 78.4. The gains are smaller than for noise detection, since test negatives are not confidence-scored.
  • Path confidence adds little to entity prediction. Weighting paths more heavily helps completion but hurts noise detection. Code and data are at github.com/thunlp/CKRL.

Interesting ideas

  • I1. Multiplying each example's loss by a learned, periodically refreshed trust score lets a model shrink the influence of suspect data without hard-deleting it. This is a soft alternative to filtering.
  • I2. Judging a fact only by how consistent it is with the rest of the graph means no external evidence source is needed. That makes the check usable on any graph.
  • I3. Treating alternative multi-hop paths between two entities as independent support for a direct link is a useful corroboration signal. The paper also shows that this signal is noisy on its own.
  • I4. Benchmarks are built by injecting known noise at several rates into a clean dataset, with the clean-data score reported as a ceiling. This allows controlled measurement of robustness.
  • I5. Per-relation decision thresholds, tuned on validation data, replace one global cutoff because relations differ in how tightly their scores cluster.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver trees agreed by witness parsers could carry a per-sentence or per-arc confidence that weights the parser's training loss, instead of a binary keep/drop.
  • I2 (world): Facts compiled from Wikidata/WordNet into the knowledge graph could be flagged as doubtful when they fit poorly with their neighbours, before readers rely on them.
  • I3 (world): When the world of a text asserts a holding or relation, agreement with a chain of other stated facts could raise its confidence, and that chain doubles as an explanation.
  • I4 (learn): Evaluation could deliberately corrupt a fraction of silver labels or knowledge facts and check that training degrades gracefully, as a negative control.
  • I5 (answer): Readers deciding whether a statement holds could calibrate a separate acceptance threshold per frame or relation type rather than a single score cutoff.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🏢TencentShenzhen · China66 works · 17,945 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Knowledge graphs & ontologies · 103 works

Cite

Cite as Ruobing Xie, Zhiyuan Liu, Fen Lin and Leyu Lin. 2017. Does William Shakespeare REALLY Write Hamlet? Knowledge Representation Learning with Confidence. arXiv:1705.03202 · AAAI-2018.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1705.03202 · arXiv 1705.03202

@misc{xie2017does,
  title = {{Does William Shakespeare REALLY Write Hamlet? Knowledge Representation Learning with Confidence}},
  author = {Ruobing Xie, Zhiyuan Liu, Fen Lin and Leyu Lin},
  year = {2017},
  doi = {10.48550/arxiv.1705.03202},
  eprint = {1705.03202},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.03202}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Auer, S.; Bizer, C.; Kobilarov, G.; Lehmann, J.; Cyganiak, R.; and Ives, Z. 2007. Dbpedia: A nucleus for a web of open data. In The semantic web. 722–735. find
  2. Bollacker, K.; Evans, C.; Paritosh, P.; Sturge, T.; and Taylor, J. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of KDD, 1247–1250. find
  3. Bordes, A.; Glorot, X.; Weston, J.; and Bengio, Y. 2012. Joint learning of words and meaning representations for open-text semantic parsing. In Proceedings of AISTATS, 127–135. find
  4. Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; and Yakhnenko, O. 2013. Translating embeddings for modeling multi-relational data. In Proceedings of NIPS, 2787–2795. find
  5. De Meo, P.; Ferrara, E.; Fiumara, G.; and Ricciardello, A. 2012. A novel measure of edge centrality in social networks. Knowledge-based systems 30:136– 150. ↗
  6. Dong, X.; Gabrilovich, E.; Heitz, G.; Horn, W.; Lao, N.; Murphy, K.; Strohmann, T.; Sun, S.; and Zhang, W. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In Proceedings of KDD, 601–610. [Gyöngyi, Garcia-Molina, and Pedersen 2004] Gyöngyi, Z.; Garcia-Molina, H.; and Pedersen, J. 2004. Combating web spam with trustrank. In Proceedings of VLDB, 576–587. find
  7. Heindorf, S.; Potthast, M.; Stein, B.; and Engels, G. 2015. Towards vandalism detection in knowledge bases: Corpus construction and analysis. In Proceedings of SIGIR, 831–834. find
  8. Heindorf, S.; Potthast, M.; Stein, B.; and Engels, G. 2016. Vandalism detection in wikidata. In Proceedings of CIKM, 327–336. find
  9. Hoffart, J.; Suchanek, F. M.; Berberich, K.; and Weikum, G. 2013. Yago2: A spatially and temporally enhanced knowledge base from wikipedia. Artificial Intelligence 194:28–61. ↗
  10. Krompaß, D.; Baier, S.; and Tresp, V. 2015. Type-constrained representation learning in knowledge graphs. In Proceedings of ISWC. 640–655. find
  11. Lehmann, J.; Isele, R.; Jakob, M.; Jentzsch, A.; Kontokostas, D.; Mendes, P. N.; Hellmann, S.; Morsey, M.; Van Kleef, P.; Auer, S.; et al. 2015. Dbpedia– a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web 6(2):167–195. ↗
  12. Lin, Y.; Liu, Z.; Luan, H.; Sun, M.; Rao, S.; and Liu, S. 2015a. Modeling relation paths for representation learning of knowledge bases. In Proceedings of EMNLP, 705–714.
  13. Lin, Y.; Liu, Z.; Sun, M.; Liu, Y.; and Zhu, X. 2015b. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of AAAI. find
  14. Lin, Y.; Shen, S.; Liu, Z.; Luan, H.; and Sun, M. 2016. Neural relation extraction with selective attention over instances. In Proceedings of ACL, volume 1, 2124– 2133. find
  15. Manago, M., and Kodratoff, Y. 1987. Noise and knowledge acquisition. In Proceedings of IJCAI, 348–354. find
  16. Nickel, M.; Murphy, K.; Tresp, V.; and Gabrilovich, E. 2016. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE 104(1):11–33. ↗
  17. Pellissier Tanon, T.; Vrandečić, D.; Schaffert, S.; Steiner, T.; and Pintscher, L. 2016. From freebase to wikidata: The great migration. In Proceedings of WWW, 1419–1428. find
  18. Socher, R.; Chen, D.; Manning, C. D.; and Ng, A. 2013. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of NIPS, 926–934.
  19. Wang, Z.; Zhang, J.; Feng, J.; and Chen, Z. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of AAAI, 1112–1119. find
  20. Xie, R.; Liu, Z.; and Sun, M. 2016. Representation learning of knowledge graphs with hierarchical types. In Proceedings of IJCAI. find
  21. Zhou, T.; Ren, J.; Medo, M.; and Zhang, Y.-C. 2007. Bipartite network projection and personal recommendation. Physical Review E 76(4):046115. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2017-william-shakespeare-really-write-hamlet/