📄 Annotation as Cultural Interpretation: Rethinking Data Labeling in NLP

Summary

The paper challenges the standard NLP assumption that annotation recovers an objective ground truth and that annotator disagreement is mainly noise. It is a conceptual position paper, published at C3NLP 2026. It draws on sociolinguistics and linguistic anthropology (Hymes; Duranti and Goodwin) and on empirical studies of annotator subjectivity to argue that labeling is a culturally situated act of interpretation. Annotators read texts through their own norms, identities and communicative expectations, so dataset labels capture particular interpretive frameworks rather than universal facts. The author ties this view to perspectivism, crowd truth and annotator-aware modeling. Published cross-cultural findings, such as Davani et al. (2024) on offensiveness, are used as evidence. The paper then proposes an agenda covering:

• documentation of annotator context,
• preservation of disagreement,
• culturally informed guidelines,
• diverse and participatory recruitment,
• cross-cultural evaluation metrics.

No new experiments are reported. The argument matters most for safety-critical, socially loaded tasks like toxicity and hate speech detection. In these tasks, unexamined cultural assumptions in the labels can be absorbed by models and reproduced as bias.

Key points

• The paper separates tasks by "cultural interpretive load":
  • High load: hate speech, offensiveness, sarcasm, stance, politeness and emotion labeling.
  • Low load: POS tagging, NER and morphological analysis, which are more constrained by linguistic form. Even these show some genuine ambiguity (Plank et al., 2014).
• It names four dimensions that make annotation interpretive: normative evaluation, pragmatic inference, social positioning and interpretive variability.
• Cited evidence includes:
  • Sap et al. (2019): hate speech systems misclassify African American English as offensive.
  • Waseem (2016): annotator awareness of inequality shifts hate speech labels.
  • Davani et al. (2022): multi-annotator models match or beat majority-vote baselines.
• Proposed cross-cultural robustness metrics:
  • performance reported per annotator subgroup,
  • agreement measured within versus across cultural groups,
  • degradation tracked as a function of cultural distance.
• Stated limitations:
  • no new annotation experiments,
  • evidence skewed toward English,
  • unanalyzed costs and privacy trade-offs,
  • no criterion for when cultural coverage is sufficient.