📄 LABR: A Large Scale Arabic Book Reviews Dataset

Summary

Arabic sentiment analysis has had very few corpora, and the existing ones are small. The authors address this with LABR, which they describe as the largest Arabic sentiment dataset at the time of publication. They crawled more than 220,000 reviews from Goodreads in March 2013, taken from books on the "Best Arabic Books" list. Most of these turned out not to be in Arabic. After removing them and stripping HTML and other noise, 63,257 Arabic reviews remained, each rated from 1 to 5 stars. The paper describes the dataset's statistics and releases fixed 80/20 train/test splits for two tasks: polarity classification and 5-way rating prediction. Each task has a balanced split (classes down-sampled to the smallest class) and an unbalanced split. Baselines use n-gram features, optionally tf-idf weighted, with Naive Bayes variants and SVMs. Polarity reaches about 90% accuracy, while rating prediction stays near 50%, so the finer-grained task remains open. The resource matters because shared splits allow later Arabic sentiment work to be compared directly.

Key points

• LABR has 63,257 reviews from 16,486 users on 2,131 books, totalling about 4.1M tokens and 342K rough sentences.
• Per-review averages are 65.4 tokens and 5.4 sentences.
• Per user, the mean is 3.84 reviews and the median 2; per book, the mean is about 30 and the median 6.
• Ratings 4–5 count as positive and 1–2 as negative; 3-star reviews are treated as neutral and left out of the polarity task.
• The label distribution is heavily skewed toward positive reviews, which the authors attribute to the list containing popular books.
• Splits are 8:2 train/test, in balanced and unbalanced versions for both tasks; cross-validation was considered unnecessary given the data size.
• Classifiers were Multinomial NB, Bernoulli NB and SVM, built with scikit-learn and Qalsadi. Features were unigrams up to trigrams, with or without tf-idf.
• Best polarity result: SVM with tf-idf n-grams on the unbalanced split, 0.910 accuracy / 0.901 weighted F1.
• Best rating result: the same setup, 0.503 accuracy / 0.491 weighted F1.
• On the balanced splits, MNB slightly beats SVM. Bernoulli NB combined with tf-idf collapses to near-chance scores.

Interesting ideas

• I1. Publish each benchmark with both a class-balanced split (built by down-sampling) and a natural-distribution split, because a skewed label prior can inflate apparent accuracy.
• I2. Use the star scale as a free source of labels, and drop the ambiguous middle value instead of forcing it into one side.
• I3. When harvesting web data, expect most of it to be in the wrong language and filter by language before computing any statistics; here over 70% of the crawl was discarded.
• I4. Report accuracy and weighted F1 side by side; where they diverge sharply (as with BNB plus tf-idf), the gap reveals a degenerate classifier.
• I5. Describe a new resource through distributions (reviews per user and per book, review lengths, a Zipf check on the vocabulary), not just total counts.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluation of speech-act or polarity readers could report both views, so gains that merely ride the majority class become visible.
• I2 (frames): When collecting silver labels for evaluative stance in prag, mid-scale or uncertain items can be withheld rather than counted as clean positive or negative examples.
• I3 (tokens): Corpora gathered for the en tokenizer and tagger should pass a language filter first, so that foreign text does not pollute lemma and morphology statistics.
• I4 (learn): Pairing two metrics can act as a cheap negative control, flagging a component that has collapsed to predicting one class.
• I5 (world): Profiling fable and tale corpora the same way, for example by entity counts per story and story lengths, would show coverage gaps before world-building rules are tuned.