📄 Calibrated to Whom? Persona and Language Effects on Cultural Values in JEV

Summary

The paper asks whose cultural values a decision-only language model expresses. Such models output a probability for each fixed answer option and never write text. The model studied is TypeSafe AI's JEV (version jev-1.13.0). The authors gave it the 24 items of the Values Survey Module 2013 (VSM 2013) under three conditions: no persona, 12 Saudi personas and 12 demographically matched American personas. They varied gender and age, used English and Arabic, and tried eight request formats, for 288,000 answers in total.

Published VSM constants are unknown, so every measure contrasts two of JEV's own profiles, which makes the constants cancel. The Saudi-minus-US gap in Hofstede's scores serves as a reference direction, not as ground truth.

JEV was extremely repeatable. Its persona-free profile leaned toward its American personas. Switching personas from American to Saudi moved answers along the reference direction, but the overall pattern of differences still deviated substantially, mainly on long-term orientation. The Arabic shift was weaker, and this was tied mostly to the wording of the items.

The main lesson is that avoiding text generation does not remove the cultural skews seen in chat models.

Key points

• The design crosses 1,200 prompts with 30 repetitions and 8 request set-ups, giving 288,000 answers. Test–retest agreement was very high (ICC 0.997).
• The reference Saudi–US difference vector h has length 94.3. Projected onto h, JEV's persona shift covered 87% of the reference magnitude with English prompts, compared with 62% with Arabic prompts. The total discrepancy from h stayed around 60% in both languages (62% and 60%), largely because long-term orientation came out reversed.
• A crossed-language analysis attributes the Arabic shrinkage mostly to item language (−29.5 points) rather than persona-description language (−3.9). The biggest drops were on agree–disagree items about hierarchy and rules.
• Age moved profiles roughly as much as nationality did. Gender mattered more for Saudi personas. In the documented set-up, confidence was lower in Arabic and tracked the language of the framing.
• The direction of the national difference held in all eight request designs. Data were collected on 28–29 September 2026, and the authors plan to release prompts, responses and code.

Interesting ideas

• I1. When absolute scales contain unknown offsets, compare only pairs of profiles measured on the same scale, so the offsets cancel and the comparison stays interpretable.
• I2. Report two numbers against a benchmark: how far the system moves along the expected direction, and how far its full difference vector departs from it, where zero shift counts as 100%.
• I3. Cross two factors that usually co-vary, such as the language of the framing and the language of the items, to see which one actually drives an effect.
• I4. Rerun the same study under several deliberately varied request formats (reversed option order, third person, batched items) and keep only conclusions that survive every variant.
• I5. A model's confidence reflects its own certainty, not agreement among people, so low confidence can flag thin evidence but cannot prove calibration to a population.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluations could score paired runs (e.g., parser before/after a repair rule on identical sentences) to cancel corpus-specific baselines that are otherwise hard to normalise.
• I2 (answer): Comparisons of reader variants on school tests could report both the gain in the intended skill and the total deviation pattern, exposing improvements that come with distortions elsewhere.
• I3 (tree): When a parser degrades on a converted treebank, crossing source text against annotation dialect could tell whether errors come from the sentences or from the UD conversion.
• I4 (answer): Multiple-choice and word-problem tests could be re-posed with shuffled options and reworded stems, catching answers that depend on layout rather than understanding.
• I5 (learn): For silver data, confidence of a single parser should be kept separate from agreement among witness parsers, and each used for its own filtering purpose.