Elamur Research⌕
☰
home / papers / 2024 / Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization

Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization

Paper · 2024 · Xuxi Chen, Zhendong Wang, Daouda Sow et al.

Summary

High-quality text for pretraining large language models is becoming scarce. This paper asks whether an already trained LLM can be improved cheaply by briefly continuing training on its own original data, without adding new sources. The authors first checked per-sample losses on C4 with LLaMA-2-7B. The very highest-loss samples were mostly noise or hard-to-model material, such as advertisements and bibliography fragments. Samples with moderately high loss tended to be informative. From this they built MidRanking, a heuristic that skips the top-loss items and trains on the next tier. Because MidRanking needs model-specific tuning, they also propose IR-DRO (Instance-Reweighted Distributionally Robust Optimization). IR-DRO is a KL-regularized min-max formulation with a closed-form weighting rule, so it slots into ordinary training loops. They tested both methods on OPT, Sheared-LLaMA and LLaMA, in continual pretraining and in instruction tuning. IR-DRO generally beat uniform training and the loss-ranking heuristics, though the gains are small. The work matters because it extracts extra value from existing corpora at low compute cost.

Key points

  • On C4, the samples with the largest loss under LLaMA-2-7B were dominated by noise and inherently ambiguous text. Training only on them can hurt performance, which motivated mid-range loss selection (MidRanking).
  • IR-DRO assigns each sample a weight inside the batch using a closed-form solution. This avoids an extra forward-backward pass and the auxiliary models used by methods such as DoReMi.
  • In continual instruction tuning of LLaMA-v2-7B (100 extra batches on Open-Platypus), the benchmark average rose from 63.22 to 63.68 with IR-DRO. This average covers ARC-C, HellaSwag, PiQA, WinoGrande, BoolQ and MMLU.
  • In standard instruction tuning, IR-DRO's average was slightly above the uniform baseline: 63.29 vs 63.20 on Alpaca and 63.36 vs 63.22 on Open-Platypus. On Open-Platypus, the ranking-based selectors fell below the baseline.
  • Ablations on OPT-350M showed that the best amount of continual training varies by benchmark: about 500 batches for BoolQ versus 100 for WinoGrande. Code is released at github.com/VITA-Group/HardFocusTraining.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Xuxi Chen, Zhendong Wang, Daouda Sow, Junjie Yang, Tianlong Chen, Yingbin Liang, Mingyuan Zhou and Zhangyang Wang. 2024. Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization. arXiv:2402.14270.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.14270 · arXiv 2402.14270

@misc{chen2024take,
  title = {{Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization}},
  author = {Xuxi Chen, Zhendong Wang, Daouda Sow, Junjie Yang, Tianlong Chen, Yingbin Liang, Mingyuan Zhou and Zhangyang Wang},
  year = {2024},
  doi = {10.48550/arxiv.2402.14270},
  eprint = {2402.14270},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.14270}
}

References

47 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. find
  3. Attendu, J.-M. and Corbeil, J.-P. Nlu on data diets: Dynamic data subset selection for nlp classification tasks. arXiv preprint arXiv:2306.03208, 2023. find
  4. Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. find
  5. Belgodere, B., Dognin, P., Ivankay, A., Melnyk, I., Mroueh, Y., Mojsilovic, A., Navartil, J., Nitsure, A., Padhi, I., Rigotti, M., et al. Auditing and generating synthetic data with controllable trust trade-offs. arXiv preprint arXiv:2304.10819, 2023. — [arXiv:2304.10819](https://arxiv.org/abs/2304.10819) find
  6. He, R., Sun, S., Yu, X., Xue, C., Zhang, W., Torr, P., Bai, S., and Qi, X. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574, 2022. — [arXiv:2210.07574](https://arxiv.org/abs/2210.07574) find
  7. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41–48, 2009.
  8. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  9. Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432–7439, 2020.
  10. Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al. Phi-2: The surprising power of small language models, 2023. find
  11. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020.
  12. Jiang, A. H., Wong, D. L.-K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G., Kaminksy, M., Kozuch, M., Lipton, Z. C., et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.00762, 2019. — [arXiv:1910.00762](https://arxiv.org/abs/1910.00762) find
  13. Chen, M. F., Roberts, N., Bhatia, K., Wang, J., Zhang, C., Sala, F., and Ré, C. Skill-it! a data-driven skills framework for understanding and training language models. arXiv preprint arXiv:2307.14430, 2023. — [arXiv:2307.14430](https://arxiv.org/abs/2307.14430) find
  14. Katharopoulos, A. and Fleuret, F. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525– 2534. PMLR, 2018. find
  15. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019.
  16. Killamsetty, K., Durga, S., Ramakrishnan, G., De, A., and Iyer, R. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464–5474. PMLR, 2021a. find
  17. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., and Iyer, R. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Con… find
  18. Qian, Q., Zhu, S., Tang, J., Jin, R., Sun, B., and Li, H. Robust optimization over multiple domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4739–4746, 2019. find
  19. Qiu, Z.-H., Hu, Q., Yuan, Z., Zhou, D., Zhang, L., and Yang, T. Not all semantics are created equal: Contrastive self-supervised learning with automatic temperature individualization. arXiv preprint arXiv:2305.11965, 2023. — [arXiv:2305.11965](https://arxiv.org/abs/2305.11965) find
  20. Lee, A. N., Hunter, C. J., and Ruiz, N. Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprint arXiv:2308.07317, 2023.
  21. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  22. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023.
  23. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  24. Loshchilov, I. and Hutter, F. Online batch selection for faster training of neural networks. arXiv preprint arXiv:1511.06343, 2015. — [arXiv:1511.06343](https://arxiv.org/abs/1511.06343) find
  25. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. find
  26. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. find
  27. Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadaee, M., and Hooker, S. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564, 2023. find
  28. Sankaranarayanan, S., Balaji, Y., Jain, A., Lim, S. N., and Chellappa, R. Learning from synthetic data: Addressing domain shift for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3752–3761, 2018. find
  29. Mirzasoleiman, B., Bilmes, J., and Leskovec, J. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950–6960. PMLR, 2020. find
  30. Sener, O. and Savarese, S. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017. find
  31. Mitra, A., Del Corro, L., Mahajan, S., Codas, A., Simoes, C., Agarwal, S., Chen, X., Razdaibiedina, A., Jones, E., Aggarwal, K., et al. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.11045, 2023.
  32. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford_alpaca, 2023. ↗
  33. Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Barnes, N., and Mian, A. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435, 2023.
  34. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  35. Qi, Q., Guo, Z., Xu, Y., Jin, R., and Yang, T. An online method for a class of distributionally robust optimization with non-convex objectives. Advances in Neural Information Processing Systems, 34:10067–10080, 2021. find
  36. Trinh, T. H. and Le, Q. V. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847, 2018. find
  37. Qi, Q., Lyu, J., Bai, E. W., Yang, T., et al. Stochastic constrained dro with a complexity independent of sample size. arXiv preprint arXiv:2210.05740, 2022. — [arXiv:2210.05740](https://arxiv.org/abs/2210.05740) find
  38. Villalobos, P., Sevilla, J., Heim, L., Besiroglu, T., Hobbhahn, M., and Ho, A. Will we run out of data? an analysis of the limits of scaling datasets in machine learning. arXiv preprint arXiv:2211.04325, 2022.
  39. Qi, Q., Xu, Y., Yin, W., Jin, R., and Yang, T. Attentional-biased stochastic gradient descent. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum? id=B0WYWvVA2r. ↗
  40. Wang, J., Zhang, T., Liu, S., Chen, P.-Y., Xu, J., Fardad, M., and Li, B. Adversarial attack generation empowered by min-max optimization. Advances in Neural Information Processing Systems, 34:16020–16033, 2021. find
  41. Wang, M., Fang, E. X., and Liu, H. Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions. Mathematical Programming, 161:419–449, 2017. find
  42. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
  43. Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023. find
  44. Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre-training. arXiv preprint arXiv:2305.10429, 2023. find
  45. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. find
  46. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  47. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19–27, 2015.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2024-take-bull-horns-hard-sample/