Elamur Research⌕
☰
home / papers / 2024 / Interpreting Multi-band Galaxy Observations with Large Language Model-Based Agents

Interpreting Multi-band Galaxy Observations with Large Language Model-Based Agents

Paper · 2024 · Zechang Sun, Yuan-Sen Ting, Yaobo Liang et al.

Summary

Interpreting a galaxy's spectral energy distribution (SED), which is broadband photometry treated as a very coarse spectrum, requires expert judgment. Researchers look for where a model misses the data, adjust its physical components, and consider physical plausibility, not just the lowest χ². The authors built mephisto, a multi-agent framework, mainly powered by GPT-4o, that controls the CIGALE fitting code. The agent receives the photometry as wavelength–flux pairs, the current model setup, and fit diagnostics. It then proposes model changes and explores them with a tree search. It also records results in a temporal memory and keeps an external knowledge base, which it builds by extracting lessons from finished fits and testing them on data. On the JWST JADES DR2 catalog, mephisto reaches similar fit quality in fewer search steps as its experience grows. For the puzzling "Little Red Dots", it finds both interpretations debated in the literature: a dusty star-forming galaxy without an AGN, or a nearly dust-free galaxy with an AGN. The authors describe this as the first example of agentic research in astronomy.

Key points

  • At each step, mephisto generates N_b = 4 candidate SED model variants. It expands them with DFS or BFS tree search and ranks child nodes by cumulative residuals. A temporal memory prevents it from repeating attempts.
  • A distillation agent and a validation agent extract insights from past fits and score them. The verified knowledge is then reused as context, so the system learns from self-play without an explicit reward function.
  • On 32 sources sampled uniformly from a Self-Organizing Map (SOM) projection of JADES, median χ² improves with both inference depth and run depth (0, 5, 10 prior fits).
  • For 31 Little Red Dots from Pablo et al. 2024, mephisto proposes two scenarios. For example, for LRD 79,803 (z_spec = 5.4007) it gives either A_v ≈ 0.03 mag with frac_AGN = 0.99, or A_v ≈ 0.92 mag, log M∗/M⊙ ≈ 9.54, and no AGN.
  • Comparing LLM backbones, LLaMA-3.1-405B performs worse than GPT-4o, and GLM-4-0520 is roughly on par with LLaMA-3.1-405B. The authors suggest mephisto could serve as a benchmark of LLM reasoning in astronomy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China439 works · 94,783 citations🏢Microsoft Research AsiaBeijing · China86 works · 61,617 citations🎓The Ohio State UniversityColumbus · United States96 works · 19,285 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Engineering · 206 worksPhysics and Astronomy · 70 worksComputational Mechanics · 8 worksAstronomy and Astrophysics · 4 worksInstrumentation · 2 works

Cite

Cite as Zechang Sun, Yuan-Sen Ting, Yaobo Liang, Nan Duan, Song Huang and Zheng Cai. 2024. Interpreting Multi-band Galaxy Observations with Large Language Model-Based Agents. arXiv:2409.14807.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.14807 · arXiv 2409.14807

@misc{sun2024interpreting,
  title = {{Interpreting Multi-band Galaxy Observations with Large Language Model-Based Agents}},
  author = {Zechang Sun, Yuan-Sen Ting, Yaobo Liang, Nan Duan, Song Huang and Zheng Cai},
  year = {2024},
  doi = {10.48550/arxiv.2409.14807},
  eprint = {2409.14807},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.14807}
}

References

75 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. G. York, J. Adelman, J. Anderson, John E., S. F. Anderson, J. Annis, and et al., “The Sloan Digital Sky Survey: Technical Summary,” The Astronomical Journal, vol. 120, pp. 1579–1587, Sept. 2000. find
  2. R. S. de Jong, O. Agertz, A. A. Berbel, J. Aird, D. A. Alexander, and et al., “4MOST: Project overview and information for the First Call for Proposals,” The Messenger, vol. 175, pp. 3–11, Mar. 2019. find
  3. DESI Collaboration, A. G. Adame, J. Aguilar, S. Ahlen, S. Alam, and et al., “The Early Data Release of the Dark Energy Spectroscopic Instrument,” The Astronomical Journal, vol. 168, p. 58, Aug. 2024. find
  4. Ž. Ivezić, S. M. Kahn, J. A. Tyson, B. Abel, E. Acosta, and et al., “LSST: From Science Drivers to Reference Design and Anticipated Data Products,” The Astrophysical Journal, vol. 873, p. 111, Mar. 2019. find
  5. Euclid Collaboration, R. Scaramella, J. Amiaux, Y. Mellier, C. Burigana, and et al., “Euclid preparation. I. The Euclid Wide Survey,” Astronomy and Astrophysics, vol. 662, p. A112, June 2022. find
  6. H. Aihara, Y. AlSayyad, M. Ando, R. Armstrong, J. Bosch, and et al., “Third data release of the Hyper Suprime-Cam Subaru Strategic Program,” Publications of the Astronomical Society of Japan, vol. 74, pp. 247–272, Apr. 2022. find
  7. B. Wang, J. Zou, Z. Cai, J. X. Prochaska, Z. Sun, and et al., “Deep Learning of Dark Energy Spectroscopic Instrument Mock Spectra to Find Damped Lyα Systems,” The Astrophysical Journal Supplement Series, vol. 259, p. 28, Mar. 2022. find
  8. G. Stein, J. Blaum, P. Harrington, T. Medan, and Z. Lukić, “Mining for Strong Gravitational Lenses with Self-supervised Learning,” The Astrophysical Journal, vol. 932, p. 107, June 2022. find
  9. B. E. Robertson, S. Tacchella, B. D. Johnson, R. Hausen, A. B. Alabi, K. Boyett, A. J. Bunker, S. Carniani, E. Egami, D. J. Eisenstein, K. N. Hainline, J. M. Helton, Z. Ji, N. Kumari, J. Lyu, R. Maiolino, E. J. Nelson, M. J. Rieke, I. Shivaei, F. Sun, H. Übler, C. C. Williams, C. N. A. Willmer, and… find
  10. J. Chardin, G. Uhlrich, D. Aubert, N. Deparis, N. Gillet, and et al., “A deep learning model to emulate simulations of cosmic reionization,” Monthly Notices of the Royal Astronomical Society, vol. 490, pp. 1055–1065, Nov. 2019. find
  11. S. He, Y. Li, Y. Feng, S. Ho, S. Ravanbakhsh, and et al., “Learning to predict the cosmological structure formation,” Proceedings of the National Academy of Science, vol. 116, pp. 13825– 13832, July 2019. find
  12. Y. Li, Y. Ni, R. A. C. Croft, T. Di Matteo, S. Bird, and et al., “AI-assisted superresolution cosmological simulations,” Proceedings of the National Academy of Science, vol. 118, p. e2022038118, May 2021. find
  13. K. Cranmer, J. Brehmer, and G. Louppe, “The frontier of simulation-based inference,” Proceedings of the National Academy of Science, vol. 117, pp. 30055–30062, Dec. 2020. find
  14. X. Zhao, Y. Mao, C. Cheng, and B. D. Wandelt, “Simulation-based Inference of Reionization Parameters from 3D Tomographic 21 cm Light-cone Images,” The Astrophysical Journal, vol. 926, p. 151, Feb. 2022. find
  15. Z. Sun, Y.-S. Ting, and Z. Cai, “Quasar Factor Analysis-An Unsupervised and Probabilistic Quasar Continuum Prediction Algorithm with Latent Factor Analysis,” The Astrophysical Journal Supplement Series, vol. 269, p. 4, Nov. 2023. find
  16. A. A. Abdo, M. Ackermann, M. Arimoto, K. Asano, W. B. Atwood, and et al., “Fermi Observations of High-Energy Gamma-Ray Emission from GRB 080916C,” Science, vol. 323, p. 1688, Mar. 2009. find
  17. M. Gillon, E. Jehin, S. M. Lederer, L. Delrez, J. de Wit, and et al., “Temperate Earth-sized planets transiting a nearby ultracool dwarf star,” Nature, vol. 533, pp. 221–224, May 2016. find
  18. L. G. Spitler, P. Scholz, J. W. T. Hessels, S. Bogdanov, A. Brazier, and et al., “A repeating fast radio burst,” Nature, vol. 531, pp. 202–205, Mar. 2016. find
  19. F. Wang, J. Yang, X. Fan, J. F. Hennawi, A. J. Barth, and et al., “A Luminous Quasar at Redshift 7.642,” The Astrophysical Journal, vol. 907, p. L1, Jan. 2021. find
  20. B. Welch, D. Coe, J. M. Diego, A. Zitrin, E. Zackrisson, and et al., “A highly magnified star at redshift 6.2,” Nature, vol. 603, pp. 815–818, Mar. 2022. find
  21. R. Maiolino, J. Scholtz, J. Witstok, S. Carniani, F. D’Eugenio, and et al., “A small and vigorous black hole in the early Universe,” Nature, vol. 627, pp. 59–63, Mar. 2024. find
  22. N. Yoshikawa, M. Skreta, K. Darvish, S. Arellano-Rubach, Z. Ji, L. Bjørn Kristensen, A. Z. Li, Y. Zhao, H. Xu, A. Kuramshin, A. Aspuru-Guzik, F. Shkurti, and A. Garg, “Large language models for chemistry robotics,” Autonomous Robots, vol. 47, no. 8, pp. 1057–1086, 2023. find
  23. G. Lei, R. Docherty, and S. J. Cooper, “Materials science in the era of large language models: a perspective,” arXiv e-prints, p. arXiv:2403.06949, Mar. 2024. — [arXiv:2403.06949](https://arxiv.org/abs/2403.06949) find
  24. K. M. Jablonka, P. Schwaller, A. Ortega-Guerrero, and B. Smit, “Leveraging large language models for predictive chemistry,” Nature Machine Intelligence, vol. 6, no. 2, pp. 161–169, 2024.
  25. C. M. Casey, J. S. Kartaltepe, N. E. Drakos, M. Franco, S. Harish, and et al., “COSMOS-Web: An Overview of the JWST Cosmic Origins Survey,” The Astrophysical Journal, vol. 954, p. 31, Sept. 2023. find
  26. D. J. Eisenstein, C. Willott, S. Alberts, S. Arribas, N. Bonaventura, and et al., “Overview of the JWST Advanced Deep Extragalactic Survey (JADES),” arXiv e-prints, p. arXiv:2306.02465, June 2023. — [arXiv:2306.02465](https://arxiv.org/abs/2306.02465) find
  27. J. R. Weaver, S. E. Cutler, R. Pan, K. E. Whitaker, I. Labbé, and et al., “The UNCOVER Survey: A First-look HST + JWST Catalog of 60,000 Galaxies near A2744 and beyond,” The Astrophysical Journal Supplement Series, vol. 270, p. 7, Jan. 2024. find
  28. P. G. Pérez-González, G. Barro, G. H. Rieke, J. Lyu, M. Rieke, and et al., “What Is the Nature of Little Red Dots and what Is Not, MIRI SMILES Edition,” The Astrophysical Journal, vol. 968, p. 4, June 2024. find
  29. J. Matthee, R. P. Naidu, G. Brammer, J. Chisholm, A.-C. Eilers, and et al., “Little Red Dots: An Abundant Population of Faint Active Galactic Nuclei at z ∼ 5 Revealed by the EIGER and FRESCO JWST Surveys,” The Astrophysical Journal, vol. 963, p. 129, Mar. 2024. find
  30. F. Gentile, C. M. Casey, H. B. Akins, M. Franco, J. McKinney, E. Berman, O. R. Cooper, N. E. Drakos, M. Hirschmann, A. S. Long, G. Magdis, A. M. Koekemoer, V. Kokorev, M. Shuntov, M. Talia, N. Allen, S. Harish, O. Ilbert, H. J. McCracken, J. S. Kartaltepe, D. Liu, L. Paquereau, J. Rhodes, M. R. Rich… — [arXiv:2408.10305](https://arxiv.org/abs/2408.10305) find
  31. J. F. W. Baggen, P. van Dokkum, G. Brammer, A. de Graaff, M. Franx, and et al., “The Small Sizes and High Implied Densities of ‘Little Red Dots’ with Balmer Breaks Could Explain Their Broad Emission Lines Without an AGN,” arXiv e-prints, p. arXiv:2408.07745, Aug. 2024. — [arXiv:2408.07745](https://arxiv.org/abs/2408.07745) find
  32. C. A. Collins, J. P. Stott, M. Hilton, S. T. Kay, S. A. Stanford, and et al., “Early assembly of the most massive galaxies,” Nature, vol. 458, pp. 603–606, Apr. 2009. find
  33. R. Mandelbaum, “Weak Lensing for Precision Cosmology,” Annual Review of Astronomy and Astrophysics, vol. 56, pp. 393–433, Sept. 2018. find
  34. V. Desjacques, D. Jeong, and F. Schmidt, “Large-scale galaxy bias,” Physics Reports, vol. 733, pp. 1–193, Feb. 2018. find
  35. R. H. Wechsler and J. L. Tinker, “The Connection Between Galaxies and Their Dark Matter Halos,” Annual Review of Astronomy and Astrophysics, vol. 56, pp. 435–487, Sept. 2018. find
  36. A. Cattaneo, S. M. Faber, J. Binney, A. Dekel, J. Kormendy, and et al., “The role of black holes in galaxy formation and evolution,” Nature, vol. 460, pp. 213–219, July 2009. find
  37. A. C. Fabian, “Observational Evidence of Active Galactic Nuclei Feedback,” Annual Review of Astronomy and Astrophysics, vol. 50, pp. 455–489, Sept. 2012. find
  38. T. M. C. Abbott, M. Aguena, A. Alarcon, S. Allam, O. Alves, and et al., “Dark Energy Survey Year 3 results: Cosmological constraints from galaxy clustering and weak lensing,” Physical Review D, vol. 105, p. 023520, Jan. 2022. find
  39. T. Wang, K. Xu, Y. Wu, Y. Shi, D. Elbaz, and et al., “Black holes regulate cool gas accretion in massive galaxies,” Nature, vol. 632, pp. 1009–1013, Aug. 2024. find
  40. G. B. Brammer, P. G. van Dokkum, and P. Coppi, “EAZY: A Fast, Public Photometric Redshift Code,” The Astrophysical Journal, vol. 686, pp. 1503–1513, Oct. 2008. find
  41. J. Chevallard and S. Charlot, “Modelling and interpreting spectral energy distributions of galaxies with BEAGLE,” Monthly Notices of the Royal Astronomical Society, vol. 462, pp. 1415– 1443, Oct. 2016. find
  42. M. Boquien, D. Burgarella, Y. Roehlly, V. Buat, L. Ciesla, D. Corre, A. K. Inoue, and H. Salas, “CIGALE: a python Code Investigating GALaxy Emission,” Astronomy & Astrophysics, vol. 622, p. A103, Feb. 2019. find
  43. B. D. Johnson, J. Leja, C. Conroy, and J. S. Speagle, “Stellar Population Inference with Prospector,” The Astrophysical Journal Supplement Series, vol. 254, p. 22, June 2021. find
  44. C. Pacifici, K. G. Iyer, B. Mobasher, E. da Cunha, V. Acquaviva, and et al., “The Art of Measuring Physical Parameters in Galaxies: A Critical Assessment of Spectral Energy Distribution Fitting Techniques,” The Astrophysical Journal, vol. 944, p. 141, Feb. 2023. find
  45. B. Wang, J. Leja, H. Atek, I. Labbé, Y. Li, and et al., “Quantifying the Effects of Known Unknowns on Inferred High-redshift Galaxy Properties: Burstiness, IMF, and Nebular Physics,” The Astrophysical Journal, vol. 963, p. 74, Mar. 2024. find
  46. G. Bruzual and S. Charlot, “Stellar population synthesis at the resolution of 2003,” Monthly Notices of the Royal Astronomical Society, vol. 344, pp. 1000–1028, Oct. 2003. find
  47. C. Maraston, “Evolutionary population synthesis: models, analysis of the ingredients and application to high-z galaxies,” Monthly Notices of the Royal Astronomical Society, vol. 362, pp. 799–825, Sept. 2005. find
  48. C. Leitherer, I. H. Li, D. Calzetti, and T. M. Heckman, “Global Far-Ultraviolet (912-1800 Å) Properties of Star-forming Galaxies,” The Astrophysical Journal Supplement Series, vol. 140, pp. 303–329, June 2002. find
  49. D. Calzetti, L. Armus, R. C. Bohlin, A. L. Kinney, J. Koornneef, and et al., “The Dust Content and Opacity of Actively Star-forming Galaxies,” The Astrophysical Journal, vol. 533, pp. 682– 695, Apr. 2000. find
  50. D. A. Dale, G. Helou, G. E. Magdis, L. Armus, T. Díaz-Santos, and et al., “A Two-parameter Model for the Infrared/Submillimeter/Radio Spectral Energy Distributions of Galaxies and Active Galactic Nuclei,” The Astrophysical Journal, vol. 784, p. 83, Mar. 2014. find
  51. G. Ferland, K. Korista, D. Verner, J. Ferguson, J. Kingdon, and E. Verner, “Cloudy 90: numerical simulation of plasmas and their spectra,” Publications of the Astronomical Society of the Pacific, vol. 110, no. 749, p. 761, 1998. find
  52. G. J. Ferland, R. L. Porter, P. A. M. van Hoof, R. J. R. Williams, N. P. Abel, and et al., “The 2013 Release of Cloudy,” Revista Mexicana de Astronomia y Astrofisica, vol. 49, pp. 137–163, Apr. 2013. find
  53. J. Fritz, A. Franceschini, and E. Hatziminaoglou, “Revisiting the infrared spectra of active galactic nuclei with a new torus emission model,” Monthly Notices of the Royal Astronomical Society, vol. 366, pp. 767–786, Mar. 2006. find
  54. P. Camps and M. Baes, “SKIRT: An advanced dust radiative transfer code with a user-friendly architecture,” Astronomy and Computing, vol. 9, pp. 20–33, Mar. 2015. find
  55. L. Li, Y. Wang, R. Xu, P. Wang, X. Feng, and et al., “Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models,” arXiv e-prints, p. arXiv:2403.00231, Feb. 2024.
  56. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, and et al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” arXiv e-prints, p. arXiv:2311.16502, Nov. 2023.
  57. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, “Tree of Thoughts: Deliberate Problem Solving with Large Language Models,” arXiv e-prints, p. arXiv:2305.10601, May 2023. find
  58. M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, and et al., “Graph of Thoughts: Solving Elaborate Problems with Large Language Models,” arXiv e-prints, p. arXiv:2308.09687, Aug. 2023. find
  59. B. Sel, A. Al-Tawaha, V. Khattar, R. Jia, and M. Jin, “Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models,” arXiv e-prints, p. arXiv:2308.10379, Aug. 2023. find
  60. W. Chen, X. Ma, X. Wang, and W. W. Cohen, “Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks,” arXiv e-prints, p. arXiv:2211.12588, Nov. 2022. find
  61. J. Walcher, B. Groves, T. Budavári, and D. Dale, “Fitting the integrated spectral energy distributions of galaxies,” Astrophysics and Space Science, vol. 331, pp. 1–52, Jan. 2011. find
  62. M. Sawicki, “SEDfit: Software for Spectral Energy Distribution Fitting of Photometric Data,” Publications of the Astronomical Society of the Pacific, vol. 124, p. 1208, Nov. 2012. find
  63. A. Diaz Rivero and C. Dvorkin, “Flow-based likelihoods for non-Gaussian inference,” Physical Review D, vol. 102, p. 103507, Nov. 2020. find
  64. Z. Sun, J. S. Speagle, S. Huang, Y.-S. Ting, and Z. Cai, “Zephyr : Stitching Heterogeneous Training Data with Normalizing Flows for Photometric Redshift Inference,” arXiv e-prints, p. arXiv:2310.20125, Oct. 2023. — [arXiv:2310.20125](https://arxiv.org/abs/2310.20125) find
  65. G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, and et al., “Voyager: An Open-Ended Embodied Agent with Large Language Models,” arXiv e-prints, p. arXiv:2305.16291, May 2023. find
  66. A. Zhao, D. Huang, Q. Xu, M. Lin, Y.-J. Liu, and et al., “ExpeL: LLM Agents Are Experiential Learners,” arXiv e-prints, p. arXiv:2308.10144, Aug. 2023. find
  67. C. Qian, S. Liang, Y. Qin, Y. Ye, X. Cong, and et al., “Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution,” arXiv e-prints, p. arXiv:2401.13996, Jan. 2024. — [arXiv:2401.13996](https://arxiv.org/abs/2401.13996) find
  68. M. Carrasco Kind and R. J. Brunner, “SOMz: photometric redshift PDFs with self-organizing maps and random atlas,” Monthly Notices of the Royal Astronomical Society, vol. 438, pp. 3409– 3421, Mar. 2014. find
  69. J. S. Speagle, A. Leauthaud, S. Huang, C. P. Bradshaw, F. Ardila, and et al., “Galaxy-Galaxy lensing in HSC: Validation tests and the impact of heterogeneous spectroscopic training sets,” Monthly Notices of the Royal Astronomical Society, vol. 490, pp. 5658–5677, Dec. 2019. find
  70. A. H. Wright, H. Hildebrandt, J. L. van den Busch, and C. Heymans, “Photometric redshift calibration with self-organising maps,” Astronomy and Astrophysics, vol. 637, p. A100, May 2020. find
  71. I. Labbé, P. van Dokkum, E. Nelson, R. Bezanson, K. A. Suess, and et al., “A population of red candidate massive galaxies 600 Myr after the Big Bang,” Nature, vol. 616, pp. 266–269, Apr. 2023. find
  72. Z. Yuan, H. Yuan, C. Tan, W. Wang, and S. Huang, “How well do Large Language Models perform in Arithmetic tasks?,” arXiv e-prints, p. arXiv:2304.02015, Mar. 2023.
  73. J. Ahn, R. Verma, R. Lou, D. Liu, R. Zhang, and et al., “Large Language Models for Mathematical Reasoning: Progresses and Challenges,” arXiv e-prints, p. arXiv:2402.00157, Jan. 2024.
  74. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, and et al., “Measuring Massive Multitask Language Understanding,” arXiv e-prints, p. arXiv:2009.03300, Sept. 2020.
  75. Y.-S. Ting, T. Dung Nguyen, T. Ghosal, R. Pan, H. Arora, and et al., “AstroMLab 1: Who Wins Astronomy Jeopardy!?,” arXiv e-prints, p. arXiv:2407.11194, July 2024. — [arXiv:2407.11194](https://arxiv.org/abs/2407.11194) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2024-interpreting-multi-band-galaxy-observations/