The event
- Name
- 1st Conference on Language Modeling (COLM 2024)
- Dates
- 7 to 9 October 2024
- Venue
- Zellerbach Theater, Annenberg Center for the Performing Arts, University of Pennsylvania, Philadelphia, Pennsylvania
- Host
- COLM Organization Committee
- Also on the programme
- Christopher Manning (Stanford), Raquel Fernandez (University of Amsterdam), Evelina Fedorenko (MIT), Hannaneh Hajishirzi (U Washington and Allen Institute for AI)
- Programme
- colmweb.org
Trip 7 to 9 October 2024 · Written up 9 October 2024 · Last expanded 29 September 2026
The first thing I posted from Philadelphia, in October 2024, freshly landed for COLM, was that it was somewhat sunny. I had nothing more considered to say yet.
It was the first Conference on Language Modeling, and I had come straight from two months of mathematicians in Cambridge, which turned out to be relevant.
Node, edge, and 2000 GPU days
The best part of COLM 2024 was a burger with Joseph Miller of FAR.ai and the ML Theory & Alignment Scholars group, MATS. He gave a talk on the last day investigating Transformer Circuit Faithfulness, which was broader than his team’s preprint and really a primer on circuit analysis (ablate parts of a network, then ask whether it still behaves the same).
The takeaway was that you have to specify the ablation type, node or edge, or the result does not mean what you think it means. First-year graph theory is built on that distinction (the vertex-edge connectivity inequality is what I reached for, though I cannot say it bears on the paper), and yet much of the literature does not specify, and small terminological differences produce vastly different interpretations. A lecturer at a Cambridge, Massachusetts workshop the week before had made the same complaint, that formalising mathematics is hard because the concepts are ill defined, and I have been hearing versions of it ever since.
The other thing I took away from COLM was a 2D plot from one of the Meta papers at the Mixture of Experts posters: an open source model trained on up to 2000 GPU days. Few academic institutions can put that behind one project, Georgia Tech included, and if you follow that line far enough science starts to look like Weyland-Yutani in Alien, with a mega-corp holding the reins of all research. I am not convinced that is desirable.
Who I met
- Joseph Miller, FAR.ai and the ML Theory & Alignment Scholars group (MATS). Talked circuit analysis over a burger at COLM 2024, and his insistence on specifying node versus edge ablation is the bit I still repeat to people.
Photographs
Links
Joseph Miller’s team’s preprint on transformer circuit faithfulness. To be added.
Open entry
This page is not finished, and is not meant to be. I add photographs, people and links as I dig them out, so it keeps growing after publication. Why the whole site works this way is on the colophon.
