Visiting EMNLP 2026 in Budapest

From 24 to 29 October, EMNLP 2026 took place in Budapest, and SciEmbed was part of it. The paper itself has its own post: a 149M-parameter embedding model for scientific documents, trained on the sentences researchers write when they cite each other. This one is about the rest: what the conference was like, and what I would tell myself before starting the project again.

After DRUID and the AOM papers, this was my first paper at a core NLP venue. The audience is different. At DRUID the question was “what does this tell us about innovation?”. At EMNLP it is “what is your baseline, and did you control for the data?”. Both are good questions. The second one is the one I had spent most of the year on.

The conference

The poster session went really well: 90 minutes of good conversations. Most of them were about what the model can and cannot do, which is exactly the part of a paper I most like to discuss. A good share were about the benchmarks, which tasks SciRepEval actually measures and why we built our own probe for long documents. And a lot of people wanted to talk about downstream use, including how I use the model in my own research on how AI methods spread through science. Those were the conversations I enjoyed most, because they started where the paper ends.

Outside my own session, I spent a lot of time in the other poster halls. Most of what I saw and talked about there was LLM agents and alignment, which is a good reminder of where most of the field’s attention is right now, and fairly far from embedding models for science. That distance made the conversations more interesting, not less.

Graham Neubig’s keynote was one of the highlights of the week, and so was the social event. I also had good talks with people from IBM and Google DeepMind. With IBM that was a nice coincidence: their Granite Embedding R2 is the baseline that taught me the most in this paper (more on that below).

SciEmbed on board 325.

The Parliament building on the Danube, one evening during the conference.

Lessons from building SciEmbed

Build the data infrastructure first

The most important decision for SciEmbed was made a year before it started. When I built the Science Data Lake, I was not planning to train an embedding model. But once eight scholarly databases sat behind one SQL interface, generating 70 million training pairs from the citation graph was a set of queries, not a month of API scripts. The idea for the model itself came from exploring that data. In my experience, the model is rarely the slow part. Getting clean data into a shape you can query is.

Report the ablation that hurts

SciEmbed has two training stages. The first, domain-adaptive pretraining on 13.4 million full-text papers, ran for 16 hours on four H200s. In the controlled ablation it changed the benchmark score by 0.1 points. That is not the result you hope for after setting it up, and it would have been easy to leave it out. It is in the paper, with a checkpoint that skips the stage, because it is the most useful thing in there for anyone with a smaller compute budget.

Pick the baseline that can beat you

The strongest comparison in the paper is IBM’s Granite Embedding R2: same ModernBERT backbone, same size, different training data. It beats SciEmbed overall by 0.6 points. It also turned out to be the most informative baseline, because with everything else held equal, the two models specialise in opposite directions. A weaker baseline would have given a nicer table and a less interesting paper.

Hyperparameters do not fix a data ceiling

When you are 0.6 points behind, the obvious reaction is to tune harder. We ran a 28-configuration sweep over temperature, batch size and data mix and tried four alternative training signals. The best run improved the score by 0.01. What finally explained the gap was looking at the data, not the optimiser. It is a lesson I keep having to learn: if many reasonable settings land in the same place, the limit is probably in what the model sees.

Build the test you need

Only 3.9% of the inputs in the standard benchmark are longer than 512 tokens, so it cannot show whether a model with an 8K context window uses it. We built a small probe for that, Body-Fact Retrieval, and released it as a dataset. It took a few days and answered a question no existing benchmark could.

Release everything

Code, configs, the evaluation harness, six checkpoints and the probe dataset are all public, MIT licensed. Part of that is principle. Part of it is practical: preparing a release forces you to rerun everything from a clean state, and that is when you find out which results you can actually reproduce.

What comes next

I am finishing my PhD at TU Hamburg in 2027 and looking for industry roles in applied ML, NLP and R&D. SciEmbed is the kind of work I would like to keep doing: training models on data that nobody has put to use yet, and checking honestly what they can and cannot do. If that sounds relevant to you, get in touch.