A note on timing: The field of large language models moves incredibly fast. This account follows the project as presented in March 2025. There is a short look at related developments since then towards the end.

During my PhD studies at the group Intelligent Software Systems in the department Intelligent and Distributed Systems of the Institute of Software Technology at the German Aerospace Center (DLR), I worked on Large Language Models for (Research) Software Engineering. My main project was about connecting scientific publications with their source code, using the methods described in a paper to train a better representation of the corresponding code repository.

The motivation for this came from a part of software engineering that is easily overlooked in the discussion around LLMs: understanding existing code. Studies of developers’ work show how much understanding, navigating and gathering information matter alongside writing code.¹ Benchmarks often show impressive success at solving greenfield tasks, where a new implementation is written from a specification. However, that says little about how well a tool helps with understanding an existing project. Especially in research, code is never without context: There is an accompanying publication, there are referenced methods and project proposals, and there is the domain knowledge of the researchers who wrote it.² In practice, I often find myself struggling to connect a claimed method in a paper to the code that actually implements it, including all the hidden assumptions and “works-in-practice” tweaks. This also connects to what already interested me in my master’s thesis: interpretable embeddings, which represent in what way two bodies of text/code are similar and what assumptions they share.

Some things are obvious in code, such as the optimizer or the data augmentation used to train a model. Other things are explained in the paper, such as the choice of dataset or the reasoning behind a particular method. Some knowledge is already available in structured databases, while other decisions are only understandable if you know the field. To make sense of an implementation, you often need to bring several of these sources together. And if you want to adapt it to a different problem, you also need to know which assumptions still apply.

The longer-term idea for my PhD was therefore a system that could use existing implementations and their publications to help researchers and research software engineers with exactly this. The kind of input I had in mind was something like:

We used method X from paper A, but instead of Y we did Z

Given such a description, the system would retrieve related work and point out assumptions or caveats worth checking. This is the idea behind retrieval-augmented generation (RAG): give a language model relevant material to work with when answering a question.³ Here, that material would include both the explanation of a method and its actual implementation. For example, if similar implementations all rely on multithreading, that might explain why a new implementation is unexpectedly slow. Or a choice of loss function in another project might be relevant because it deals with the same kind of imbalanced data, even though it comes from a different research field.

To connect this to other research at DLR, I used an example from materials research: locating the tip of a crack from measured displacement fields, obtained by tracking how a material deforms in images.⁴ A researcher working on this needs knowledge of the material and the measurement process, but also has to decide how to formulate the learning task, deal with the small number of relevant pixels, and evaluate the result. Similar decisions arise in other image-analysis tasks. There is useful knowledge in those other solutions, if you can find the connection and work out which parts actually transfer.

Repository-level embeddings

As a first step towards this, I worked on representing papers and their implementations in a way that would allow them to be matched by content. An embedding is a numerical representation of something, in this case text or code, that allows a model to compare it with other examples. The aim is for things with a similar meaning to have similar representations. This gets especially interesting when code and text are embedded into a shared space, such that a description like “implement quicksort in Python” is close in latent space to a function that does exactly that.⁵ A high-level task description can then lead you to actual code.

However, finding a single function is quite different from representing a whole research project. A repository contains the main algorithm, but also preprocessing, evaluation, helper functions, and code for running different experiments. For this work, I wanted to create a representation that reflects the method implemented by the project as a whole. Repository-level representations already existed: Rep2Vec, for example, combines code content with relationships between repositories.⁶ My focus was on learning from the implementation itself and its corresponding paper. A straightforward baseline was to combine embeddings of individual pieces of code through operations such as mean pooling, which simply averages their vectors.

Simply averaging these embeddings did not seem like a particularly good way to do this. My initial comparisons showed that embedding a complete source file gives a different result from aggregating the embeddings of its contents. Similarly, the representation of an entry point differs from that of the whole codebase. Treating all parts equally can blur the contribution of the code that actually implements the method I am interested in. A large amount of helper code should not become more important simply because there is more of it!

My idea was to learn how to combine these parts, using the corresponding paper as a high-level description of what the code is supposed to do. In particular, the methods section should describe the core of the implementation, without having to explain every utility function around it. This relies on the assumption that the paper faithfully represents the code, which is not automatically true for every paper–repository pair. But where that correspondence holds, it provides a useful training signal: the publication describes what should matter in the repository’s representation.

Learning from papers and code

I assembled a dataset of publications and their linked source code repositories, starting from PapersWithCode. Creating a usable dataset from these links involved quite a few practical restrictions: I focused on Python repositories, needed papers with an identifiable methods section, and had to limit the size of the codebases. Downloading and parsing the publications and repositories also came with its own collection of exceptions. Even with these restrictions, the first roughly 30,000 samples I collected took up about half a terabyte. Having a link between a paper and a repository was a useful starting point, but there was still a lot of work between that and a usable input for the model.

On the code side, I built a representation based on the abstract syntax tree, together with call and data-flow information. This describes the structure of the program and how its parts relate to each other: which functions belong to which files, which functions call others, and how data moves through the code. I embedded the individual pieces of code and used a graph neural network to aggregate them into a single repository-level embedding. The graph allows information to pass between connected parts, so that a function’s representation can take its context into account. It also allows the model to learn how much different parts should contribute to the final representation.

I implemented and trained this model using contrastive learning. The training objective brings the embeddings of a paper and its corresponding repository closer together, while moving non-matching pairs further apart. The existing paper–repository links provide the supervision, without requiring someone to manually describe and label each piece of code. On the repository side, the representation was based on the code and its structure, rather than metadata that could make a match easy without saying much about the implementation itself.

The preliminary evaluation was promising: the trained graph-based model improved on the pooling baselines. These were still early experiments, with plenty of scope to tune the model and investigate the results more thoroughly, but they gave an encouraging first indication that the approach was worth pursuing. For personal reasons, I discontinued the PhD before taking that work further, leaving a working research prototype.

The wider idea extended beyond matching a whole paper to a whole repository. A next step would have been to identify individual methods or modules, so that a researcher could find the relevant preprocessing or evaluation code within a larger project. Combining these representations with information from publications and knowledge graphs was another direction I wanted to explore; approaches such as GraphRAG were part of that research context.⁷ Ultimately, I want representations that would make it easier to find similar approaches across research fields and explain what connects them.

A note on timing

LLMs and the tools around them have moved incredibly quickly since this project. Several related developments appeared later in 2025: GitHub introduced a specialized embedding model for retrieving code and documentation in Copilot, and cAST explored using abstract syntax trees to split code into more meaningful units for retrieval.⁸⁹ It is exciting to see learned representations become part of everyday coding tools, alongside continued research into how program structure can make retrieval better.

These developments overlap with parts of my motivation, while addressing different tasks. Retrieving relevant pieces of code for an assistant differs from learning a representation of an entire research implementation through its associated publication. That paper–implementation connection, and the longer-term possibility of understanding shared assumptions across domains, was the particular part of the problem I was interested in.

Full Abstract

Unfortunately, I cannot share the code here due to a non-disclosure agreement. Below is the original abstract for the paper I had planned to write about this work, kept as a snapshot of the project at the time.