Étienne Simon

Compositionality & Capabilities of Language Models

I have an interest in formal methods to analyse the capabilities of language models, in particular I recently worked on compositionality with my PhD student Sondre Wold who has now defended.

Our last paper is on relaxing the notion of synonymy in the formal semantics definition of compositionality and apply that definition to modern transformer models.

We also have a paper on evaluating systematic generalisation as a function of certain entropy characterisations of the input dataset, and a paper on the generalization of grounding language models on knowledge graphs using graph neural networks.

I am currently in the early stages of investigating some generalization properties of in-weight and in-context knowledge use with Syrielle Montariol.

Information Extraction

I am currently also working on event extraction as part of the PSI project. An important focus of the project is the (partial) automation of conflict events extraction for UCDP. We published some surveys of datasets and generative models for the task which is characterised by a difficult-to-capture abstraction step (the "extracted" fields do not appear directly in the source documents). If you want to have a go at that task yourself, we have a dataset paper, with an update coming soon.

Before that, my PhD focused on unsupervised (or "self-supervised" as people properly say now) relation extraction. In particular, we have a paper on regularising VAE for that task.

Other

Before starting my PhD I did an internship at Facebook AI Research in Paris for my end of master internship, working on Neural Machine Translation with Memory Networks.

During a 1 year stay in Montréal, I've won a Kaggle competition on taxi destination prediction as part of team 🚕 with Alex Auvolat and Alexandre de Brébisson.

I organised some reading groups in the labs I joined, usually paper reading groups like at MLIA, but also some book reading groups like Bishop's PRML.