Home / Publications / Talks / Teaching / Students / Software / Data

For a full list, see my Google scholar profile.

Research topics

• Differentiable programming
• Language models
• Deep learning
• Loss functions
• Optimization
• Optimal transport
• Polynomial networks
• Machine learning software

Differentiable programming

• The Elements of Differentiable Programming.
  Mathieu Blondel, Vincent Roulet.
  Book draft, March 2024.
  Website / arXiv / Code
• Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective.
  Michael Sander, Joan Puigcerver, Josip Djolonga, Gabriel Peyré, Mathieu Blondel.
  In Proceedings of International Conference on Machine Learning (ICML), July 2023.
  arXiv / Code / Slides
• Deep embedding and alignment of protein sequences.
  Felipe Llinares-López, Quentin Berthet, Mathieu Blondel, Olivier Teboul, Jean-Philippe Vert.
  Nature Methods, volume 20, pages 104–111, 2023.
  Link / bioRxiv / Code
• Efficient and Modular Implicit Differentiation.
  Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares-López, Fabian Pedregosa, Jean-Philippe Vert.
  In Proceedings of Neural Information Processing Systems (NeurIPS), December 2022.
  arXiv / Code
• Differentiable Divergences Between Time Series.
  Mathieu Blondel, Arthur Mensch, Jean-Philippe Vert.
  In Proceedings of Artificial Intelligence and Statistics (AISTATS), April 2021.
  arXiv / Code
• Learning with Differentiable Perturbed Optimizers.
  Quentin Berthet, Mathieu Blondel, Olivier Teboul, Marco Cuturi, Jean-Philippe Vert, Francis Bach.
  In Proceedings of Neural Information Processing Systems (NeurIPS), December 2020.
  arXiv / Code
• Fast Differentiable Sorting and Ranking.
  Mathieu Blondel, Olivier Teboul, Quentin Berthet, Josip Djolonga.
  In Proceedings of International Conference on Machine Learning (ICML), July 2020.
  arXiv / Code (see also PyTorch reimplementation by Teddy Koker).
• SparseMAP: Differentiable Sparse Structured Inference.
  Vlad Niculae, André F. T. Martins, Mathieu Blondel, Claire Cardie.
  In Proceedings of International Conference on Machine Learning (ICML), July 2018.
  arXiv / Code
• Differentiable Dynamic Programming for Structured Prediction and Attention.
  Arthur Mensch, Mathieu Blondel.
  In Proceedings of International Conference on Machine Learning (ICML), July 2018.
  arXiv / Code
• A Regularized Framework for Sparse and Structured Neural Attention.
  Vlad Niculae, Mathieu Blondel.
  In Proceedings of Neural Information Processing Systems (NIPS), December 2017.
  arXiv / Code
• Soft-DTW: a Differentiable Loss Function for Time-Series.
  Marco Cuturi, Mathieu Blondel.
  In Proceedings of International Conference on Machine Learning (ICML), August 2017.
  PDF / Code

Language models

• Direct Language Model Alignment from Online AI Feedback.
  Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, Mathieu Blondel.
  arXiv
• Decoding-time Realignment of Language Models.
  Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, Mathieu Blondel.
  arxiv

Deep learning

• How do Transformers perform In-Context Autoregressive Learning?
  Michael Sander, Raja Giryes, Taiji Suzuki, Mathieu Blondel, Gabriel Peyré.
  arXiv
• Routers in Vision Mixture of Experts: An Empirical Study.
  Tianlin Liu, Mathieu Blondel, Carlos Riquelme, Joan Puigcerver.
  arxiv
• Sinkformers: Transformers with Doubly Stochastic Attention.
  Michaël Sander, Pierre Ablin, Mathieu Blondel, Gabriel Peyré.
  In Proceedings of Artificial Intelligence and Statistics (AISTATS), March 2022.
  arXiv / Code
• Momentum Residual Neural Networks.
  Michaël Sander, Pierre Ablin, Mathieu Blondel, Gabriel Peyré.
  In Proceedings of International Conference on Machine Learning (ICML), July 2021.
  arXiv / Code

Loss functions

• Learning Energy Networks with Generalized Fenchel-Young Losses.
  Mathieu Blondel, Felipe Llinares-López, Robert Dadashi, Léonard Hussenot, Matthieu Geist.
  In Proceedings of Neural Information Processing Systems (NeurIPS), December 2022.
  arXiv
• Sparse Continuous Distributions and Fenchel-Young Losses.
  André F. T. Martins, Marcos Treviso, António Farinhas, Pedro M. Q. Aguiar, Mário A. T. Figueiredo, Mathieu Blondel, Vlad Niculae.
  Journal of Machine Learning Research (JMLR).
  arXiv / Code
• Structured Prediction with Projection Oracles.
  Mathieu Blondel.
  In Proceedings of Neural Information Processing Systems (NeurIPS), December 2019.
  arXiv / Code
• Geometric Losses for Distributional Learning.
  Arthur Mensch, Mathieu Blondel, Gabriel Peyré.
  In Proceedings of International Conference on Machine Learning (ICML), June 2019.
  arXiv / Code
• Learning with Fenchel-Young Losses.
  Mathieu Blondel, André F. T. Martins, Vlad Niculae.
  Journal of Machine Learning Research (JMLR), Volume 21, pp. 1−69, 2020.
  arXiv / Code
• Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms.
  Mathieu Blondel, André F. T. Martins, Vlad Niculae.
  In Proceedings of Artificial Intelligence and Statistics (AISTATS), April 2019.
  arXiv / Code

Optimization

• Implicit Diffusion: Efficient Optimization through Stochastic Sampling.
  Pierre Marion, Anna Korba, Peter Bartlett, Mathieu Blondel, Valentin De Bortoli, Arnaud Doucet, Felipe Llinares-López, Courtney Paquette, Quentin Berthet.
  arXiv
• Dual Gauss-Newton Directions for Deep Learning.
  Vincent Roulet, Mathieu Blondel.
  arXiv
• Implicit Differentiation for Fast Hyperparameter Selection in Non-Smooth Convex Learning.
  Quentin Bertrand, Quentin Klopfenstein, Mathurin Massias, Mathieu Blondel, Samuel Vaiter, Alexandre Gramfort, Joseph Salmon.
  Journal of Machine Learning Research (JMLR).
  arXiv / Code
• Implicit Differentiation of Lasso-type Models for Hyperparameter Optimization.
  Quentin Bertrand, Quentin Klopfenstein, Mathieu Blondel, Samuel Vaiter, Alexandre Gramfort, Joseph Salmon.
  In Proceedings of International Conference on Machine Learning (ICML), July 2020.
  arXiv / Code
• Large-scale Multiclass Support Vector Machine Training via Euclidean Projection onto the Simplex.
  Mathieu Blondel, Akinori Fujino, Naonori Ueda.
  In Proceedings of the 22nd International Conference on Pattern Recognition (ICPR), August 2014.
  PDF / Code
• Online Passive-Aggressive Algorithms for Non-Negative Matrix Factorization and Completion.
  Mathieu Blondel, Yotaro Kubo, Naonori Ueda.
  In Proceedings of the 17th International Conference on Artifical Intelligence and Statistics (AISTATS), April 2014.
  PDF / Supplementary material
• Block Coordinate Descent Algorithms for Large-scale Sparse Multiclass Classification.
  Mathieu Blondel, Kazuhiro Seki, Kuniaki Uehara.
  Machine Learning, May 2013. The final publication is available here.
  Open-science honorable mention at ECML PKDD 2013.
  PDF / Code / Data

Optimal transport

• Sparsity-Constrained Optimal Transport.
  Tianlin Liu, Joan Puigcerver, Mathieu Blondel.
  In Proceedings of International Conference on Learning Representations (ICLR), May 2023.
  arXiv / Code
• Large-Scale Optimal Transport and Mapping Estimation.
  Vivien Seguy, Bharath Bhushan Damodaran, Rémi Flamary, Nicolas Courty, Antoine Rolet, Mathieu Blondel.
  In Proceedings of International Conference on Learning Representations (ICLR), April 2018.
  arXiv / Code
• Smooth and Sparse Optimal Transport.
  Mathieu Blondel, Vivien Seguy, Antoine Rolet.
  In Proceedings of Artificial Intelligence and Statistics (AISTATS), April 2018.
  arXiv / Code / Data

Polynomial networks

• Multi-output Polynomial Networks and Factorization Machines.
  Mathieu Blondel, Vlad Niculae, Takuma Otsuka, Naonori Ueda.
  In Proceedings of Neural Information Processing Systems (NIPS), December 2017.
  arXiv
  
• Higher-order Factorization Machines.
  Mathieu Blondel, Akinori Fujino, Naonori Ueda, Masakazu Ishihata.
  In Proceedings of Neural Information Processing Systems (NIPS), December 2016.
  arXiv
• Polynomial Networks and Factorization Machines: New Insights and Efficient Training Algorithms.
  Mathieu Blondel, Masakazu Ishihata, Akinori Fujino, Naonori Ueda.
  In Proceedings of International Conference on Machine Learning (ICML), September 2016.
  PDF
  Open-source implementation by Vlad Niculae
• Convex Factorization Machines.
  Mathieu Blondel, Akinori Fujino, Naonori Ueda.
  In Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), September 2015.
  PDF / Slides
  Open-source implementation by Vlad Niculae

Machine learning software

• Scikit-learn: Machine Learning in Python.
  Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, Édouard Duchesnay.
  Journal of Machine Learning Research (JMLR), volume 12, pp. 2825−2830, 2011.
  PDF / Homepage
• API design for machine learning software: experiences from the scikit-learn project
  Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Müller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake Vanderplas, Arnaud Joly, Brian Holt, Gaël Varoquaux.
  ECML PKDD Workshop: Languages for Data Mining and Machine Learning, September 2013.
  PDF

Copyright 2010-Present Mathieu Blondel