By Conor Hamill (Senior Data Scientist, AI R&D)
In the fast-evolving world of technology and financial services, one thing is clear: the pace of advancement in AI has been nothing short of astonishing. With the annual publication of AI research papers soaring, we are witnessing capabilities that seemed far-fetched just a few years ago now being integrated into everyday workflows for individuals and businesses alike. The ideas and innovations in cutting-edge techniques in machine learning and AI from academia and industry are vividly showcased at international conferences like NeurIPS, often hailed as one of the largest and most influential gatherings in the realm of machine learning and AI.
It was great to see NatWest Group as a bronze sponsor of NeurIPS 2024, with Conor Hamil and Raad Khraishi representing the bank during the week-long conference in December.
NeurIPS 2024 was held in the Vancouver Conference Centre, with the city nestled between the mountains and the sea serving as an inspiring environment for the conference.
Unsurprisingly, large language models (LLMs) were a major focal point of this year’s conference, with many ground-breaking studies addressing key challenges and advancements in this area. Researchers explored mitigating hallucinations in outputs (Truth is Universal: Robust Detection of Lies in LLMs), in-depth studies predicting the emergence of abilities during pre-training (Observational Scaling Laws and the Predictability of Language Model Performance), and optimising prompts for tasks (Efficient Prompt Optimization Through the Lens of Best Arm Identification).
Simultaneously, there was a huge focus on maintaining the same performance of LLMs while reducing the amount of compute (Once Read is Enough), training data (Not All Tokens Are What You Need for Pretraining), and model parameters (Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers) required. Progress in these endeavours is essential for developing faster, more cost-effective models and enhancing computing capabilities on edge devices.
The development of LLM agents using these LLMs made up a large amount of the work in the conference. This included works exploring different workflows of LLM agents (Chain of Agents: Large Language Models Collaborating on Long-Context Tasks) and enhanced agent capabilities (Multi-Agent Coordination via Multi-Level Communication), as well as the emerging capabilities of multi-model agents (FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement).
As LLMs get deployed in more and more applications, some of which are critical or contain sensitive information for end users, the understanding of the potential risks to these systems and how these can be mitigated is crucial, and research into these threats was very prominent at the NeurIPS conference. Works looking at jail-breaking of LLMs (Mission Impossible: A Statistical Perspective on Jailbreaking LLMs), toxicity detection (Toxicity Detection for Free) and the emerging risks from agents (Watch Out for Your Agents!).
Additionally, many discussions focused on the pressing challenges of fairness and bias in AI, as well as innovative efforts to address them. This included processes for de-biasing datasets (A Simple Remedy for Dataset Bias via Self-Influence), methods for estimating the fairness of algorithms (OxonFair: A Flexible Toolkit for Algorithmic Fairness), and improvements on performance on under-represented groups in datasets (Improving Subgroup Robustness via Data Selection).
All of these discussions were vital for us as scientists and engineers to ensure we keep these challenges at the front of our minds when using technology to develop services aimed at improving how we serve our customers.
NeurIPS wasn’t just significant in terms of attendance; it also featured an impressive variety of events throughout the week and beyond. Participants could choose from engaging workshops and tutorials, as well as innovative projects presented by high school students. As well as this, there were affinity events and creative AI sessions throughout, and a huge number of community groups organised, meaning, despite the thousands of people there, there was an opportunity for everyone to connect with others who shared similar interests, making the experience inclusive and enriching despite the large turnout.
The Exhibits Hall was alive with excitement, buzzing as vendors showcased their ground-breaking innovations and shared invaluable insights with conference presenters and attendees. Renowned tech giants like Google and Amazon drew large crowds, while dynamic startups such as Wayve exhibited their fresh ideas.
The past decade has been an incredible time for the field of machine learning and AI and the next ten years promises to be just as exciting. NeurIPS has a longstanding tradition of recognizing research that significantly impacts the field through its “Test of Time” award. This prestigious accolade honours the authors of seminal papers presented at the conference a decade ago. At the 2024 conference, two noteworthy works received this prize: the influential paper on “Generative Adversarial Networks” (GANs), which introduced a revolutionary framework for generative models, and “Sequence to Sequence Learning with Neural Networks”, which laid the groundwork for the advanced architectures that power today’s notable language models.
A reflection of the former paper was presented by Ian Goodfellow and David Warde-Farley, while Ilya Sutskever, still one of the most well-known individuals in AI, presented reflections on the latter work and what he thought the future held for the field.
In his engaging talk, Sutskever highlighted that we are transitioning away from an era where advancements in models solely relied on scaling training compute and data. He pointed out the field is changing rapidly, with the focus shifting to agents, synthetic data, and test-time compute. This has been borne out in recent months, with the recent surge in research on agents and the release of reasoning models like OpenAI’s o3 model.
Attending this event and seeing the cutting-edge of AI research, along with the individuals that have driven the field towards such success, was an amazing experience. Seeing NatWest Group sponsoring an event like this, alongside names like Duolingo, showed me how committed the bank is to being at the forefront of research.
We arrived home with loads of research ideas in our minds, eager to explore how the innovations we had seen could steer and be built on by the AI research happening across the bank and with our university collaborators. Being able to see so much in such a short space of time kept us at the forefront of AI transformation and the support of senior data leaders in the bank meant that we were able to do this. AI is helping to accelerate change in financial services, and attending events like NeurIPS informs our research and equips us to deliver for our customers and stay on top of the cutting-edge developments that are shaping the future of banking.
If you found this blog post interesting and would like to work on similar problems, we encourage you to take a look at our available job openings!
The views and opinions expressed in this article are those of the author and do not necessarily represent the views of the NatWest Group.
<hr><p>NatWest Group AI team at NeurIPS 2024 was originally published in NatWest Group AI & Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>