NeurIPS 2024: Capital One showcases leading AI research

The 38th Annual Conference on Neural Information Processing Systems (NeurIPS), returns this December, and Capital One is excited to be a part of it! As a company committed to responsible and innovative AI, we're eager to share our latest research, connect with fellow researchers and engage in the vibrant exchange of ideas that defines this event.

At Capital One, we're leveraging AI to unlock new possibilities in financial services and deliver exceptional customer experiences. NeurIPS provides a crucial platform to engage and exchange ideas with some of the best minds in AI and science. We're eager to engage with leading academics, researchers and industry experts to discuss the latest advancements and challenges in AI. Our research efforts are focused on applying AI/ML techniques to address real-world challenges in the financial domain, such as improving the efficiency and interpretability of models, developing advanced techniques for analyzing financial time series data and building transparent and understandable AI systems. This work is critical to developing trustworthy AI solutions, and we're particularly excited to share our progress in areas like deep learning, sequence modeling, explainable AI and generative AI.

At NeurIPS 2023, our Applied Research team had several works accepted. This year, we’re continuing to showcase our research, foster collaboration and support the next generation of AI talent

We have contributed to three papers accepted to the main conference track:

• Distributional Preference Alignment of LLMs via Optimal Transport: Distinguished Applied Researcher, Igor Melnyk, takes the lead as first-author exploring novel methods for aligning large language models (LLMs) with user preferences using optimal transport theory. This work enhances LLMs' ability to generate content that aligns with specific needs and values. 
• Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices: VP of AI Research Bayan Bruss and Senior Machine Learning Engineer Christopher Ferri contribute their expertise to this collaborative research with NYU, which focuses on optimizing the efficiency of neural networks by exploring a continuous space of structured matrices for linear layers. This has the potential to lead to faster and more efficient models.
• Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series: In partnership with IBM Corp, Senior Distinguished Applied Researcher Nam Nguyen helps to introduce innovative techniques for time series forecasting with Tiny Time Mixers (TTMs). These pre-trained models are optimized for fast and accurate predictions, even with limited fine-tuning data.

Dr. Furong Huang, our inaugural Visiting Scholar and an Associate Professor in Computer Science at the University of Maryland, also contributed to six additional NeurIPS papers. Dr. Huang’s expertise in trustworthy machine learning strengthens our research collaborations and reflects our commitment to bridging academia and industry. 

Capital One's presence at NeurIPS extends beyond the main conference with seven accepted workshop papers, further demonstrating our commitment to advancing AI research and fostering the next generation of AI talent through our internship programs. 

Five of the accepted papers are authored by talented individuals from our 2024 Applied Research Internship Program (ARIP), a program designed to provide PhD students with hands-on experience tackling real-world AI challenges in the financial sector. 

• Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models: This paper explores a novel method for controlling the refusal behavior of large language models by introducing "refusal tokens" during training. This technique allows for fine-grained control over the model's tendency to refuse certain prompts or questions, enhancing safety and reliability.
• Dense Backpropagation Improves Routing for Sparsely-Gated Mixture of Experts: This paper investigates the use of dense backpropagation in Mixture of Experts (MoE) models, demonstrating its effectiveness in improving routing decisions and overall model performance. This work contributes to the advancement of MoE models, which are known for their efficiency and scalability.
• Language Model Scaling Laws and Zero-sum Learning: This research delves into the relationship between language model scaling laws and zero-sum learning, exploring how the competitive dynamics of zero-sum games can influence the scaling behavior and performance of large language models.
• StructMoE: Augmenting MoEs with Hierarchically Routed Low Rank Experts: This paper proposes StructMoE, a novel architecture that enhances Mixture of Experts (MoE) models by incorporating hierarchically routed low-rank experts. This approach aims to improve the efficiency and expressiveness of MoE models, further advancing their capabilities in various applications.

We are also proud to highlight a workshop paper authored by a former intern from our Data Science Internship Program (DSIP), which offers aspiring data scientists the opportunity to contribute to cutting-edge research and development.

• Enhancing Table Representations with LLM-powered Synthetic Data Generation: This paper explores the use of large language models (LLMs) to generate synthetic tabular data for improving table representations. This approach aims to enhance the performance of similar table recommendation systems, which are crucial for efficient data management and analysis in data-driven enterprises. The research introduces a novel synthetic data generation pipeline that leverages LLMs to create a large-scale dataset tailored for table-level representation learning, leading to improved accuracy in recommending similar tables.

Finally, we have workshop papers co-authored by Senior Distinguished Applied Researcher Nam Nguyen, and Distinguished Applied Researcher Supriyo Chakraborty further demonstrating our commitment to collaborative research and knowledge sharing within the AI community.

• Scaling-laws for Large Time-series Models: This research, conducted in collaboration with Johns Hopkins University, explores the scaling laws that govern the performance of large time-series models. By examining the relationships among model size, data volume, and computational resources, this study offers valuable insights into the efficient training and deployment of these models for diverse time-series forecasting tasks in finance and other domains.
• MyCroft: Towards Effective and Efficient External Data Augmentation: This research introduces MyCroft, a new data-efficient framework implementing techniques to evaluate relative utility of  relevant external data sources that can augment internal data to improve model performance. These techniques leverage feature space distances and gradient matching to identify small but informative data subsets to maximize performance with minimal data exposure.

Led by Senior Distinguished Applied Researcher Nam Nguyen, this talk delves into the intricacies of applying sequence modeling to financial data. Learn about cutting-edge research, including novel approaches for leveraging powerful transformer models for enhanced insights and predictions.

• Date/Time: Tuesday, December 10th, 4:00 PM local time
• Location: West Ballroom B

Led by Distinguished Machine Learning Engineer Doron Bergman, this talk explores the challenges and opportunities of deep learning for tabular data in finance. Discover how deep learning can surpass traditional methods and unlock new possibilities for financial modeling.

• Date/Time: Wednesday, December 11th, 1:00 PM local time
• Location: West Meeting Room 109/110

We're excited to connect with you at NeurIPS 2024! Come visit us at booth #315 where you can:

• Explore our research: Dive deep into our latest advancements in AI and machine learning.
• Discover career opportunities: Learn about exciting applied research career paths at Capital One for researchers and engineers passionate about AI and join our world-class team.
• Engage with our team: Meet our researchers and AI experts, ask questions and discuss the future of AI in finance.