Alignment methods like RLHF and DPO expect feedback in the form of preferences (e.g., Output A is better than B for input X). Utilizing human annotation efforts for this feedback quickly gets very expensive, and can also result in conflicting data. Kahneman-Tversky Optimization (KTO) matches or exceeds (state-of-the-art) DPO performance without using preference data. KTO is far easier to use in the real world, where preferences are scarce and expensive to collect.

An interesting problem in representation learning is training on an optimal index that is helpful towards the learning task. We explore the impacts of sequentially compressing an unordered set of assets as opposed to random priority assignment. Unfortunately, general set compression methods show minimal gains in terms of rate-distortion tradeoff when used with our proposed associative compression architectures.

A longstanding goal in the field of AI is to find a strategy for compiling diverse experience into a highly capable, generalist agent. In the subfields of vision and language, this was largely achieved by scaling up transformer-based models and training them on large, diverse datasets. Motivated by this progress, we train one generalist control agent on a diverse set of offline data to play 46 Atari games. We demonstrate scaling of performance in the model size and rapid adaptation to novel games with fine-tuning. Compared to existing methods in behavioural cloning, online, and offline RL, MGDT offers the best scalability and performance.

We develop a new class of differentiable operators that leverage the underlying regularity of natural and artificial objects at various scales. Neural Collages are a class of implicit operators that discover self-similar representations of data and are efficient neural compressors, powerful decoders in deep generative models, and may be used towards various creative and artistic generations.

Modern language models do in-context learning and show amazing abilities to zero-shot generalize to unseen conditions. We represent compositions of prompted language models as probabilistic programs termed Model Cascades. With this general framework for sampling and inference, we formalize prompting, reasoning, and tool use as graphical models over random variables with complex string values.

We explore the concept of infinite-dimensional stochastic variational inference in learning the dynamics of continuous-time Neural ODEs with instantaneous noise. Our framework trains Bayesian neural networks by parameterizing weight uncertainty with stochastic differential equations (SDEs) and associate efficient gradient-based algorithms. We also conjecture and derive zero variance gradient estimators to the infinite-dimensional case as the approximate posterior converges to the true posterior.

We introduce Noisy Feature Mixup (NFM), an inexpensive yet effective method for data augmentation that combines the best of interpolation based training and noise injection schemes. We use noise-perturbed convex combinations of datapoint pairs in both input and feature space to learn smoother decision boundaries, leading to improved robustness. The advantageous implicit regularization effects of NFM compared to previous mixup training methods are further understood with our theoretical insights. We show that residual networks and vision transfromers trained with NFM have favorable trade-offs between predictive accuracy and out-of-distribution generalization.

We introduce Goldilocks Selection, a technique for faster model training which selects a sequence of training points that are "just right". We propose an information-theoretic acquisition function -- the reducible held-out loss -- and compute it with a small proxy model -- GoldiProx -- to efficiently choose training points that maximize information about a validation set. We show that the selected sequence not only prioritizes learnable, yet information rich data relevant to the evaluation task but also effectively transfers across architectures and vision tasks.

We validated the utility of two novel next-generation-sequencing assays (ChIP-Exo and ChIP-Nexus) in epigenetic exploration of cancer transcriptomes. I implemented custom bioinformatics pipelines, unsupervised ML algorithms (Segway), and the stable marriage algorithm.

As part of the 2019 ICLR Reproducibility Challenge, we implemented this paper (later accepted) that investigated a novel improvement to Equilibrium Propagation, which is a method of energy based models (EBMs).