Jonathan Chung

Today, we’re launching Substrate. We’re also announcing our $8M Series Seed led by Lightspeed. We believe the most robust, reliable, and productive uses of AI are when many models are used in coordination with each other, with regular programming mixed in. Most people building with AI already know this. But the available tools are suboptimal, unless you work for Google. For the rest of us, we’re left creating an unwieldy mess of chained API calls to multiple providers, with many roundtrips in between. We took a hard look at this state of affairs, and recognized how much it is stifling progress. Building large multi-step AI workloads requires sophisticated high-performance tooling and infrastructure. Nobody wants to deal with more tooling and infrastructure… but everyone would benefit from simple, intuitive interfaces that abstract away a powerful system underneath. No tooling, no infrastructure – just elegant abstractions.

27

1 Comment

The best part of investing at the earliest stages of company building is every once in a while you meet founders who can see the future. Ben Guo & Rob Cheung are two of those founders and after a year of building in stealth they're launching ꩜Substrate, the first inference API optimized for multi-step AI workloads. 👉 https://substrate.run With ꩜Substrate, you connect nodes from a curated library that includes optimized ML models, built-in file and vector storage, a code interpreter, and logical control flow. By simply connecting nodes, you describe a graph workflow, which Substrate then analyzes and runs as fast as possible. Entire graphs of many nodes will often run on a single machine, with microsecond communication between tasks. We are thrilled to participate in their 💵 $8M Seed led by Lightspeed (Guru Chahal/Nnamdi Iregbulem) alongside our friends at South Park Commons (Arian Agrawal), Craft Ventures, Red Swan Ventures (Eric Rubin/Sandy Cass), joined by amazing 🧠 founders and 👨‍🏭 operators like Guillermo Rauch, Immad Akhund, William Gaybrick, Chris Best, Gokul Rajaram, Shreyas Doshi, Jana Messerschmidt, Michael Manapat, and Matthew Hartman.

23

1 Comment

👉🏼 Performance Assessment of GPT 4.0 on the Japanese Medical Licensing Examination 🤓 Hong-Lin Wang 👇🏻 https://lnkd.in/e3b4kjU8 🔍 Focus on data insights: - 📈 GPT 4.0 achieved a correct response rate of 80.4% on the Japanese Medical Licensing Examination, indicating strong performance in comprehension and analysis. - 🔍 Analysis revealed significant differences in accuracy based on question difficulty and type, highlighting areas for targeted improvement. - 🏆 The Global Quality Scale

1

Hi LinkedIn Community,   Short version:  Grad student building low-code tool for ML data pipelines. Goal: Make ML dev work enjoyable again, not just endless debugging. Ask: Shadow sessions (see Calendly link below) Longer version: As ML devs, we often leave the office thinking, “Do I actually love doing my work?” because we had spent most of the day on inefficient whack-a-mole debugging and maintenance work. I want to get us back to where we can love coding for those reasons we originally chose to pick up coding in the first place.   A lot of AI tools are restrictive and end up mangling codebases, and I believe the current state of AI coupled with strong design principles can enable much better code native developer experiences. I’ve gone through prototypes/testing and have some competing hypotheses. I’d love to observe coders working on ML data pipelines (happy to sign NDAs if necessary, can also be personal projects instead). If you're open to sharing your expertise, please schedule a shadow session: https://lnkd.in/eBW6p6qE Let's make ML data development more efficient and actually enjoyable! #MachineLearning #DataPipelines #LowCode #DevTools #AI #SoftwareEngineering #GradStudent #Research #MLOps #DataScience #ArtificialIntelligence #ProductivityTools #DeveloperExperience #CodeQuality #TechInnovation #ComputerScience #DataEngineering #CloudComputing

84

8 Comments

I'm excited to share two papers from the Retrieval team at Walmart that were presented at CIKM last week. They're both about improving the relevance of Embedding-Based Retrieval. 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗙𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴 𝗳𝗼𝗿 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹. https://lnkd.in/gNQe7QhA The first is about how to intelligently choose a query-dependent cutoff on the cosine score to remove irrelevant items from the retrieved set. Since cosine scores aren't comparable across queries, we introduce a "Cosine Adapter" to map the scores to interpretable scores. 𝗘𝗻𝗵𝗮𝗻𝗰𝗶𝗻𝗴 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗼𝗳 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗮𝘁 𝗪𝗮𝗹𝗺𝗮𝗿𝘁. https://lnkd.in/gdqkc2hR The second is about model training techniques we learned to improve the relevance of embedding models. In particular, we describe how to use a teacher model to provide relevance labels for training Juexin (Joy) Lin Nicholas Rossi, PhD Feng LIU Zhen Yang Sachin Yadav Praveen Reddy Suram Satya Chembolu Prijith Chandran Hrushikesh Mohapatra Alessandro Magnani Ciya Liao

143

1 Comment

Last weekend, I read "Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought" by Lee et al, 2024. In this paper propose to use small (<1B params) language model (SLM) to provide a sequence of rationale steps to a LLM to predict the output of a task. The authors argue that this is an efficient way to use LLMs to solve a task by only training the SLM that provides the rationale to the LLM. To me, this approach is interesting because they use reinforcement learning by assigning reward signals for a SLM from a rationale-oriented and task-oriented perspectives to improve the outputs of a LLM. Enjoy! Paper: https://lnkd.in/gcQwW-ty

3

I still remember the early days of my career as a Machine Learning engineer, navigating various sites looking for data -- Github, Kaggle, UCI, you name it. One particular project comes to mind—a predictive analytics tool for early diagnosis of diabetic retinopathy. Despite the importance and potential impact of the project, we hit a major roadblock: insufficient data. The patient data we had was not only limited but also fraught with privacy concerns. This challenge was an eye-opener, underscoring the critical need for synthetic data in healthcare. Now, that I see multiple services are offering synthetic data. Synthetic data is artificially generated rather than obtained by direct measurement. It mirrors the statistical properties of real-world data, making it a powerful tool in scenarios where data is scarce, sensitive, or difficult to collect. Today we have LLMs, especially GPT-4 that can generate domain-specific (textual) data. Many LLMs are even finetuned and benchmarked using synthetic data generated from GPT-4. But, how can you start and what are some of the best practices? Here are some of the key points that you must consider: 1. Define Clear Objectives: Understand the purpose of generating synthetic data. Whether it's for augmenting training datasets, finetuning like I mentioned preserving privacy, or creating data for underrepresented scenarios, clarity in objectives is crucial. 2. Choose the Right Model: Today, there are various LLMs, according to your budget choose them wisely. GPT-4, Claude, Mistral, LLaMA, etc all good but some are expensive and others are free. If you are low on budget and low on requirements then an open-source model with free Kaggle or Colab support will do. But if you are required to generate a huge amount of data then you might need to spend a little on storage. Keep in mind that running open-source LLMs might require you to spend money because they are very large and require extra storage and computing power. 3. Ensure Data Quality: This is where your model will be tested. Synthetic data should mirror the statistical properties of real-world data. Regular validation against real data ensures the synthetic data’s utility and realism. 4. Maintain Privacy: One of the primary benefits of synthetic data is enhanced privacy. Ensure that the generation process and the synthetic data itself do not inadvertently expose sensitive information. In any case research thoroughly and choose the model according to your requirements. I have attached a code snippet that might help you understand how to generate synthetic and also how to design a productive prompt. The link to the original code can be found in the comment section.

19

7 Comments

💥💥💥 LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, Jie Tang Abstract Long-Context Question Answering (LCQA), a challenging task, aims to reason over long-context documents to yield accurate answers to questions. Existing long-context Large Language Models (LLMs) for LCQA often struggle with the "lost in the middle" issue. Retrieval-Augmented Generation (RAG) mitigates this issue by providing external factual evidence. However, its chunking strategy disrupts the global long-context information, and its low-quality retrieval in long contexts hinders LLMs from identifying effective factual details due to substantial noise. To this end, we propose LongRAG, a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to enhance RAG's understanding of complex long-context knowledge (i.e., global information and factual details). We design LongRAG as a plug-and-play paradigm, facilitating adaptation to various domains and LLMs. Extensive experiments on three multi-hop datasets demonstrate that LongRAG significantly outperforms long-context LLMs (up by 6.94%), advanced RAG (up by 6.16%), and Vanilla RAG (up by 17.25%). Furthermore, we conduct quantitative ablation studies and multi-dimensional analyses, highlighting the effectiveness of the system's components and fine-tuning strategies. 👉 https://lnkd.in/d7PaajBQ #machinelearning

1

It's hard to run inference for llms because of their large memory footprint because of all model parameters and activations required at inference time and parallelization is hard because of the autoregressive nature of generating tokens.    So to optimize inference in general you need to lower the memory footprint (use fewer GPUs), compute(FLOPS) and latency.      There are some approaches to make inference efficient(less memory usage and fast):      1. Apply parallelism techniques to run inference across a large number of GPUs      2. Offloading unused data to the CPU and reading it back when needed (less memory usage but high latency)      3. Smart batching strategies like continuous batching (vllm uses this strategy)      4. Compression techniques like quantization, distillation or pruning    5. Architectural optimizations like using PagedAttention (efficiently manages key and value in memory)

14

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

I just published Vol. 59 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Unifying Context Ranking and Generation in Retrieval-Augmented LLMs, from NVIDIA 📚 A Systematic Approach to Identifying Best Practices in Retrieval-Augmented Generation, from Fudan University 📚 Exploring the Mutual Benefits of LLMs and Search Engine Integration, from IEEE Members 📚 Rethinking Ranking Algorithms for Map Interfaces in Airbnb Search, from Airbnb 📚 Reassessing Context Relevance in LLM Question Answering, from USC 📚 A Benchmark for Evaluating Long-Context Language Models and RAG Systems, from Salesforce AI Research 📚 A New Taxonomy for Long-Context Language Model Evaluation, from Bar-Ilan University 📚 A Multi-Stage Content-Centric Framework for Advancing Retrieval-Augmented Generation, from Tsinghua University 📚 Efficient Document Expansion for E-commerce Search, from Walmart 📚 A Comparative Study of RAG vs. Fine-Tuning in API-Driven Code Generation, from Microsoft #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

67

2 Comments

I just published Vol. 65 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction, from Sarmah et al. 📚 A Method for Enhancing Precision in Dense Retrieval for E-commerce Search, from Walmart 📚 Enhancing E-commerce Search with LLM-Powered Attribute Extraction, from Amazon 📚 Enhancing Ad Retrieval with Joint Optimization of Hierarchical Clustering and Neural Networks, from Meta 📚 Integrating Web Search and Knowledge Graphs in Retrieval-Augmented Generation, from Xie et al. 📚 Enhancing BM25 with Entropy-Weighted Similarity and Semantic Understanding, from Li et al. 📚 A Novel GNN Framework for Multimodal and Multilingual Product Recommendations in E-commerce, from Amazon 📚 Human-Guided Optimization of Embedding-Based Retrieval for E-commerce, from Walmart 📚 A Comprehensive Guide to Similarity Measures in Data Science, from Levy et al. 📚 Evaluating Mamba Architecture for Dense Retrieval Tasks, from Renmin University #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

74

2 Comments

I just published Vol. 67 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Comprehensive Study of Noise Types in RAG, from Wu et al. 📚 Lessons from Adobe Express’s Iterative Enhancement of Multi-Modal Search, from Adobe 📚 Dynamic Hard Negative Mining for Enhanced Text Embeddings, from Tencent 📚 Tackling the Unique Challenges of Location-Based Search, from Airbnb 📚 Jina-ColBERT-v2: Enhancing ColBERT for Multilingual Retrieval, from Jha et al. 📚 Enhancing Music Recommendation through Psychology-Informed Session Embeddings, from Deezer 📚 Analyzing [MASK] Token Behavior in ColBERTv2, from RIT 📚 A Critical Evaluation of Traditional Models vs. LLMs in Sequential Recommendation, from Xu et al. 📚 Collaborative Discourse for Complex Information Discovery, from Jiang et al. 📚 A Large-Scale Synthetic Test Collection for Advancing Passage Retrieval Research, from Rahmani et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

34

1 Comment

🚀 I'm excited to share that our paper, "Enhancing E-commerce Product Title Translation with Retrieval-Augmented Generation and Large Language Models," has been accepted at CIKM 24' - Data-centric AI. Thanks for the co-authors Stephan Walter and Taichi Nakatani for the collaboration! 😀 Our study explores how we can improve the machine translation of e-commerce product titles using LLMs, particularly languages that the LLMs do not primarily support. The results of our experiments show that our proposed approach leverages large volumes of bilingual product data, and utilizes a probabilistic retrieval model to retrieve similar bilingual product information for few-shot learning, can improve overall product title translation quality by up to +15.3% in chrF scores. Notably, we observe that language pairs where the LLM has limited proficiency can benefit the most from our method, addressing a common challenge in this domain. As new and more capable LLMs continue to emerge, our RAG-based approach allows for the quick integration of these models (without extensive fine-tuning). By using the large volume of bilingual product data, including titles, bullet points, and descriptions, as few-shot learning examples, our method provides a flexible and adaptable solution that is particularly suitable for industrial applications, where the ability to keep pace with changing requirements is crucial. #amazonscience #LLM #CIKM #RAG #machinetranslation #searchengine #ecommerce #translation #GenerativeAI #search #MultilingualSearch #mixtral #localization #retail

66

2 Comments

I just published Vol. 56 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Leveraging Large Language Models for Synthetic Query Generation in Virtual Assistants, from Apple 📚 Matryoshka Representations for Enhanced Recommender Systems, from HKBU 📚 Towards Cost-Aware and Accurate Query Retrieval via Progressive LLM-Assisted Expansion, from UCR 📚 In-context Learning vs. Fine-tuning: Evaluating LLM Adaptation for Diverse Dialogue Tasks, from UNITN 📚 A Unified Framework for Text-to-Image Generation and Retrieval with Multimodal Large Language Models, from Qu et al. 📚 Asynchronous Learning of User Representations for Large-Scale Recommendation, from Meta 📚 A Comprehensive Survey for Bridging Generative AI and Industrial Recommenders, from Xu et al. 📚 Leveraging Multi-Head Attention for Improved Retrieval Augmented Generation, from Besta et al. 📚 Rectifying Unsuitable Recommendations without Retraining, from Lai et al. 📚 An Efficient MLP-like Architecture for Sequential Recommendation, from Jian et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

45

1 Comment

I just published Vol. 68 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Retrieval-Augmented Generation for Superior Long-Context Question Answering, from NVIDIA 📚 A Near Real-Time Ranking Platform for Modern E-Commerce, from Zalando 📚 A Method for Automated Task-Specific Dataset Creation and Model Fine-Tuning, from Ziegler et al. 📚 A Lightweight Framework for Combining Domain Experts in Information Retrieval, from Lee et al. 📚 Dynamically Determining Compression Rates for Improved RAG Performance, from Zhang et al. 📚 A Comprehensive Study of Pooling and Attention Strategies for Optimizing LLM-Based Embedding Models, from HKUST 📚 Unbiasing Short-Video Recommenders through Causal Inference, from Dong et al. 📚 Non-Parametric Embedding Fine-Tuning for Improved k-NN Retrieval, from Zeighami et al. 📚 A High-Performance Batch Query Architecture for Industrial-Scale Recommendation Systems, from Zhang et al. 📚 Correcting Stale Embeddings for Computationally Efficient Dense Retrieval, from Google DeepMind #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

37

1 Comment

I just published Vol. 54 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Graph Retrieval-Augmented Generation for Context-Aware Reasoning, from Emory University 📚 Enabling Fine-Grained Ranking at Varying Granularities via Multi-Vector Embeddings, from Reddy et al. 📚 Learning to Ask Clarification Questions for Ambiguous Retrieval Queries, from UC Berkeley 📚 Principled and Practical Multi-Vector Retrieval with Approximation Guarantees, from Google Research 📚 Towards User-Centric Recommendations via LLM-Based Simulation and Tool Learning, from USTC 📚 Building Pseudo-Graph Databases from Raw Texts for Retrieval-Augmented Language Models, from Renmin University 📚 Retrieval-Augmented Early Exiting for Efficient Large Language Model Inference, from Huang et al. 📚 A State-of-the-Art Generalist Embedding Model, from NVIDIA 📚 Enhancing Multilingual Language Model Training via Repeat Ranking, from Devine et al. 📚 A Real-World Evaluation of HNSW for Vector Retrieval, from Marqo #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

46

2 Comments

Trying to reduce costs of your LLMs by 2x? Use a small highly optimized "router" LLM to decide if you need a heavy or a light LLM based on the context. Simple questions go to the light LLM, tougher ones to the heavy LLM. Most surprising finding: the router LLM generalizes across different heavy and light LLMs -- no fine tuning required! (Also this is my first academic paper in 20 years -- it was a lot of fun working with Amjad Almahairi on this)

112

6 Comments