High-quality data and robust data processing and harmonization pipelines are essential when creating AI-driven predictive models. Such data pipelines enable aligning data from different sources thus facilitating development of accurate predictive models without having to create single data lakes.

However, different data sources may lack homogeneity in data representation, and some data representations are not suited for ML and AI modeling. Instead of breaking down data silos and creating a single data lake, we adopted the strategy of applying common preprocessing using unified data pipelines. This approach ensures consistency and comparability across datasets, resulting in harmonized, ML-ready datasets optimized for training models for predictive retrosynthesis and yield prediction.

I will describe my experience and best practices of such reaction data preprocessing.

In a recent customer project aimed at reducing synthesis failures in the lab, our goal was to create a predictive model capable of recommending favorable conditions for desired chemical transformations. We leveraged Elsevier's ML-optimized reaction dataset, compiled from published literature and patents, which primarily represents successful transformations with high yields. Consequently, we faced the challenge of underrepresentation of failed reactions. Such reactions can typically be found in company Electronic Laboratory Notebooks (ELNs) but are rarely covered in reaction collections extracted from publications.

By synchronizing these diverse datasets using our data preprocessing pipeline, we successfully unlocked the potential of disparate data sources. The result is a powerful predictive model capable of recommending optimal reaction conditions and reducing synthesis failures.