Google DeepMind Develops Generative Data Refinement for AI Training

Researchers at Google DeepMind developed Generative Data Refinement (GDR), a method designed to mitigate the shortage of high-quality training data for large language models. The technique employs pretrained generative models to rewrite and purify data that would typically be discarded, such as text containing obsolete facts or personally identifiable information like Social Security numbers.

By isolating and cleaning small sections of toxic or private content, GDR allows AI labs to retain the rest of the document for training. Minqi Jiang, a researcher on the project, noted that the method is superior to using synthetic data, which can cause model collapse. Jiang stated that the approach outperforms current industry standards for data purification.

While the initial proof of concept focused on coding and text, the researchers indicated that the GDR method could eventually be expanded to include audio and video modalities.