Name: Pile
Start: 2024-09-28T15:00:00-05:00
End: 2024-09-28T20:00:00-05:00
Location: Bottlerocket Social Hall

Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present \textit{the Pile}: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations. Through an in-depth exploratory analysis, we document potentially concerning aspects of the data for prospective users. We make publicly available the code used in its construction.
🌟 Prepare for an extraordinary experience from September 28, 2024 to September 28, 2024 at Bottlerocket Social Hall. Join us for Pile, a mesmerizing event that promises to redefine your expectations. From captivating presentations to interactive showcases, this CONCERTS in Pennsylvania, POP / ROCK is designed to leave you inspired and amazed. Join us for this incredible event 🚀. http://maps.seatics.com/GeneralAdmissionEvent_2021-07-08_1018_07062023_125834_SVGC_tn.png