Pile (Concert)

Wed Oct 18 2023 at 12:00 am to 02:00 am

Black Cat | Washington

Pile (Concert)
Advertisement
Recent work has demonstrated that increased training dataset diversity
improves general cross-domain knowledge and downstream generalization
capability for large-scale language models. With this in mind, we present
\textit{the Pile}: an 825 GiB English text corpus targeted at training
large-scale language models. The Pile is constructed from 22 diverse
high-quality subsets -- both existing and newly constructed -- many of which
derive from academic or professional sources. Our evaluation of the untuned
performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on
many of its components, such as academic writing. Conversely, models trained on
the Pile improve significantly over both Raw CC and CC-100 on all components of
the Pile, while improving performance on downstream evaluations. Through an
in-depth exploratory analysis, we document potentially concerning aspects of
the data for prospective users. We make publicly available the code used in its
construction.
Elevate your expectations on 18 Oct, 2023 at 12:00 AM. Pile is more than an event – it's an opportunity to immerse yourself in a world of Rock excellence. From enriching workshops to awe-inspiring performances, this Concert at Black Cat in , promises to elevate every facet of your experience. Don't miss your chance to be part of something extraordinary with tickets starting at 29.00 .
Advertisement

Event Venue & Nearby Stays

Black Cat, 1811 14th St NW, Washington DC, DC 20009, Washington, United States

Tickets

Sharing is Caring: