Advertisement
Large Language Models (LLMs), despite their recent impressive accomplishments, are notably cost-prohibitive to deploy, particularly for applications involving long-content generation, such as dialogue systems and story writing. Often, a large amount of transient state information, referred to as the KV cache, is stored in GPU memory in addition to model parameters, scaling linearly with the sequence length and batch size. In this paper, we introduce a novel approach for implementing the KV cache which significantly reduces its memory footprint. Our approach is based on the noteworthy observation that a small portion of tokens contributes most of the value when computing attention scores. We call these tokens Heavy Hitters (H$_2$). Through a comprehensive investigation, we find that (i) the emergence of H$_2$ is natural and strongly correlates with the frequent co-occurrence of tokens in the text, and (ii) removing them results in significant performance degradation. Based on these insights, we propose Heavy H?️ Mark your calendar from May 10, 2024 to May 10, 2024 at White Oak Music Hall - Upstairs. Heavy//Hitter is a celebration of passion, innovation, and community. Embrace the opportunity to explore the world of in a dynamic setting in Texas, HARD ROCK / METAL. Don't miss out on this remarkable event ?.
Advertisement
Event Venue & Nearby Stays
White Oak Music Hall - Upstairs, White Oak Music Hall - Upstairs, Houston, United States of America
Tickets