X-VS: Crossbar-based processing-in-memory architecture for video summarization

Nagadastagiri Challapalle, Makesh Chandran, Sahithi Rampalli, Vijaykrishnan Narayanan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Video summarization techniques identify the most interesting frames in a video based on their uniqueness/importance or relevance to a user query. Deep learning based automated video summarization techniques have gained significant importance due to the growing need to analyze the exploding video data from user devices, surveillance cameras, and social media platforms etc. In contrast to the image classification, object detection tasks which predominantly use convolutional neural networks (CNNs), video summarization techniques comprise a pipeline of more diverse networks such as text processing networks, attention and content similarity mechanisms. In this work, we present X-VS, a ReRAM processing-in-memory (PIM) hardware accelerator architecture for video summarization workloads. We augment a baseline ReRAM CNN accelerator with a systolic array-based crossbar architecture to incorporate efficient support for recurrent neural networks, attention and content similarity mechanisms and hash-based word embedding lookup to support the video summarization networks. The proposed architecture achieves an average speedup of '450x, and energy savings of '1600x for two state-of-the-art video summarization networks over CPU and GPU implementations.

Original languageEnglish (US)
Title of host publicationProceedings - 2020 IEEE Computer Society Annual Symposium on VLSI, ISVLSI 2020
PublisherIEEE Computer Society
Pages592-597
Number of pages6
ISBN (Electronic)9781728157757
DOIs
StatePublished - Jul 2020
Event19th IEEE Computer Society Annual Symposium on VLSI, ISVLSI 2020 - Limassol, Cyprus
Duration: Jul 6 2020Jul 8 2020

Publication series

NameProceedings of IEEE Computer Society Annual Symposium on VLSI, ISVLSI
Volume2020-July
ISSN (Print)2159-3469
ISSN (Electronic)2159-3477

Conference

Conference19th IEEE Computer Society Annual Symposium on VLSI, ISVLSI 2020
CountryCyprus
CityLimassol
Period7/6/207/8/20

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture
  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'X-VS: Crossbar-based processing-in-memory architecture for video summarization'. Together they form a unique fingerprint.

Cite this