In January 2016 Netflix expanded worldwide, opening service to 130 additional countries and supporting 20 total languages. Later in 2016 the TV experience evolved to include video previews during the browsing experience. More members, more languages, and more video playbacks stretched the times series data storage architecture from part 1 close to its breaking point.
In part 2 here, we will explore the limitations of that architecture and describe how we’re re-architecting for this next phase in our evolution. Part 1’s architecture treated all viewing data the same, regardless of type (full title plays vs video previews) or age (how long ago a title was viewed). The ratio of previews to full views was growing rapidly as that feature rolled out to more devices.
By the end of 2016 we were seeing 30% growth in one quarter for that data store; video preview roll-outs were being delayed because of their potential impact to this data store. The naive solution would be to scale the underlying viewing data Cassandra (C*) cluster to accommodate that growth, but it was already the biggest cluster in use and nearing cluster size limits that few C* users have gone past successfully. Something had to be done, and that too soon.
We challenged ourselves to rethink our approach and design one that would scale for at least 5x growth. We had patterns that we could reuse from part 1’s architecture, but by themselves those weren’t sufficient. New patterns and techniques were needed.
We started by analyzing our data set’s access patterns. What emerged was three distinct categories of data: Full title playsVideo preview playsLanguage preference (i.e., which subtitles/dubs were played, indicating what is the member’s preference when they play titles in a given language) Language preference (i.e., which subtitles/dubs were played, indicating what is the member’s preference when they play titles in a given language)
Source: medium.com