news

A quick dive into Python’s “__slots__”

For my first Medium article, we’re going to go into a quick and easy way to speed up your Python code (and pass those pesky HackerRank tests where you’re just a bit short on time!), as well as some of the technical implementation details for the curious. For instances of this class, you can use self.x and self.y in the same ways as a normal class instance. However, one key difference between this and instancing from a normal class is that you cannot add or remove attributes from this class’ instances.
Read more

Optimal Shard Placement in a Petabyte Scale Elasticsearch Cluster

The number of shards on each node, and tries to balance the number of shards per node evenly across the clusterThe high and low disk watermarks. Elasticsearch considers the available disk space on a node before deciding whether to allocate new shards to that node or to actively relocate shards away from that node. A nodes that has reached the low watermark (i.e 80% disk used) is not allowed receive any more shards.
Read more

Embracing failures and cutting infrastructure costs: Spot instances in Kubernetes

Spot Instances are unused servers that are available for less than the regular price. Therefore, you can significantly save on your infrastructure costs. It does come with a price, though. Your cloud provider can take away your spot instance at any time, and give to another client who has requested it at a standard cost. How can you save money, but work around disappearing servers? Learn how you can leverage Kubernetes to self-heal your infrastructure and cut costs with Spot Instances.
Read more

GraphQL: A success story for PayPal Checkout

At PayPal, we recently introduced GraphQL to our technology stack. At PayPal, GraphQL has been a complete game changer to the way we think about data, fetch data and build applications. This blog post takes a close look at PayPal Checkout and explains our journey from REST to Batch REST to GraphQL and lessons learned along the way. PayPal’s Checkout products spread across many web and mobile apps, supporting millions of users across ~200 countries and has hundreds of experiments running at any time.
Read more

Cross shard transactions at 10 million requests per second

Dropbox stores petabytes of metadata to support user-facing features and to power our production infrastructure. The primary system we use to store this metadata is named Edgestore and is described in a previous blog post, (Re)Introducing Edgestore. In simple terms, Edgestore is a service and abstraction over thousands of MySQL nodes that provides users with strongly consistent, transactional reads and writes at low latency. Edgestore hides details of physical sharding from the application layer to allow developers to scale out their metadata storage needs without thinking about complexities of data placement and distribution.
Read more

Running integration tests in Kubernetes

Linux containers changed the way that we run, build and manage applications. As more and more platforms become cloud-native, containers are playing a more important role in every enterprise’s infrastructure. Kubernetes is currently the most well-known solution for managing containers, whether they run in a private, a public or a hybrid cloud. With a container application platform, we can dynamically create a whole environment to run a task that is discarded afterwards.
Read more

Development and Debugging with Kubernetes

I’m working with an osx laptop, and in this article, I will show how to quickly create a dev/test kubernetes cluster, and the usage of some useful tools for working and debugging with Kubernetes. Source: itnext.io

29 Docker security tools compared

There are quite a few Docker security tools in the ecosystem, how do they compare? We have gathered a list of the most popular Docker security tools so you can evaluate what fits your needs better, including features and use cases. Here you will find both open source projects and Docker security commercial vendors. But is Docker insecure? Not at all. Actually features like process isolation with user namespaces and resource encapsulation with cgroups reduce the attack vector providing a great deal of protection.
Read more

Learning Concepts with Energy Functions

We’ve developed an energy-based model that can quickly learn to identify and generate instances of concepts, such as near, above, between, closest, and furthest, expressed as sets of 2d points. Our model learns these concepts after only five demonstrations. We also show cross-domain transfer: we use concepts learned in a 2d particle environment to solve tasks on a 3-dimensional physics-based robot. Many hallmarks of human intelligence, such as generalizing from limited experience, abstract reasoning and planning, analogical reasoning, creative problem solving, and capacity for language require the ability to consolidate experience into concepts, which act as basic building blocks of understanding and reasoning.
Read more

Scaling Time Series Data Storage—Part II

In January 2016 Netflix expanded worldwide, opening service to 130 additional countries and supporting 20 total languages. Later in 2016 the TV experience evolved to include video previews during the browsing experience. More members, more languages, and more video playbacks stretched the times series data storage architecture from part 1 close to its breaking point. In part 2 here, we will explore the limitations of that architecture and describe how we’re re-architecting for this next phase in our evolution.
Read more