Ask Sendai

What are BlackHole and Passthrough clusters? Understanding, controlling and securing your external service access is one of the key benefits that you get from a service mesh like Istio. From a security and operations point of view, it is critical to monitor what external service traffic is getting blocked as they might surface possible misconfigurations or a security vulnerability if an application is attempting to communicate with a service that it should not be allowed to.

Before delving into our specifics, I want to take a moment to discuss the technical stack backing our pipeline. Our platform uses a mixture of Spark and Hive jobs. Our core pipeline is primarily implemented in Scala. However, we leverage Spark SQL in certain contexts. We leverage YARN for job scheduling and resource management, and execute our jobs on Amazon EMR. We use Airflow as our task orchestration system that takes care of the orchestration logic.

Mayank Bansal and Min Cai present Peloton, a Unified Resource Scheduler for collocating heterogeneous workloads in shared Mesos clusters. Its goal is to manage compute resources more efficiently while providing hierarchical max-min fairness guarantees for different teams. Peloton schedules large-scale batch jobs with millions of tasks and supports distributed TensorFlow jobs with thousands of GPUs. Source: infoq.com

Traditional banks make extensive use of labor-intensive, human-centric control structures such as Production Support groups, Security Response teams, and Contingency Planning organizations. These control structures were deemed necessary in order to segment responsibilities and to maintain a security posture that is risk averse. Unfortunately, this traditional model tends to keep the subject matter experts in these organizations at a distance from the development teams, reducing efficiency and getting in the way of innovation.

Hiya folks, Brian ‘Penrif’ Bossé, your local friendly Tech Lead of League here. I’m taking some time in between matches of TFT to wax philosophic about game engines and how we on League make decisions around what direction to take our custom game engine. Join me on a moderately long look at one dimension of game engine design, where League currently exists on that dimension, and where we’re taking the game from there.

Earlier this year, the Streaming PubSub team at Lyft got multiple Apache Kafka clusters ready to take on load that required 24/7 support. The team’s operational burden for Kafka quickly started heading towards burn-out territory. On-call rotations started getting miserable because we’d get woken up at night due to failing hosts. Business requirements kept coming and requiring us to scale the clusters further. The more we scaled, the more we’d get woken up.

Document understanding is the practice of using AI and machine learning to extract data and insights from text and paper sources such as emails, PDFs, scanned documents, and more. In the past, capturing this unstructured or “dark data” has been an expensive, time-consuming, and error-prone process requiring manual data entry. Today, AI and machine learning have made great advances towards automating this process, enabling businesses to derive insights from and take advantage of this data that had been previously untapped.

On September 4th, 2019, Containous, a cloud infrastructure software provider, released Maesh, an open-source service mesh written in Golang and built on top of the reverse proxy and load balancer Traefik. Maesh promises to provide a lightweight service mesh solution that is easy to get started with and to roll out across a microservice application. Source: infoq.com

Google recently released research about creating a tool for searching Similar Medical Images Like Yours (SMILY). The research uses embeddings for image-based search and allows users to influence the search through the interactive refinement of concepts. Source: infoq.com

With the acquisition of startup Semmle, GitHub aims to make continuous vulnerability detection part of their continuous integration/continuous deployment service. Source: infoq.com