The Technology Upskill Mess For Application Developers
It’s been a few years since Oliver kicked off his career, as a full-stack Java developer, in this BigCorp. Inc. He learned quite a bit, primarily working on the backend: writing features, plumbing APIs, debugging code, fixing bugs, DENTTAL (Documentation, Exception Handling, Null pointers, Time complexity, Test coverage, Analysis of code…
Low Code: The Only Article You Will Ever Need
What Is Low Code? Low code tools and platforms, just like no code, offer visual drag and drop builders along with pre-built components and features aiming to build applications fast. Compared with no code, low code tools provide developers access to the code they generate to enable them to customize…
No Code: A Profound Insight For Citizen and Professional Developers
The no code movement is quite picking up steam these days, as you might know. Also, the no code tools, especially the full-stack no code platforms, have come across as a silver bullet, claiming to empower a citizen developer to develop an app like Instagram, at the scale of Instagram in just a few weeks as…
Distributed Systems and Scalability Feed #1 – Heroku Client Rate Throttling, Tail Latency and more
What does 100 million users on a Google service mean? 10 billion requests/day100k requests/second (average)200k requests/second (peak)2 million disk seeks per second (IOPS) How are these served?100 million users need (at peak) 2 million IOPS at 100 IOPS per disk, that’s 20k disk drives at…
Zero to Software/Application Architect – Learning Track
Zero to Software/Application Architect learning track is a series of four courses that I am writing (2 courses published) with an aim to educate you, step by step, on the domain of software architecture & distributed system design. The learning track takes you right from…
Distributed Systems and Scalability Feed
Facebook photo storage architecture
Facebook built Haystack, an object storage system designed for storing photos on a large scale. The platform stores over 260 billion images which amounts to over 20 petabytes of data. One billion new photos are uploaded each week which is approx—60 terabytes of data. At peak, the platform serves over one million images per second.
In the original NAS-based photo storage architecture, Facebook faced throughput and latency issues as the photos and the associated metadata lookups in NAS caused excessive disk operations almost upto ten just for retrieving a single image.
Tail latency in distributed systems
Tail latency is that tiny percentage of responses from a system that are the slowest in comparison to most of the responses. They are often called as the 98th or 99th percentile response times. This may seem insignificant at first but for large applications like LinkedIn, this has noticeable effects. This could mean that for a page having a million views per day 10,000 of those page views would experience the delay. Read how LinkedIn deals with longtail network latencies.
There can be multiple causes of tail latency: increasing load on the system, complex and distributed systems, application bottlenecks, slow network, slow disk access and more. Read more on it.
RobinHood: Tail latency-aware caching
RobinHood is a research caching system for application servers in large distributed systems having diverse backends. The cache system dynamically partitions the cache space between different backend services and continuously optimizes the partition sizes.
Microsoft research has a talk on getting rid of long-tail latencies.
> Spotify Engineering: From Live to Recording
> Ingesting LIVE video streams at a global scale at Twitch
> $64,944 spent on AWS, to support 25,000 customers, in August by ConvertKit.
> Read how Storytel engineering computes customer consumption of books transitioning from batch processing to streaming bookmarks data with Apache Beam and Google Cloud.
> How Pokemon Go scales to millions of requests per second?
> Insight into how Grab built a high-performance ad server.
SUBSCRIBE TO MY NEWSLETTER to be notified of new additions to the list. Fortnight/monthly emails.
Looking for developer, software architect jobs? Try Jooble. Jooble is a job search engine created for a single purpose: To help you find the job of your dreams!!
- State of Backend #2 – Disney+ Hotstar Replaced Redis and Elasticsearch with ScyllaDB. Here’s Why.
- State of Backend #1- Distributed Task Scheduling with Akka, Kafka and Cassandra
- Live Video Streaming Infrastructure at Twitch
- Web Application Architecture Explained With a Real-World Example
- Wide-column Database, Column Databases – A Deep Dive