Uber's Michelangelo vs. Netflix's Metaflow

-
  Uber's Michelangelo vs. Netflix's Metaflow Michelangelo Pain point Without michelangelo, each team at uber that uses ML (that’s all of them - every interaction with the ride or eats app involves ML) would need to build their own data pipelines, feature stores, training clusters, model storage, etc.  It would take each team copious amounts of time to maintain and improve their systems, and common patterns/best practices would be hard to learn.  In addition, the highest priority use cases (business critical, e.g. rider/driver matching) would themselves need to ensure they have enough compute/storage/engineering resources to operate (outages, scale peaks, etc.), which would results in organizational complexity and constant prioritization battles between managers/directors/etc. Solution Michelangelo provides a single platform that makes the most common and most business critical ML use cases simple and intuitive for builders to use, while still allowing self-serve extensibi...
Post a Comment

Paxos vs. Blockchain: A User's Perspective

-
A robust distributed system can tolerate partial failures in the system - that means that the system should continue to work as expected even if parts of it are failing.  There are 3 main 'partial failures' that the field of Distributed Systems tries to solve for:

1.) Out-of-sync system clocks
2.) Process pauses
3.) RPC requests with no response

Paxos is one algorithm that tries to solve for these by using consensus among nodes to decide on the sequence and order of events, but system administrators must tune the timeout parameter to find the best middle ground between waiting too long for a node (one computer in the network of systems) and not waiting long enough.

The blockchain algorithm gets around the idea of time entirely by deciding on an order of events once, and then getting as many nodes as possible to agree and persist the actual ordering itself.  Here's an excerpt straight from Satoshi's paper itself:

The solution we propose begins with a timestamp server. A timestamp server works by taking a hash of a block of items to be timestamped and widely publishing the hash, such as in a newspaper or Usenet post. The timestamp proves that the data must have existed at the time, obviously, in order to get into the hash. Each timestamp includes the previous timestamp in its hash, forming a chain, with each additional timestamp reinforcing the ones before it.

For those interested in advanced Distributed Systems concepts, learning about blockchain is a great way to expand your knowledge and play with the latest ideas in consensus.

Comments

Post a Comment

Popular posts from this blog

ChatGPT - How Long Till They Realize I’m a Robot?

-
Image
I tried it first on December 2nd... ...and slowly the meaning of it started to sink in. It's January 1st and as the new year begins, my future has never felt so hazy. It helps me write code. At my new company I'm writing golang, which is new for me, and one day on a whim I think "hmmm maybe ChatGPT will give me some ideas about the library I need to use." Lo-and-behold it knew the library. It wrote example code. It explained each section in just enough detail. I'm excited....It assists my users. I got a question about Dockerfiles in my teams oncall channel. "Hmmm I don't know the answer to this either"....ChatGPT did. It knew the commands to run. It knew details of how it worked. It explained it better and faster than I could have. Now I'm nervous....It writes my code for me. Now I'm hearing how great Github Copilot is - and it's built by OpenAI too...ok I guess I should give it a shot. I install it, and within minutes it'...
Read more

Architectural Characteristics - Transcending Requirements

-
Image
Building a system means meeting a set of requirements dictated by the customer.  But the customer isn't always going to translate what they want into engineering terms.  Even if you say please, they might not even know how.  If they ask for an online ordering system, they probably won't specify that it needs to be available 24/7 and auditable for tax purposes.  Yet these aspects could mean the difference between project success and failure.  Those unsaid aspects are called anything from 'nonfunctional requirements' to 'Architectural Characteristics'. In their book 'Fundamentals of Software Architecture', Ford and Richards define Architectural Characteristics to have 3 criteria: Specifies a nondomain design consideration Let's dive into these bullets to better understand them. Aspects of Architectural Characteristics Specifies a nondomain design consideration If you're building an ordering system, then order numbers, customers, items and prices ar...
Read more

Laws of Software Architecture

-
Image
As the discipline of software engineering matures, few things remain constant.  A few years ago, a large portion of the community thought that TDD was always the best methodology to use - after all, you move faster by being thorough and preventing bugs.  Nowadays it's clear that unit tests do not ensure a successful project - when you're A/B testing new features or products and you're not even sure if your code will be around 3 months from now, it's better to churn out production code and test it in prod.  You can clean up the mess later, if the business value makes it worth it. But despite the pace of change in software engineering , in software architecture  there exist a set of 'laws' which don't change over time and which apply to all architectural endeavors.  I'd like to talk about 2 of them in this post as they are important ideas to keep in the back of your mind as your skills grow past coding, past design and into architecture. Laws of Software A...
Read more