Posts

Showing posts from June, 2019

Uber's Michelangelo vs. Netflix's Metaflow

  Uber's Michelangelo vs. Netflix's Metaflow Michelangelo Pain point Without michelangelo, each team at uber that uses ML (that’s all of them - every interaction with the ride or eats app involves ML) would need to build their own data pipelines, feature stores, training clusters, model storage, etc.  It would take each team copious amounts of time to maintain and improve their systems, and common patterns/best practices would be hard to learn.  In addition, the highest priority use cases (business critical, e.g. rider/driver matching) would themselves need to ensure they have enough compute/storage/engineering resources to operate (outages, scale peaks, etc.), which would results in organizational complexity and constant prioritization battles between managers/directors/etc. Solution Michelangelo provides a single platform that makes the most common and most business critical ML use cases simple and intuitive for builders to use, while still allowing self-serve extensibi...

Paxos vs. Blockchain: A User's Perspective

A robust distributed system can tolerate partial failures in the system - that means that the system should continue to work as expected even if parts of it are failing.  There are 3 main 'partial failures' that the field of Distributed Systems tries to solve for: 1.) Out-of-sync system clocks 2.) Process pauses 3.) RPC requests with no response Paxos is one algorithm that tries to solve for these by using consensus among nodes to decide on the sequence and order of events, but system administrators must tune the timeout parameter to find the best middle ground between waiting too long for a node (one computer in the network of systems) and not waiting long enough. The blockchain algorithm gets around the idea of time entirely by deciding on an order of events once, and then getting as many nodes as possible to agree and persist the actual ordering itself.  Here's an excerpt straight from Satoshi's paper itself: The solution we propose begins with a timestamp...

The Best Intro to Distributed Systems

I recently switched teams at Facebook.  I went from working on user-facing search technologies to running the distributed systems that allow other teams at my company to move fast.  To be clear, I used to bake the break, now I build the bakers an oven!  Although I had some background in these systems, things change quickly and the field is deep with decades of knowledge to pull from and build upon.  I needed a refresher! I found one - a friend recommended a book called 'Designing Data Intensive Applications'.  It sounded a bit off-topic - the title didn't have 'distributed' or 'systems' in it!  Turns out it gave me exactly what I needed.  DDIA is the best introduction and deep dive into all the areas of expertise that the DS specialty has to offer - and wow is this field deep!  The book requires a bit of background - you'll need to have a bachelor's in CPSC or similar experience, as the first few chapters build on that knowledge.  Once the ...