Posts

Uber's Michelangelo vs. Netflix's Metaflow

  Uber's Michelangelo vs. Netflix's Metaflow Michelangelo Pain point Without michelangelo, each team at uber that uses ML (that’s all of them - every interaction with the ride or eats app involves ML) would need to build their own data pipelines, feature stores, training clusters, model storage, etc.  It would take each team copious amounts of time to maintain and improve their systems, and common patterns/best practices would be hard to learn.  In addition, the highest priority use cases (business critical, e.g. rider/driver matching) would themselves need to ensure they have enough compute/storage/engineering resources to operate (outages, scale peaks, etc.), which would results in organizational complexity and constant prioritization battles between managers/directors/etc. Solution Michelangelo provides a single platform that makes the most common and most business critical ML use cases simple and intuitive for builders to use, while still allowing self-serve extensibi...

Grow Your Career, Be A Senior Engineer

Image
My personal experience with navigating a career in software engineering has been dotted by fits and starts.  9 years into my professional life I can look back and see what worked, what didn't, and why.  I want to share that knowledge so other young engineers can do more with their first 9 years than I did.  Plus it's just fun to reminisce and will help me visualize the next steps in my career. To be clear, the north star goal in my career is and has always been to learn and grow towards more positive impact on those I serve.  If you have the same or a similar goal, this article will be useful. I wrote up a simple formula which I think can help any aspiring engineers out there.  It can be used as a template for a career plan.  When you're thinking about how to get to the next level, make sure you're putting equal emphasis on all these categories.   1.) ACTION from you, despite imperfect information 2.) COWORKERS whom you respect 3.) MENTOR(s) inves...

The Terms that Blind Us

Image
Blacklist.  Master-Slave Architecture.  Black Hat.  These terms helped build the foundations of software development as we know it today.  But do they support the narrative that black is bad and white is good?  And that having masters and slaves is ok?  Is it a racist act to continue to use them? Let's first address the 'rolling my eyes' crowd.  It's true that, given the power of God, of all the things to change with a snap of your fingers, this one wouldn't be the first.  Let's describe, to the best of our ability, the argument that we should continue using these terms: "these terms were not introduced with the intent to hurt people or make black people think they're less-than.  They're just part of a common vocabulary, and having that is important to the field of software development.  We need to be able to communicate with each other efficiently, and if we don't use the words that are already well-understood by most engineers/develope...

Why are Distributed Systems So Hard?

Image
Isn't it true you can just write deterministic code and if you do it right and work to fix all the bugs, eventually you'll have a simple that never does the wrong thing?  If that's not the case then why not?  Computers are deterministic - they're predictable and they only ever do exactly what you ask them to do and nothing more...right? It's complicated. On a single node (computer) most failures mean the entire node completely stops working - for example the power supply fails, the disk dies, or the motherboard gets fried.  These types of failures are easy to detect because the node won't be in a 'partially failed' state where sometimes it performs the functions it's asked and sometimes it doesn't. In a network, you have multiple nodes - possibly hundreds or thousands.  When one node fails, it is impossible to be 100% sure that node is completely down.  And if you can't be sure that a node won't come back to life, then you can...

Cluster Management at LinkedIn

Image
In 2014 LinkedIn released a cluster management solution called Helix. Helix solves some problems that arise when a system scales to be too large to manage even on just a few hosts. A successful system will start to go through a few transition states that, when large enough, will become frequent enough to require an automated solution. First, your system will become too large to host on a single machine. So now you need to shard it. Then your system will either have hosts fail once in a while, or some shards might start getting too big or taking too much load. So then you can start using replication to solve for that. As your cluster grows, the average size of shards also grows - sometimes you’ll have to split shards because they become too big, or more broadly redistribute. So now you need something to allow for that. Partitioning/sharding, fault tolerance and scalability - these are the higher level concepts just described, and the problems Helix solves for. If you can sol...

The Curious Case of the Document Database

Image
Let’s talk about an oft-overlooked NoSQL database type. It’s got all the best parts of a k-v store and allows limited SQL-like query-ability too! They’re called document databases and they’re all the rage. A document database stores objects that can be serialized to JSON or some similar serialization format. These JSON ‘documents’ are keyed by an ID, similar to how k-v stores work. When you want to fetch an entire document, all you need is the key for it. But the magic of document databases allows you to fetch only pieces of a document and also to fetch data from multiple documents using selection criteria that mirrors basic SQL query functionality. This is all made possible by the tree-like structure that documents in a doc DB must conform to. JSON data can contain keyed fields, nested structures, and lists. Using this structure a doc DB can extract specific pieces of a doc so that the entire doc doesn’t have to be returned and parsed in the application layer. Query-ability...

Consistency in Redis

Image
Most uses of Redis will focus more on latency and availability rather than consistency - that’s because at its core, Redis is essentially a cache. Generally speaking, you store things in Redis in memory and you update or read them extremely quickly. You need to make sure that the cache is always available, so in most cases you’d only choose Redis if you’re leaning towards an A class system (A for Availability) rather than a C class system (Consistency). However, it’s important to know that a replicated instance of Redis is capable of giving you different levels of consistency up to and including read-after-write consistency - the kind of consistency that guarantees data reads from anywhere that happen after a successful response to a write will receive that write. Even if the read goes to a different replica than the write did. What Redis can’t give you is linearizability - the guarantee that any set of observers of the system will only be able to see a single copy of the syste...

CAP Theorem Explained

Image
When building large-scale software systems today, you have to make tradeoffs.  You can't have an ACID compliant data store with infinite storage/throughput/connections that's always available in any part of the world with super low latency where clients can read/write concurrently without any risk of inconsistencies that's free.  If you could, the problem would be solved and our industry could go build spaceships at SpaceX or retire and make sourdough every Sunday. Instead, we need to make tradeoffs.  Does our product/system need ACID semantics?  Is latency more important?  Can we allow certain types of data inconsistencies for a short time in favor of availability?  How much are we able to spend so that we don't have to sacrifice as much? These are some questions that everyone building a large-scale software system has to grapple with in the design phase.  A great way to begin your thinking is using CAP Theorem - or at least what it's slowly be...