Uber's Michelangelo vs. Netflix's Metaflow

-
  Uber's Michelangelo vs. Netflix's Metaflow Michelangelo Pain point Without michelangelo, each team at uber that uses ML (that’s all of them - every interaction with the ride or eats app involves ML) would need to build their own data pipelines, feature stores, training clusters, model storage, etc.  It would take each team copious amounts of time to maintain and improve their systems, and common patterns/best practices would be hard to learn.  In addition, the highest priority use cases (business critical, e.g. rider/driver matching) would themselves need to ensure they have enough compute/storage/engineering resources to operate (outages, scale peaks, etc.), which would results in organizational complexity and constant prioritization battles between managers/directors/etc. Solution Michelangelo provides a single platform that makes the most common and most business critical ML use cases simple and intuitive for builders to use, while still allowing self-serve extensibi...
Post a Comment

How to Frame Metric Collection

-



Depending on the type of software development you’re doing, it can be tough to figure out what metrics you need to collect. An iterative process (aka fancy words for trial-and-error) will work to get you to where you need to be eventually, but along the way the MTTR of outages will suffer and you might lose users/revenue.

There’s a simple way to think about metrics that will help you build an intuition on what to monitor and what to measure. It’s this:

Measure the business, measure the software. But don’t conflate the two.

Measuring the business is critical to being able to notify and escalate to the proper personnel when there is an outage. Business metrics include things that impact your bottom line - user signins per minute, items added to the cart per second, items sold per day. Anything that directly and immediately affects customers is a business metric.

Software metrics are signals about how your software is running. There are 3 categories of software metrics: OS metrics, generic server metrics and application metrics. OS metrics are things that can be measured at the OS level without knowing anything about the process(es) running into top of the OS - CPU, memory, network connections, etc. These will allow you to tune the software to measure performance and are necessary for debugging the hardiest of issues. Generic server metrics are things you’d be able to collect from any web server, application container, DB, message queue, cache server etc. - things like web requests per second, DB transactions per second or cache hit ratio. And lastly, application metrics are things you can collect that are specific to your application - whatever you want to publish to tell you the state and/or current operations of your application - it could mean you measure how long it took one important function to run per request.

This is a good place to start when creating a holistic set of metrics to monitor your system, and if you start here, you’ll have a good chance at getting all the details right as you go.

Comments

Post a Comment

Popular posts from this blog

ChatGPT - How Long Till They Realize I’m a Robot?

-
Image
I tried it first on December 2nd... ...and slowly the meaning of it started to sink in. It's January 1st and as the new year begins, my future has never felt so hazy. It helps me write code. At my new company I'm writing golang, which is new for me, and one day on a whim I think "hmmm maybe ChatGPT will give me some ideas about the library I need to use." Lo-and-behold it knew the library. It wrote example code. It explained each section in just enough detail. I'm excited....It assists my users. I got a question about Dockerfiles in my teams oncall channel. "Hmmm I don't know the answer to this either"....ChatGPT did. It knew the commands to run. It knew details of how it worked. It explained it better and faster than I could have. Now I'm nervous....It writes my code for me. Now I'm hearing how great Github Copilot is - and it's built by OpenAI too...ok I guess I should give it a shot. I install it, and within minutes it'...
Read more

Architectural Characteristics - Transcending Requirements

-
Image
Building a system means meeting a set of requirements dictated by the customer.  But the customer isn't always going to translate what they want into engineering terms.  Even if you say please, they might not even know how.  If they ask for an online ordering system, they probably won't specify that it needs to be available 24/7 and auditable for tax purposes.  Yet these aspects could mean the difference between project success and failure.  Those unsaid aspects are called anything from 'nonfunctional requirements' to 'Architectural Characteristics'. In their book 'Fundamentals of Software Architecture', Ford and Richards define Architectural Characteristics to have 3 criteria: Specifies a nondomain design consideration Let's dive into these bullets to better understand them. Aspects of Architectural Characteristics Specifies a nondomain design consideration If you're building an ordering system, then order numbers, customers, items and prices ar...
Read more

Laws of Software Architecture

-
Image
As the discipline of software engineering matures, few things remain constant.  A few years ago, a large portion of the community thought that TDD was always the best methodology to use - after all, you move faster by being thorough and preventing bugs.  Nowadays it's clear that unit tests do not ensure a successful project - when you're A/B testing new features or products and you're not even sure if your code will be around 3 months from now, it's better to churn out production code and test it in prod.  You can clean up the mess later, if the business value makes it worth it. But despite the pace of change in software engineering , in software architecture  there exist a set of 'laws' which don't change over time and which apply to all architectural endeavors.  I'd like to talk about 2 of them in this post as they are important ideas to keep in the back of your mind as your skills grow past coding, past design and into architecture. Laws of Software A...
Read more