Joe Edelman: "Is anything worth maximizing?", a talk about how tech platforms optimize for metrics
Video: https://www.youtube.com/watch?v=GyVHrGLiTcc (46m20s)
Transcript: https://medium.com/what-to-build/is-anything-worth-maximizing-d11e648eb56f (10,314 words with footnotes and references)
Excerpt:
...for simple maximizers, its choices are just about numbers. That means its choices are in the numbers. Here, the choice between two desserts is just a choice between numbers. We could say its choice is already made. And that it has no responsibility, since it’s just following what the numbers say.
Reason-based maximizers don’t just see numbers, though, they also see values. Here, there’s a choice between two desserts — but it isn’t a choice between two numbers. See, it’s also a choice between two values. One option means being a seize-the-day, intensity kind of person. The other means being a foody, aristocratic, elegance kind of person.
My personal thoughts about this talk: it's a kind of strange, kind of dubious philosophical and multi-disciplinary reflection on metrics for organizations, especially metrics for tech companies, and on the pitfalls of optimizing for metrics in what the speaker argues is too "simple" a way.
I don't entirely trust the speaker or the argument, but there was enough in the talk to stimulate curiosity and reflection that I thought it was worth watching.
It seems like they are arguing for more intrusive data collection because the data about user behavior that companies have is superficial. It doesn’t tell them what users really want.
There are other approaches. How about making it easier to for users to explicitly say what they’re looking for? That is, better search and better customization. Or how about giving users better information about the choices available?
I’m thinking in particular about how useless app stores are at recommending new apps. I always have to do an external search to find good reviews and then search for an app by name. But app stores have lots of information about apps that they don’t allow you to search on.
If a company cannot really understand its users, it can’t be in charge of deciding what to do next. Users have to be in charge.
I personally would opt-in to more intrusive data collection if it were for the purpose Joe Edelman describes in the video.
For example, YouTube currently tunes its video recommendations for me based on simple signals like what I click on, video watch time, clicking thumbs up or thumbs down, and selecting "Not interested" for bad suggestions. If I could give YouTube more detailed information about what I want, and if YouTube could and would actually use that information to give me suggestions for videos that are closer to what I actually want, then I would do it and I think it would be great.
My biggest doubt about this is whether it would even be possible from a technical perspective. Can software engineers write code or train neural networks that achieve what Joe Edelman wants YouTube to achieve? So that I can tell YouTube that I want it to recommend videos that instill hope and optimism while acknowledging the tragic aspects of life and have YouTube actually deliver on that?
Maybe this would be possible with LLMs analyzing the transcripts of videos. I'm not sure how else YouTube would even know what it's recommending on that granular a level.
Personally, I'm extremely skeptical that more data would lead to better recommendations. Netflix used to have a quality recommendation algorithm. Near when Tiktok got started, I remember jokes about it's algorithm knowing you were queer before you did. Both of these were before the current LLM craze, and I'd wager both were done with less data than Google had access too.
Recommendations aren't bad because of a technical limitation or lack of data; they're bad because Google is optimizing them for it's actual customers, stockholders and advertisers, rather than the product, i.e. the users.
To understand the context of what we're talking about, watch the video.
Also, please never reply to anything I say and start it with "Eh,". I'm not assuming your intention is malicious, but that really bothers me. It reads to me as dismissive.
My apologies on both counts; that's what I get for insomnia posting. I'd assumed I'd picked up enough context from the comments, and the 'eh' was meant to be casual rather than dismissive, but it didn't read that way in retrospect. Sorry again. Today was busy, but tomorrow I'll watch the video and edit this post if I have any more relevant thoughts.
Well, I’m going to have to listen to the whole thing, but first I’m going to acknowledge and voice my frustration about the general over-reliance on metrics and the all too often lazy abuse of them. I’m sure this guy is going to get into much of the same stuff- but IMO there’s (to abuse a terrible saying) there’s no baby in that old “metrics” bathwater anymore.
You can’t just go spelunking on numbers and magically come up with new markets or new understanding of your market. IMO other orgs are trying to replace actual research with some new quant formula. It’s lazy.
Want to fix an App Store? Do some qualitative research (language analysis) on reviews. Even rudimentary analysis will shine a massive light on the crappy apps and the crappy devs and the out-of-date code bases. Hell, do a search for curse words in reviews and you’ll have a stack of things to fix.
But internally that is NOT what’s ever going to be mentioned in UX meetings. Because that might reduce download numbers. Crappy apps are great for increasing those- and that’s a metric used to measure App Store usage- which impacts the budget for the App Store division.
Metrics are frequently more perpetuated internal turf wars than actually useful for improving UX.
Real research into users and product usage is where the traction and insights are.
So, my assumption is this guy is trying to suggest metrics can be improved with some kind of innovation, and while he’s not wrong, what he is trying to avoid saying is the word “research” because it implies a cost that nobody wants to talk about.
Sorry for posting about this before watching the whole thing. I’ll put it on later when I’m in the car.
Edit: a typo, and turning down a little hyperbole. Thank you.
I feel like you and Joe Edelman (the guy who made the video) share a sense of anger toward tech companies optimizing for simple metrics like downloads in a way that has negative effects.
He has this interesting idea that me, the user, and YouTube or Apple's App Store can have a relationship where YouTube or the App Store is actually responsive to my reasons for using the platform, respects my values and identity, and tries to meet my needs.
The analogy he gives is his hairstylist. He and his hairstylist have a real human relationship. It's friendly and warm. His hairstylist cares about giving him the kind of hair cut he wants. It's simply human but also it's good business — because by showing consideration and establishing trust, the hairstylist gets a repeat customer (and maybe good word of mouth).
Can tech platforms be like that? Can any large business or any large organization, such a government department, achieve such attunement to the people who go to it for their needs? Incentives aside, I am trying to think about how a tech company that was 100% committed to this idea could actually implement it.
I think doing qualitative research (as you suggested) like customer interviews would be one way of trying to aggregate the preferences of many users — by taking a sample that you hope ends up being representative.
Joe has a more high-tech idea in mind that would allow for much greater personalization. That's appealing to me because my needs and preferences may be quite different from the average or the aggregate.
But the only technology I can think of that is capable of this level of personalization is LLMs. Not on the level of personalizing the model, but personalizing the query, e.g., "Based on this list of movies I like, recommend me a movie that shows people struggling with emotions they don't understand, and then finally achieving some clarity. Anything with LGBT characters is a bonus, but that's not a requirement."
I don't know that a technical solution to this problem actually exists yet.
This is a really refreshing take on things. It's not completely novel but I agree this was stimulating to watch.
To me, the video comes of as quite naive. That might be because it was made in 2016. Anyway, the distinction between reason based maximizers and simple maximizers essentially boils down to what heuristic is picked to maximize. Is it simply the average watch time for the recommended content? Is it a weighted score on different metrics where the weights are determined by a guess on my presences? Expecting companies to maximize for something other than profit is doomed to disappoint. Now, that doesn't necessarily mean that my interest and a company's are at odds. For instance, me finding an interesting book will make me more likely to purchase it. So, e.g. Amazon is incentivized to recommend good books... In order for companies to change, their incentives need to change either by people migrating away from their products/services or via government mandates.
Additionally ,the bit about trustseem wrong. I am fairly sure that YouTube for instance categorizes me and tailors it's recommendation to what "I want". It's just that YT optimizes for views. I can trust that. To me trust doesn't necessitate alignment of interests (which is what the video seem to propose). Trust for me is the expectation that I can rely on consistency in behaviour. If you always betray me I can "trust" you to behave that way in the future. Sure there's a deeper sense of trust I can have with people I interact with frequently, but that might just be that we tend to avoid others that harm us (e.g. through betrayal)...
What the video is promoting also seem to have been largely realized. How often do you get a "How likely are you to recommend X?" type of pop/question. This is a proxy for how satisfied you are with a product/service. A lot of companies try to collect this data, so much so I am a bit fatigued by it. They also often seem quite bad at actually acting on that information.