CPattern C

Scenario Modeling in the Time It Takes to Walk Your Dog

Peter Cleary · · 4 min read

#modeling#experiment#scenarios#product-lab#dogs

Scenario modeling used to involve extensive planning, a substantial data science investment and months of work.

Today, you can run effective, good-enough scenarios in the time it takes to walk your dog. (And you can even walk your dog while running the scenario.)

Let’s start with an example. You’re the product manager for Acme Same-Day Plumbing Service, where we guarantee a plumber will be at your house within three hours. You recently introduced an automation to continuously optimize your network of plumbers. The goal was to move from a manual scheduling process to a system similar to what ride-hailing apps use to match a car to a passenger.

Efficiency skyrocketed - the CFO is very happy. But the plumbers are mad. They’re getting bounced around all over the place. In some cases they start on their way to Rehoboth and get rerouted to Revere - all in the name of efficiency. They’re threatening to quit. So while the CFO is happy today, she may be wicked mad in a few weeks when revenue plummets after all the plumbers quit.

You need a solution quick - and turning off the automation is not a viable option.

Enter the product lab.

You quickly pull historical data going back six months. You also check out the code that runs the automation. As you’re doing this, you come up with a few options for how to make the plumbers happier:

  • Once the appointment is less than two hours away, never switch an assignment
  • Give the plumber a 20% bonus on an assignment if it is switched at the last minute
  • Station each plumber in a very small region and give them - and only them - assignments in that region

You think the first option will work best - it keeps an automated process and should also get the company back to having plum plumbers. But your gut has been wrong before.

Using your AI - such as Claude Code - it’s easy to set up some experiments to challenge your assumptions and validate what the best options are. You can even use the experiments to come up with new options. For simplicity here, though, we’ll stick to just testing these three options rather than generating new ones.

Here’s how I would proceed:

  1. Open up your favorite AI. For this discussion, we’ll use Claude Code.
  2. Give the AI read-only access to the folder with the code in it, or - better yet - connect the AI directly to the repository.
  3. Give the AI access to the folder with an export of the data. If the data structure isn’t easy to discern, also be sure to provide a description of what is in each column.
  4. Tell the AI - in clear and explicit terms - what exactly you are trying to do. You want to reduce drive times and fill up plumber schedules, but also avoid frequently switching plumber assignments. Give it all three options you’ve identified up front, but tell it you want to run them one at a time, then compare results.
  5. Before you feed the data in, ask it what parts of the functionality it is going to use. And ask it to be explicit. You want to ensure it’s hitting the right logic.
  6. It’s important that you don’t give the AI model the historical outputs. You want to give it the inputs of the requests for plumbers and make sure the outputs reasonably match what you’ve seen historically. If the model can’t reproduce what already happened, nothing it tells you about the proposed changes means anything.
  7. Ask the AI to walk you through its analysis and results step-by-step. Also let it know that you want to see the justification behind each result and not just its findings. The AI aims to please - you want to confirm it’s being objective. Rather than just trusting the results, spot-check a few values by hand.

Once you review the results and analysis - and do a bit of your own analysis - you may be surprised. In this (hypothetical) example, the lab results indicate that locking plumbers into smaller sub-territories led to the least movement. This is a novel but logical conclusion.

Smaller territories on their own will reduce drive times. And if there’s only one plumber in a territory, there’s never a risk an assignment will switch to a different plumber. The real insight is that you could fully accommodate the demand by shrinking to smaller territories.

The CFO will be happy - this approach shows efficiency rising by, say, another 5% beyond just straight optimization. Even though the model can’t say this directly, you also expect the plumbers will be happy: they’ll be spending more time plumbing and less time driving around.

Building out a model based on your production code and feeding in real data is a great, low-cost way to use AI to test out different theories before investing engineering time and before disrupting real end users. This advances product management in three key ways:

  • Quick testing and validation of theories
  • Good-enough modeling without needing to pull in engineers and data scientists
  • An easy path to data-backed product development

With all this in hand, the model becomes the spec. Ask your AI to write a scope document based on the model, create story tickets and hand it off to engineering.

Comments (0)

No comments yet. Be the first to share a thought.

Leave a comment

Comments are reviewed before they appear. I store your name and comment, not your email.

← All patterns