On Knowing Your Audience’s Behavior: Beware the “Average” with Comic Sans Nightmare Graphics

POST PREVIEW— in this post you’ll get awesome graphics with Comic Sans, like this:

expected

 

(Note regarding graphs: TD Haines thinks the axis should be swapped on each of these, so that duration is on the X axis instead of Y. He’s probably right but these graphs are diamonds, baby! They’re staying this way forever!)

This is a brief warning for anyone getting into analytics—

You developed an iPhone app, a perfect clone of the Facebook app. You have some new features you want to add:

  • a quick post view to quickly scroll through titles of recent posts—a 15-second experience.
  • an article view that provides links to all the articles posted by friends—a 10 minute+ experience.
  • an unnamed feature your manager wants—a 5-minute experience.

Which should you build first? The executive over your group wants to be sure your decision is driven by data, so you look to analytics to prioritize your work*. How much time do users spend using your app?

(*Assume ALL other factors are equal and time spent is the only factor that matters.)

Over the last 8000 sessions, your users averaged 5 minutes per session.

Do you go with the 5-minute feature? It’s the perfect average time for one of our features!

If you say yes, you are assuming that your data looks like this:

expected

This doesn’t reflect reality. You’ve seen reality. You know it’s hot and dusty and dirty and is rarely represented by a pretty curve that makes sense.

So you jump into Google Analytics and look at “Engagement” (these are actual #s from a tool I maintain).

how_to_engagement

Unexpected! The “average” use isn’t in the fat part of the curve. This graph also highlights another important point: This is a measurement of behaviors, not users. There is no “average user” that logs in for five minutes. There is a cluster of sessions that are a short duration, and a cluster of sessions that are a long duration. Those are behaviors, not human beings.  When you design new features, they’ll be designed to take advantage of different behaviors, which might be driven by the same user in different contexts.

Compare those expected and actual curves that represent ‘behaviors’ (not users):

actual

Those are headed in the opposite directions.

Designing for a new “average” feature makes less sense when viewed through this lens.

decision

Based on this, you feel comfortable setting a priority of:

  1. article view
  2. quick view
  3. feature for 5 min avg.

Next up: mmock up a quick and dirty MVp of the article view, share it with some users, and get some initial feedback. Tomorrow, if possible.

Takeaways:

  • Look for groups of behaviors, not ‘average’ behaviors.
  • Separate behavior from users. These graphs depict sessions and durations, NOT users. The same user could have sessions at the bottom or top of the graph.
  • Beware seductive numbers! Always dive deeper. Even in this example, you could dive deeper into the session numbers to understand how the short-duration users are actually using the app, or investigate why some sessions run long. Is it always based on meaningful interactions, or something else? (Maybe the short durations are all launched from push alerts that momentarily ping the app when acknowledged…or maybe some users leave the app running while they watch a how-to video a friend posted. Or maybe the analytics are just broken.)
Advertisements

One thought on “On Knowing Your Audience’s Behavior: Beware the “Average” with Comic Sans Nightmare Graphics

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s