Habitual Analysis: How We Study the Media Industry

Key Takeaways:

Analyzing the media industry – or any industry – well depends on efforts to understand where and how relevant information is produced, testing ideas, and building models (conceptual ones, if not spreadsheet-based ones) in order to focus on the data and the associated insights that matter. We review some of our best practices in this week’s note.

January is always full of announcements, whether at CES, other trade shows, or with public companies revealing key initiatives ahead of or during earnings. This means it’s also a particularly important time to analyze the news well rather than take every piece of information in at face value. I was recently asked how we go about doing this work, and in response, I identified several habitual actions that help with our analyses. Some of these approaches may be useful to others who also need to form views on a given topic. We can loosely organize them into three groups:

1. Understand where and how relevant information is produced.
2. Test ideas and observations and refine analyses with other people.
3. Build models and gather requisite data to firm up your ideas and concentrate on the data and the associated insights that matter.

Understand how information is produced and how it makes its way into news or content that may inform analysis. Much of what we think we know about our own industry comes from reporting in the trade press and through general business news, while other information comes from academics or from thinktanks. Often, reporting may have started with a press release, through public / government records, or through work originally produced by other news organizations. In some instances, news may have originated because a company has provided a publication with information either uniquely or broadly, and in other instances, reporters may have performed their own research, outreach and interviews. Different participants in the news-generating process may have different motives for participating (or not participating) with varying degrees of forthrightness, and different publications have different thresholds for assessing and including or omitting pieces of information, which can get repeated in all subsequent reports. As an example, in 2011 a prominent consulting firm prophesized that by 2017 CMOs (Chief Marketing Officers) would spend more on information technology than CIOs (Chief Information Officer or Chief Information-Technology Officer). While the direction of the report wasn’t wrong, few bothered to note that the underlying survey was focused on high-tech organizations, not companies across the different sectors, and the interpretation of the report was applied more broadly than was likely intended.

• Be conscious of basic statistical measures and data gathering processes. When we hear of surveys that indicate a preference one way or another for a given population, what we often need to know is what the distribution curve looks like. And even when the questions asked have binary yes/no answers, we need to know how, if at all, the sample might be skewed or non-representative, and what the sampling error is. More fundamentally, be mindful of biases in different processes for gathering information and then try to understand how information was gathered. For example, surveys performed using mobile phones will have a different skew than surveys performed using land-lines; passive media measurement panels will always be more accurate than self-reported media measurement. Further, different countries will have different cultural biases in different kinds of responses that may make cross-country analyses difficult.
• Question what you know (or think you know). Far too often, conventional wisdoms persist and sometimes dominate. This can be because those wisdoms may have reflected the best ideas of the past, given limited availability of better data, or because ideas which sounded plausible were never tested. Or, perhaps they were truths under some circumstances but not under all. Of course, sometimes those conventional wisdoms are actually true even if they were never formally tested. Great examples often live on in the form of clichés.

Test ideas and observations and refine analyses with other people. Ask “dumb” questions; try to prove your ideas wrong. Sometimes the questions that an individual has when studying a topic for the first time are the questions that no-one else has dared to ask, perhaps because of a perception that everyone knows the answer. Those questions are often the best ones to ask because they may relate to topics which have gone unchallenged. Relatedly, if you think you have an interpretation of a fact that is not widely held, ask practitioners with opposing views to poke holes in that interpretation. The relative success or failure of that effort will help to reinforce or counter your view. And, even if the conventional views turn out to be well-founded at the present time, they won’t necessarily hold for all times.

• Interact with practitioners at all levels within the industry. When trying to prove ideas wrong or more generally looking for broader points of view on a given topic, consider that sometimes the most knowledgeable people will work in parts of the industry which are less credentialed (because of formal education levels, corporate affiliation, titles, etc.). Such individuals can serve as go-to experts, in part because they are less exposed to the group-think that can dominate other parts of the industry or because they are on the front-lines.
• Explore similar ideas from other industries. To the extent possible, it can be helpful to talk to experts in other industries about similar concepts that might exist in unrelated industries, as lessons learned in those other industries might help to better understand our own. This will be especially true when trying to study issues which are opaque and understood in-depth by relatively few people.

Build models or think in terms of models you would use to describe what you are observing or expecting. While spreadsheet models aren’t necessarily the goal of every given analytical exercise, it is very useful to think in terms of models because of the numerical discipline imposed by them. Toward those ends, it is worth remembering that a model is meant to be an abstraction of reality, and all analytical work can help capture inputs which help to build that abstraction. Often, numbers around a given topic are available, but if they are not the true drivers of the model, they should ignored. Research should focus on finding ways to estimate the numbers that actually drive the behaviors we are trying to model. As a general rule, when an actual model is needed for making decisions, it is beneficial to build one’s own models, but if it is not possible and someone else is responsible for model building, it’s still important to understand the model in depth — and spot-check the math (perfect models are an ideal, but rare in reality).

• Relative-size things as you go, when you can. Whenever a claim is made around a dollar figure, for example, compare it to a broader industry or the overall economy. For example, billions of minutes of time with a given service among a group of people over a given number of days or months could be compared to all activities or all consumption of a medium such as television. All spending on a given product could be compared to the size of the broader industry that product is part of, or possibly the overall economy in the countries that product is available in. As well, be conscious of inappropriate comparisons: for example, comparing the value of a company or asset to the GDP of a nation is essentially never analogous, as one is a figure determined at a fixed period in time based upon an accumulation over all periods of time, while the other is a metric based upon a flow of money over a limited period of time.

New information is often provided by companies looking to put their best foot forward when they communicate information; journalists almost uniformly try to produce what reporter Carl Bernstein described as “the best obtainable version of truth.” Such truths commonly inform perceptions of reality, although there can still be gaps between some press reports, what the industry believes and reality. While the actual truth may be elusive to everyone in some instances, efforts to find it are helpful for participants within an industry. Doing so helps everyone better understand the environment, make better decisions as competitors, customers and suppliers — and ultimately help make the industry both more efficient and more resilient.