Land of confusion and confabulation

So, let me get this straight.  All market research questions that ask you to recall when you last did an activity are flawed.  Memories get distorted over time.  Not only could the event you are recalling happen earlier or later than in reality (telescoping bias), it might not have happened at all (false memory bias, also known rather splendidly as confabulation).  If you want true, census-type behavioural information then you must use passive tools such as mobile apps or online cookies to capture real-time data analytics without having to remember when asked by an interviewer at some point in the future.

If true then the market research industry has a problem, as do some high profile media surveys like NRS and TGI.

In actual fact, the industry has long recognised that there exists something called memory bias.  For audience measurement surveys, memory bias as a statistical error is taken to be constant.  The real value of these trading currencies therefore is not necessarily with the absolute estimates of readers, listeners, viewers, etc, they produce (and always important to conduct significance tests by the way) but with the period-on-period changes or audience trends.

Memories are very precious things.  They are moments in time that are imprinted on the brain.  As a consequence, it has been argued that they are especially valuable when recalling a media or advertising consumption event.  Conversely, passive behavioural metrics have to manage issues such as whether we are measuring machines or people; and consumption or exposure.  All good fun!

Interestingly, these two methods are fused together with NRS PADD (Print & Digital Data), which takes the print readership audiences, based on recall questions (NRS), with Nielsen’s publisher website audience panel data, based on cookies (UKOM).  The two worlds collide and live together harmoniously so it seems.  Publishers now know how many read print newspapers and magazines, the corresponding websites, one or the other or both.  We know for example in the latest set of data out today (31 October) that adds 35% daily reach to the Daily Telegraph; adds 46% monthly reach to Empire magazine, and so on.

In fact, market research is already heading in this direction thanks to Smartphones and tablets.  We now have the ability to capture a wealth of behavioural data passively (social media, geo-location, media consumption, purchasing behaviour) as well as ask survey questions eg on attitudes all on a single source one-stop-shop device.

The market research industry does not have a problem at all.  In fact its challenge is to contain its excitement at the massive opportunity…if it can remember what excitement is.

John Carroll is Senior Director in Ipsos MediaCT and Chairman of the Media Research Group (follow John on Twitter @MediaCarroll)

  • David Brennan

    I think the Market research industry has an excellent opportunity to minimise it’s reliance on false, misleading or partial memory…but it’s taken a long time. Too many clients still rely on awareness and recall measures and research agencies still encourage it, even though they know the limitations. Technology and psychology are leading us towards more relevant and accurate measures but they are still yet to be seen as mainstream, unfortunately.

    • John Carroll

      I agree in the main. I think the future is a hybrid one as technology doesn’t provide all the answers and I sight. It provides data. Research would be required for context and colour. But it is an excellent opportunity and one that the Media Research Group and the Market Research Society are starting to recognise at last!


    Despite the flaws in some of these approaches the real problem is when people try and use very differently structured surveys conducted for various media to compare – and even evaluate – one medium vs another.  Even though NRS is based on recall it’s fine if you are comparing the Mirror to the Sun or yoy performance of an individual title.  And it’s fine as a basis for a currency.  But you can’t use NRS and compare it to BARB – or RAJAR or UKOM – to help you decide on your media investment.  You need econometrics for that surely.