Using search engine data to shape and create online content

Keyword research (analysing what people search for on the internet) has  been used by savvy marketers and webmasters for some time now in order to assess the phrases people use to find products or services and then to match their descriptors, content and links accordingly in order to achieve more relevance in search engine results.

Historically it’s been little different in theory (although more sophisticated in practice) to the act of naming your company in an alphabetically astute way in order to appear high up in the phone book listings under your profession.

Perhaps because of what some perceive as the unsavoury, commercially driven motivation of early keyword research, the use of search data for more serious research has been overlooked with a few notable exceptions (such as Google who have been using search data for a number of years now) and applying it to real world problems such as recognising flu outbreaks weeks before the department of health are able to for example.

Recently however, in what can be seen as a greater acceptance of search data,  the bank of England announced that it was using search data in its economic indicators, for example, measuring  demand for desirable goods through monitoring searches such as ‘flatscreen televisions’.

The Bank of England’s report points to the advantage of search data over other, more traditional methods of gathering population sentiment.  In particular, it cites the timeliness, vast sample of respondents (60% of the UK population) and avoidance of complications associated with traditional surveys.  The same report does mention potential weaknesses with the data too, the duplicity of intent behind certain searches (increases in searches for ‘flatscreen television’ could be an indicator that more people feel they have cash to burn or perhaps people are becoming more savvy by shopping online?).  It also recognises the importance of filtering out ‘noise’ from the search data  and the limits of the data provided by the search engines themselves (they use the Google insights for search tool which allows for far less malleability of the data than Hitwise).

Applying search demand to a content production model.

So search demand is becoming an important tool in assessing real world sentiment.  How is it then applied to a content production process?  What are the obstacles involved? And the potential benefits?

Applying search demand to a content production process.

The first question is ‘why should search demand be applied to a content production process?’

Online content has traditionally been driven by a range of factors.  What content there has been commissioned outside of traditional drivers has often been, to put it bluntly, made mostly on a whim.  This has had mixed results and, unlike TV commissioning, any bad decisions made around online content stay online for a long time.

So let’s look at what search demand offers us as content producers.  If we could magically see the complete online demand around a subject, what would that give us?

  • An understanding of a subjects demand in proportion to other topics
  • A compass to help people new to the topic navigate the important areas of demand
  • A reminder to those familiar to a topic of the vernacular, thus engaging a wider audience
  • The ability to ‘drill down’ into topics to fully explore their content potential
  • The ability to connect with audiences desires
  • ‘evergreen’ audiences that will come to the site month after month
  • A wealth of inspiration

All compelling reasons to use search demand as a powerful tool in our every day commissioning, micro-commissioning processes.

Search data is not, in my opinion a panacea.  It shouldn’t be used in isolation to inform us what content to produce but should be used as a useful tool alongside our editorial skill and other traditional drivers .

What are the obstacles?

Search demand is, at its core a record of the terms that people use to search and how often those terms were used.  This results in literally thousands of rows of data, often in spreadsheet format which can seem overwhelming and indecipherable to a lay person who’ll often take the most basic information within the data (the most popular search term) and either reject or accept it based on pre existing views of content commissioning.

Getting the most out of data is an ART that requires SKIll and PRACTICE.  And therefore the obstacles encountered are often due to lack of concrete direction within the data itself (the art is to assimilate all strands of evidence and make decisions based on this which takes skill, practice and a strong will).

Overcoming the obstacles

Overcoming the obstacles therefore requires the data to be made valuable, accessible and understandable.  It can’t be ‘dumbed down’ to such an extent as to offer only the most basic insights however we do need an entry point where lay people can easily see the value of the data without wading through reams of it.  The visualisation below are an attempt to create such an entry point, using bright colours, interesting categories and simplified language in order to hook people into the idea of using search data as an aid.

How does it work in practice?

Let’s say we were in the realm of science.  The first question we should ask ourselves is ‘what are people interested in?’

It seems reasonable to look at the search engine traffic going to science related websites, what are people searching for and how often?

Using Google insights, Experian Hitwise (if you can afford it) or other keyword research tools you then get the data and turn it into insights by grouping similar search terms, looking for trends and patterns and measuring volumes.

Once you’ve done that you should have a pretty good idea of audience demand around a topic which you can then turn into a pretty picture like below!


This shows us that ‘space’ is the most popular reason people look at science related content online and that they’re mainly interested in things within our solar system such as the moon, planets, sun etc.

We then dig a little deeper.  Let’s say the moon landings have an anniversary coming up and so we decide that actually search demand and events are in alignment.

We can use Google insights for search to explore topics around the moon that people are most interested in.

Here, we’ve done a little bit of filtering to weed out any duplicity of intent behind the searches i.e. Were people interested in ‘moon the movie’?  I did this by restricting our view to only those searches that went on to visit a science related website.

we can see that actually the majority of people who search for ‘the moon’ are interested in the phases of the moon / moon calendar.  See why it’s important to dig a little deeper?  If we’d taken the initial insight at face value, we may have gone off and created a whole load of content around the moon thinking that’s where the biggest demand lay (of course that’s not to say we can’t still create content around the moon landings, it’s just that we now know where interest in the landings stands in relation to the moons phases).

It’s important to keep digging and digging at the data in this way until you’ve exhausted it to get a full picture, rather like pass the parcel, taking away and examining layers until you find the prize of a great content idea inside.  You might find that most people who’re interested in the moon landings are interested in the astronauts, or one of them or even if the landings were hoaxed.  We just don’t know until we’ve explored the data fully.

Once you understand the search data, you then understand your web visitor and can communicate with them confidently.  For example, if you know that most people are interested in the moon phases then make sure that it’s mentioned in your meta description so it appears in search engines results pages.  You could make the video / image / calendar of the phases of the moon the first thing you link to on your moon page and describe / link out to other content accordingly.

The future?

Search data and content creation have not had a happy marriage so far.  Search data is the brash, swaggering trickster trying to pull down the high minded values of editorial insights.  Yet they’re starting to speak each others language, albeit slowly.  Search data has refined its language and content creators are starting to be less snobbish and more accepting of the potential benefits.  It seems that large organisations such as BT, Channel 4 and AOL to name but a few have put search data at the heart (or very close to the heart) of their content strategies and smaller, more nimble operators have been doing so for a while not to mention the journalists who’re quickly becoming adept at spotting zeitgeist in reams of social and search data.

The jury is out but my money’s on a surge in demand for skills and training in analysis, visualisation and interpretation of this data.  Watch this space.

Related articles

6 thoughts on “Using search engine data to shape and create online content

  1. This is an excellent article, many thanks. The kind of customer insight that search offers is invaluable, and your “pass the parcel” analogy is bang on. Great stuff.

  2. It’s seems to me that you are very adept to understanding this art form. Are you a Professor or highly educated IT person. What would you suggest as to how I could learn how to dig deeper into search criteria in hope of creating high quality relevant content for building web based businesses? I’m very grateful for having found this site.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s