Institute of Internal Communication

Whether you are running a communication evaluation exercise yourself or employing an agency to do it for you, how do you ensure that the results are valid, useful and reliable? Domna Lazidou, an academic researcher and consultant offers her top tips.

Be clear and precise about your purpose
At the outset clearly define what you are trying to measure and why. General statements like ‘we want to measure the effectiveness of our communication activity’ are not enough.

What are your criteria for measuring effectiveness? Disseminating information? Meeting audience needs? Reaching the right people at the right time? Creating clarity about corporate objectives? Helping managers with their communication role? Keeping costs down? All of these? Or more than one?

Plus, what will you do with the results when you have them? Will you focus on developing the most successful of the channels? Will you use them to try and improve the less effective ones? Are you planning to benchmark/compare areas of the business in terms of their communication effectiveness? Do you want to put together guidelines/training materials for how to improve communication effectiveness?

Knowing the answers to such questions will help you ask the right questions of the right people in the right way. As a result you will get much better findings from your measurement process.

Select tools and processes fit for purpose
Although questionnaires and group interviews – focus groups – tend to be the most often used evaluation tools in the communication field, they are not always the most appropriate and they are far from being the most cost-effective.

Knowing exactly which tool is good for what purpose and considering all possibilities before embarking on your project will both save money and significantly improve the quality of your findings.

For example, why run a costly questionnaire-based survey when all you need to know is whether key messages are covered in a consistent way across all your internal communications channels? A media analysis will be by far the best methodology to use.

Similarly, if you want to explore the best practice in cascade briefings, identifying those managers with a good reputation and observing a series of sessions run by them should produce as good, or even better results, than trying to interview lots of people on the subject.

Allow time for planning
You need to allow at least a couple of weeks, preferably a month to plan a measurement exercise properly – possibly more if it is a complex exercise with a broad company-wide scope. Rushing into a survey without thinking it through properly will only cause you problems later in the process.
It is also important that you set up a steering group early in the process to help you during the planning and implementation stages.

Design and follow a rigorous process
First and foremost don’t cut corners! Stick to your plan and document everything with as much precision as you can. This is particularly important when you are collecting qualitative data. It also requires that you develop (or buy in) the right technical expertise for the right part of the evaluation process.

It is a myth that anyone can run a piece of research. The answer to the question ‘how difficult can it be to ask a few questions’ is that there’s more to it than meets the eye. Asking the right questions, in the right way and interpreting the results without bias is very difficult indeed – although seasoned researchers may make it look easy.

Don’t forget to pilot! Pilot early to ensure your process works: have you allowed enough time? Have you considered correctly the perspective of those who will take part? Are the questions understood as intended? Is your process producing the expected results? Are you prepared for analysing the data the process is producing? Are you getting unforeseen data/problems?

Sample carefully
It is a myth that if you sample a large enough percentage of the overall employee population you are always ‘covered’. In many cases it is completely inappropriate to sample from the whole employee body.

For example, if you are only interested in managers, intranet users or new recruits.
In many cases a small sample of the right people is enough. For example, if you are exploring best practice in certain parts of the business or if you are comparing impact of manager communication style on productivity or motivation. In either case, you could identify a small number of case studies and evaluate them in depth, rather than attempt a large-scale survey.

Get behind the headlines – understand what your findings really mean
Carefully planned evaluation studies can be completely ruined by lack of attention when the results come in. When it comes to analysing your findings it’s simply not enough to produce a list of answers in terms of what’s good (high scores), what’s bad (low scores) and what’s neutral.

You need to understand and be able to relate the interesting ‘stories’ your findings tell about your organisation and its communication.

These ‘stories’ can frequently be told in terms of relationships between the things you are investigating. For example, is satisfaction with communication positively correlated with satisfaction with the communication behaviour of line managers? Does the credibility of senior managers seem to influence the level of clarity people feel they have about the company’s vision and strategy? Does best practice correlate with how much departments spend on their communication infrastructure?

Naturally, what relationships you discover in your data will depend on what you set out to evaluate in the first place as well as the quality of your data gathering.

Be sensitive to your context.
You are not measuring in a vacuum and your research has to take into account the realities of your specific business and its people.

In practice this means you need to make sure your research has the backing of key stakeholders – senior managers, HR and in some cases union representatives. It’s also a good idea to build an internal steering committee of people who can advise you on local issues, sensitivities and practicalities.

Finally, you must make sure your timing is right and whatever you do fits with other business activities. There is nothing worse than issuing a questionnaire to staff immediately after a redundancy announcement or an organisational upheaval.

Put in place effective feedback loops
There are two strands to this. The first is to link in to organisational influencers early on, explaining what you are doing, why and helping them to understand the business importance of the exercise.

The second is to constantly test your assumptions and conclusions with the people who provided you with your data in the first place.

This is particularly important in organisations with large-scale diversity – for example, a multinational company with subsidiaries and sites in other countries, where assumptions, norms and communication practices may be very different from those in the home country.

Take action!
It is now well documented that organisations which survey their employees and fail to take action tend to perform less well than those organisations that do not attempt to evaluate at all.

A Watson Wyatt study found that total return to shareholders of organisations which were seen to take action on surveys was 117 per cent. This compares to 97 per cent in organisations which did not survey at all and 77 per cent in organisations which surveyed and failed to act.

The same study identified a significant difference in employee commitment levels between organisations that survey and act (92 per cent average commitment) and those that don’t act on survey results (46 per cent average commitment).

Demonstrating that you and the business are acting on results is the only way to ensure that your employees will continue to respond to future measurement exercises with goodwill and without bias.

Domna Lazidou is director of Warwick-based communication consultancy Omilia. She specialises in communication strategy and measurement, particularly in cross-cultural business contexts. She can be contacted at: dlazidou@omilia.co.uk