by Susan Walker

---

Susan Walker’s background is in the field of internal communication and staff research. Her experience included working for Thomson Holidays as its internal communication manager. She is an Accredited Business Communicator (IABC) and a Fellow of Communicators in Business and the RSA.  Susan headed MORI’s HR and communication research practice before launching her own business specializing in communication and engagement research. The main focus for her work is to develop action orientated management information from internal research data to contribute towards business success.

 

---

 

 

 

Measurement is not just an optional extra for communicators, but an essential part of their professional tool kit. It has been seen sometimes as a threat (will they cut my budget? will they cut me?) It can be, however, an exciting opportunity to evaluate, guide and direct communication initiatives and investment.

The advent of easy to use, do-it-yourself on line surveys puts the facility to research and measure employee experiences, opinions and attitudes in the hands of functions and individuals in the organisation.

On one hand, this has the advantage of being able to keep in touch quickly and simply. On the other, it can lead to a multitude of unregulated surveys with decisions made on unreliable results.

This chapter outlines what you need to know to advise colleagues, carry out your own surveys or commission research from external professionals.

Effective measurement has two essential steps: getting the basics right and developing a strategic approach.

But before we move onto these, the fundamental question is to ask: “why are we embarking on this measurement?” If you do not get this stage right, then the rest of the process will not succeed. This may seem too fundamental to mention, but it is surprising how many research projects start without clear, stated, achievable objectives.

The impetus behind the initial purpose for measurement may include:

• The strategy/values of the organisation: knowledge/commitment · Understanding of and involvement with a new initiative
• The communication role of the line manager/supervisor
• Visibility and image of senior management
• Channel access, usage and effectiveness
• Accessibility, tone and honesty/openness of organisational information
• Upward communication – whether a listening process is in place and how well it works
• Lateral communication – sharing best practice and exchange of knowledge across the organisation
• A good way to define objectives is to envisage the end of the process; what information would you want to see in the results? How will the information be used? In fact will it be used at all?

So do not jump into measurement – take a long, hard look at the water before leaping in. Consult others – function heads, line managers and senior management. What feedback would they find useful to help improve communication?

And do not make the mistake of just considering the information channels in isolation. The only way you will be able to assess communication effectiveness is to look at the media (channels such as the intranet, briefing meetings, publications), the messages (organisational strategy, objectives, values, employee feedback) and meaning (understanding, commitment, involvement and how this might influence behaviours.)

 

The basics may sometimes seem tedious and even petty and unnecessary. But these are the essential firm building foundations to produce reliable, constructive information. Aspects like methodology, sampling, statistical reliability, objective questionnaire design, and coverage all need to be considered. These will not be separate issues but all are interrelated so each will need to be taken into consideration together .before decisions are taken.

Methodology

The first step will be to consider the outcomes needed. Will a qualitative approach of focus groups/interviews or quantitative, questionnaire based research best meet your needs?

Both have their advantages and disadvantages.

Qualitative research is likely to be a much smaller scale and therefore less intrusive for the organisation It will:

• Give faster feedback: Provide understanding and insight
• Be able to probe, question and challenge
• Adapt for different functions/levels of employees
• Be well suited to more sensitive topics

But it will not:

• Give hard numbers
• Provide data for future tracking of trends
• Provide information by employee roles/functions
• Be able to be analysed to reveal key drivers of good communication

Quantitative research on the other hand will provide:

• Firm figures and reliable data
• A base for tracking any changes in the future
• Comparisons with normative data
• Detailed results by demographics, functions, locations etc

But it may not answer all the questions in depth, or explain the reasons behind employee views and opinions.

The two approaches are not necessarily mutually exclusive – both are needed for the full picture. Focus groups sometimes precede the questionnaire design stage to help develop the questionnaire. The approach can also be used effectively after the quantitative stage to explore the issues raised and gain fuller understanding of the research outcomes.

Qualitative research also has a place as a stand alone project, especially if the topic is sensitive, new initiatives tested or input sought to develop solutions Quantitative options

If the questionnaire based research is selected as the most effective approach for your measurement objectives then there will be more options to be considered and decisions made.

Telephone: a short questionnaire via a telephone interview is a good choice to follow up an event or a new initiative quickly. However, the time/cost of such an approach means it is not suitable for large numbers of respondents.

Face to face: in a similar way to telephone interviews, this method can be useful for a short, sharp survey with respondents interviewed personally

Postal: questionnaires can be circulated with reply paid envelopes so respondents can post these back when completed

Group self completion: here employees are gathered together in groups and given the questionnaire to fill in which is collected when completed. This methodology is most appropriate in a manufacturing or similar work environment where the response rate for a postal survey may be low

On-line: this is becoming the preferred approach as it gives people the chance to complete a questionnaire on-line. Typically a link is sent out to employees which leads to an electronic questionnaire to fill in on-line.

Sample or census

A further decision may be whether to sample a proportion of employees or invite all to participate. With any organisation employing under about 5,000, the most sensible method will probably be a census. Larger organisations may wish to consider a sampling approach.

There will be a number of factors to take into account. For some methodologies (postal, for example) a sample will be less costly (saving print and postage.) However, there is much less investment difference between a sample and census for an electronic survey.

A sample tends to get a lower response rate as communication campaigns/follow up reminders (see below) cannot be publicised to all employees.

Another factor will be the purpose of your measurement objectives. If the intention is to provide management information by all units then a census may be your choice so results can go back to line managers. If the feedback is needed only at corporate level, then a sample may be sufficient to provide the top line data required.

The question is sometimes asked: how big should a sample be? Again there is no easy answer as this depends on the measurement objectives. You will need to build up your sample from the required result outputs with statistical reliability in mind.

Statistical reliability

The subject of statistics may sound boring but is an essential part of the measurement process to gain reliable, robust data. When reporting around election time, the media often refers to plus or minus 3% on a sample of 1,000. This means that a survey result of, say, 70% could be either 3% higher or lower than the survey figure, If only 100 people were interviewed, the difference could be plus or minus 10%.

So you will need to take this into account when deciding the size of your sample. The employee groups that you want to look at will need to be sufficiently large to give reliable data.

Questionnaire design

Another essential to gain reliable measurement is a relevant, objective questionnaire. When developing the content, it is valuable to consider the information which will be needed for assessing and prioritising communication. It is said there is little point in asking a question which cannot bring action. So, if there is no chance that the quarterly publication will be published more frequently, a question about frequency is useless.

However, if a more general question about timeliness of information shows this to be an issue, there may be a number of ways this can be addressed. There may be some “standard” questions about the level and credibility of information, information sources, opportunities for upward and lateral communication. There will also be specific questions relating to individual organisations such as new initiatives, values and mission statements.

To make the questionnaire more interesting to complete, a variety of question types can be used rather than a long list of agree/disagree statements. It can be tempting to ask for “yes” ”no” answers for simplicity. But a balanced scale is essential.

There is also a school of thought that dismisses the midpoint as the “easy option” which should not be included. However, this is an important group. If they have no strong views either way then – as “fence sitters” – there is more potential to move their views into the positive group.

In addition to the pre-coded questions, open questions can be included which invite people to respond spontaneously in their own words. Beware of including too many; these questions will have to be coded into response categories.

How long should be questionnaire be?

It is not so much the length as the time taken to complete the questionnaire – between eight to 20 minutes depending on the audience. Managers and staff in professional roles, for example, may take the 20 minutes needed – others may get bored after 10 minutes and fail to complete the questionnaire.

Before going live, the next step is to “test” the questionnaire by asking several employee groups to complete the draft. This will reveal the length of time taken to fill in, relevance and comprehension. Some of the questions may be couched in a way open to misinterpretation so this is an opportunity to get the questionnaire right before circulating to all.

Demographics

One section of the questionnaire will ask about the profile of respondents. Too much detail may produce piles of computer tables but not add useful information to the final findings. This is especially true of communication surveys where it may not be possible to respond to individual groups of employees. Location, employee level and service length are usually included. People can be concerned about the privacy of their individual responses and concerned about cross analysis that could identify them so only essential personal details should be included.

The Market Research Society best practice guide recommends that no group smaller than 10 should be analysed separately as reassurance that the views of individuals/small groups will be protected from identification.

Stimulating response rates

When a survey is conducted postally or online, the challenge is to gain the highest possible response rate. Typically a communication survey gains a lower level than a full employee survey. Again the type of audience is an influence: for part-timers in the retail sector the response rate might be around 30% while it might treble for a manager survey. The main stimulus in maximising the response rate is belief that action will follow based on the results. So a communication from senior management giving a commitment to take the findings seriously will assist attracting responses. The most effective time period for questionnaire completion is just over two weeks which gives those who are on holiday the opportunity to respond. A reminder usually helps raise the response rate.

Stage 2: Strategy

At the same time as putting the basics are in place to create a firm foundation, the strategy for the measurement project should be considered equally.

Communication - At an early stage, a communication programme needs to be developed to give information about the purpose of the research, its timetable, proposed feedback process and commitment to action. This campaign can help raise the response rate and make the measurement programme a part of the planning process.

Linking with the business - This is an opportunity to develop a measurement programme which links communication with the business. Review organisational aspects such as:

• Vision and values of the organisation
• Strategy and future plans
• Image among external audiences such as the customers
• Dashboard or similar organisational targets
• Other organisational research which may already exist
• Relevant business measures in place.

In this way the measurement programme can form a real part of the business instead of being just an “add on” .This means that the feedback of the results will be able to be fashioned into action plans rather than just “nice to know” information.

Feedback at all levels

It is never too early to consider the feedback process. When the research is complete, the first level output will be the overall results with the total score for the whole organisation. Next will be the results by the relevant group such as location, job level etc. How the results are fed back will depend on the original objectives of the measurement. The overall results will be of interest at corporate level for action planning while other findings may be useful to other functions such as IT.

If areas such as team meetings are included, then line managers will need their individual results. Insights There are further insights that can be gained from the basic data. Cross analysis can provide greater depth and understanding. For example, the value of team meetings can be gauged by looking at those people who attend and those who do not attend against understanding of organisational strategy. This case study example shows that the 50% understanding organisational strategy also attend team meetings. However, there are only 10% who both attend team meetings and understand organisational strategy. This suggests a correlation between attending team meetings and understanding strategy – a powerful tool to persuade line managers of their value.

Another insightful cross analysis is to look at those people who trust their organisation/believe information against the various information sources. This will reveal the most trusted sources and therefore those most useful to make use of in times of change. This type of cross analysis can be done by location, job type etc to identify where the “hot spots” are for action planning.

Communication return on investment

How can the value of internal communication be proved to the organisation? A number of published studies already reveal a link with business success and communication. However, this is much more difficult to achieve for individual organisations. A host of factors can influence commercial success/profitability so it is hard to isolate the impact of good communication. This can only be done by taking an aspect that can be tracked over time – a customer service project, economy drive or other specific initiative where a communication campaign has played a significant role.

By identifying any improvements/savings achieved as a result of that campaign (minus campaign communication costs), a figure for return on investment can be estimated. Perspective Comparisons against normative data - the average scores from other organisations – can give additional perspective to the results. However, some of these average scores are not best practice in themselves so may provide comfort but should not be regarded as aspirational targets. Acton Measurement without making full use of the results is pointless.

When considering action points stemming from the survey findings there are several stages. The actions need to be practical and attainable with responsibilities assigned. Many of these may belong to the internal communication team while others may belong to functions such as customer service, IT or training. Other plans can only be achieved with the active input of senior management and line managers.

How successful has your measurement programme been?

Changes in behaviours may be apparent. But to measure any alteration in views and attitudes a follow up survey will be needed. Frequency is yet another issue that has no definitive answer. When change is fast, every few months might be the most effective frequency but with a large, static organisation every two years may be sufficient. Whatever action plans are put in place, it will be vital to inform your audience that the measurement programme has listened to their views and – where possible – taken them into account when developing new programmes and initiatives.

 

 

 Create your own business wiki!Support CommunityPublic, not yet premium Privacy Policy RSS