Planning and Evaluation Wizard (PEW)

Developing a project plan

What difference is your project wanting to make, and what is it going to do to achieve this change? These are the central questions of concern in the planning zone. In evaluation language, this is known as “defining your goals, objectives and strategies”. This section contains some questions to guide you through developing a project goal, objectives and strategies. You can enter your responses in the field provided and the wizard will help you to construct your project plan.

Topics covered in this section include:

• What are goals and objectives?
• What is a strategy?
• Examples
• Writing your goals and objectives
• Begin developing your project and evaluation plan

What are goals and objectives?

Goals and objectives are statements about change.

A goal is a statement about the broad, long-term change your project is working toward. It refers to what you ultimately want to achieve, or your destination e.g., to reduce social isolation of elderly carers in the region.

Objectives are statements about more specific and immediate changes you want in order to progress towards your goal. As a statement about change, it often includes words such as: to increase, to improve, to reduce). The changes might be in skill levels, attitudes, knowledge, processes, awareness or behaviour e.g., to increase the networks of elderly carers, to improve access to culturally appropriate respite services.

Features of ‘good’ goals and objectives

• Clearly defined
• Focus on one thing at a time
• Are able to be measured in some way
• Focus on the change you are wanting rather than the doing of activities
• Are realistic and achievable

A closer look at goals and objectives

Your goal can contain information about:

• WHAT you are trying to change
• WHO will be affected by your project/program (identify community groups)
• HOW they will be affected
• WHERE the changes will take place.

Writing up your goal and objectives

Before developing the goal for your project, think about what your project is trying to achieve.

You may know this from your original project proposal. If so, review this goal to make sure it still applies.

If you are starting from scratch, ask yourself:

At the end of the project, what will be different/what will have changed?

• Who will have been affected?
• How will they have been affected?

Developing your plan

Using the printed blank project and evaluation plan as a guide (download above), you can begin to write in your own details and develop your plan. Once you have decided on your project name you will need to come up with a goal and some objectives to help you to progress towards that goal. You may need to try a few times before finalising on the goals and objectives.

Project objectives are a statement about the specific preferred outcome or change you are hoping to make as a result of your project.

What are the more immediate changes you are hoping to make in order to make progress towards the goal. You can have more than one goal.

The next step is for you to come up with some strategies:

• Process and impact indicators and suitable data collection methods

Goals and objectives checklist

Goals

Is your goal written in a way that identifies the broad, long-term change you want to achieve?

• One way of checking whether your goal refers to change is to see if it contains words such as increase, reduce, improve etc.

Does your goal include what, who, how and where?

• It is not absolutely necessary for your goal to have all of these components, as some of this information may be obvious from the title of the project

Is it written as clearly and concisely as possible and can be clearly understood by someone unfamiliar with the project?

• Sometimes what can seem clear, obvious and appropriate to us, may not be so for others. A simple way of checking this is to ask others involved in your project and outsiders.

Objectives

Do your objectives focus on one thing at a time?

• It helps to have objectives on one thing at a time for a number of reasons. It makes it clearer for others and also helps further down the track when you come to your evaluation.

Do your objectives refer to change?

• As with your goal, a good way of checking whether your objectives refer to change is to see if they contain words such as increase, decrease, improve, widen, expand etc

Do your objectives relate to your goal?

• It’s important that your objectives are clearly related to your goals. Look at your goals and objectives and check there is a logical relationship between them.

Strategies

What is a strategy?

A strategy describes how you are going to get the changes you want. Strategies are activities, they are statements about what you are going to do to achieve your objectives and contribute to the broader goal.

Examples of strategies include: run a training seminar, consult with the community of interest about needs, strategies, produce a pamphlet, develop organisational policies, establish decision-making forms, develop and distribute information, write policies, develop and trial models, run training workshops, undertake a literature review, support the formation of action groups etc.

It is important to be clear about how these objectives fit with the change you are hoping for. In others words how your strategies fit with your objectives.

For example if a project had objectives such as to “improve the knowledge and skills of self management activities for people with chronic back pain” and, the strategies could include:

Objective:

To improve the knowledge and skills of self management activities for people with chronic back pain.

Strategies:

• Plan and run a 6 week back care course
• Produce a simple booklet in English, Vietnamese, Cambodian
• Distribute booklet through GPs, pharmacists, community centres and local industries

Objective:

To reduce the incidence of back injury in local factories.

Strategies:

• Establish a regional forum of hospital, community, union and industry representatives to inform progress of the project
• Review incident reports, and identify problematic areas
• Review worksite policies
• Provide training for supervisors
• Develop recommendations of action for all stakeholders

Features of ‘good’ strategies

Good strategies are ones that:

• Are appropriate for the communities with whom you are working
• Are directly relevant to the change you are seeking ie., your goals and objectives
• Are realistic in terms of number of strategies undertaken, time, resources and skills available
• Have support from relevant players
• Have either proven to be successful in similar circumstances or are innovative

Developing my plan

Strategies

You will need to come up with strategies for each of your objectives. Using your Blank Project and Evaluation Plan form (DOCX) (which you will have already printed off), list what activities your project will undertake to meet your objectives. You may have just one strategy, or you may have more. And some strategies may be relevant to more than one objective.

A useful starting point might be to ask yourself :

• how will the project get the change mentioned in the objective? or
• what will the project do to get this change?

If you are happy with your strategies, you now need to come up with some process indicators.

Strategies checklist

Are your strategies related to your objectives?

• Breakdown the various elements of each objective
• Relate a strategy for each element of the objectives
• Ensure all elements are covered
  

Do they focus on the activities of your project?

• Some strategies may have unintended outcomes and you can address these and their effect on the objectives in your methods.

Are they realistic, eg. number of strategies, time resources & skills?

• Can you do everything you want?
• If you spread yourself too thinly you may not achieve the indicators of success for your strategies

Are they considered appropriate by the workers and community members involved?

• This should be discussed with stakeholders and partners during the planning stage. The feasibility and cultural suitability of the strategies are particularly important to consider here.

Now that you have defined what difference your project is intending to make (goal and objectives), and what activities will be undertaken to get this change (strategies), you now have a good foundation for your evaluation plan.

This section takes you through some of the basics of evaluation and how to develop an evaluation plan. It briefly explores what evaluation is, why we might do it, whose interests the evaluation can serve and what resources may be needed to conduct your evaluation. These are important things to consider before developing an evaluation plan.

The second part of this section explores what evaluation information (indicators) you need to collect at different stages. PEW will assist you to develop relevant and useful indicators of both process and impact.

The final part of this section guides you through assessing and selecting appropriate ways to gather the information you need for your evaluation.

What is evaluation?

Evaluation is the process by which you make a judgement about the worth of something.

Some definitions of evaluation…

“The process by which we decided the value of worth of something. For health promotion, this process involves measurement and observation and comparison with some criterion or standard” (Hawe, Degeling and Hall, 1990)

“Evaluation is the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy” (Weiss, 1998)

Evaluation involves observing, documenting and measuring. It compares what happened with what you expected to happen.

It involves looking at the project and judging whether you are doing what you said you’d do, whether it’s going well, how you could improve it, and whether it resulted in any unexpected developments.

Evaluating, or assessing the value or worth of something, it is an activity that involves making judgements. “Value” is not absolute – people have different views about what is of value. This will influence what information about the project is important to collect. This is explored later in “who could have an interest in your evaluation”.

Why evaluate?

Evaluation can be a tool for:

• Checking out how your project is going
• Seeing whether the project is making a difference
• Identifying its strengths and weaknesses
• Getting ideas on how to improve
• Supporting advocacy for the continuation or extension of the project
• Being accountable to funders and community members involved
  

A good project evaluation process will tell you and others

• What your project has done
• How well it has:
  • Contributed to the goal
  • Met the objectives
  • Undertaken the strategies
• What worked well and what didn’t, and why
• Whether there were any unintended outcomes, and
• What can be learnt from this project to improve practice and inform other projects

Who could be interested in your evaluation?

Before embarking on your evaluation plan, it may be useful to consider who could be interested in it, what they want to know, and why. There are a number of people who may be interested, some of the reasons will be similar across all groups, and others may not be. Some examples could be…

You

• To make changes to improve the project as you go along
• To find out if the project has been successful, how & why
• To be able to argue the importance of the project to others (eg. funders and potential funders)

Participants

• To find out if the project has been effective
• To help ensure continuation of project, or aspects of the project that were successful
• To know that their views have been incorporated into the evaluation
• What difference the evaluation will make

Funding bodies

• To know how the funds were spent, if they were spent on what was intended
• To know if the project has done what it said it would do
• To know what difference the project has made
• To know what the main outcomes are
• To know what the lasting or ongoing benefits will be
• To find out if your project could be replicated elsewhere
• To know why the project should be kept going, or changed, or whether something else should be done with the money instead.

Future projects

• To know what helped and hindered your project
• To know what your project was able to achieve
• To gain insight from your hindsight

Whose interests are being served by the evaluation?

It is important that you are clear about whose interests are being served by the evaluation. It could be useful to approach each of the people with a direct interest in the evaluation to find out if there are particular evaluation questions they have about the project. Identifying who will be interested in your evaluation, what they want to know and why, will assist you in the evaluation design. It will help ensure your evaluation answers the questions it needs to.

Types of evaluation

When you come to evaluate your project, you will need to focus on two aspects of your project. You will need to look at firstly the activities, and secondly the effect your project has had. In evaluation language this is known as process and impact/outcome evaluation.

Apart from process and impact evaluation, it is also useful to consider summative evaluation.

Process evaluation

This involves judging the activities (or strategies) of your project. This often involves looking at what has been done, who has been reached, and the quality of the activities. It involves seeking answers to questions such as:

• Has the project reached the appropriate people?
• Are all the projects activities going to plan? If not, why not?
• Where any changes made to the intended activities? If so, why?
• Are materials, information, presentations of good quality?
• Are the participants and other key people satisfied?
  

Impact/outcome evaluation

This involves judging the extent to which your project has had an effect on the changes you were seeking. In other words, the extent to which your project has met its goal and objectives. Impact evaluation judges how well the objectives were achieved and outcome evaluation involves judging how well the goal has been achieved. It involves seeking answers to questions such as:

• What progress has been made toward achieving the goal?
• To what extent has the project met its objectives?
• How effective has the project been at producing changes?
• Are there any factors outside of the project that have contributed to (or prevented) the desired change?
• Has the project resulted in any unintended change?
  

Summative evaluation

This is done at the end of the project and involves considering the project as a whole, from beginning to ‘end’. It is meant to summarise and inform decisions about whether to continue the project (or parts of it), whether it is valuable to expand into other settings. It involves seeking answers to questions such as:

Overall…

• What were the main benefits and disappointments?
• What things helped and hindered the project?
• In retrospect, what could have strengthened it?
• What would you advise others embarking on something similar?
• What aspects will be sustained and how?
• Is it worth continuing in its current form? Why/why not?
• What recommendations have emerged about where to from here?

What are ethics?

The term ‘ethics’ refers to a set of standards or principles by which a group or community regulates its behaviour, allowing distinctions to be made about what procedures are ‘acceptable’ or not, (see: Flew 1984). This usually involves consideration of a range of interests in the specific context of the study, some of which compete with each other, such as:

• The interests of the Research project in furthering knowledge;
• The interests of wider society;
• The interests of the Research Community;
• The interests of the ‘field-workers’ (particularly if any dangers are involved);
• The interests of any ‘gatekeepers’ or ‘sponsors’;
• The interests of the research subjects

It is the last set of interests, which tends to get the greatest consideration, particularly in evaluation. The interests of the research subjects centre on the following issues:

1. Anonymity (subjects remain nameless throughout the evaluation – this is usually very difficult to maintain as subjects generally become known to the researcher) and so…
2. Confidentiality is usually assured – a purposive attempt to remove from the research documentation any elements that might reveal the subjects’ identities. This may involve the use of pseudonyms, but even here, it is still important to ensure that the information gathered cannot be traced back to individuals (Gibbons, DC., 1975). Additionally the interests of researched subjects may be infringed by...
3. Deceiving subjects – through deliberately misleading or omitting information,
4. Coercing subjects to participate – this may be subtle in that subjects may feel obligated to those who help them. In this case, the request to participate should come from someone not known to the subjects. It is also important to stress that their agreement or refusal will in no way effect the services they receive, and that they can pull out of the study at any time,
5. Leading subjects to answer in a particular way,
6. Generally harming the subjects in some way through inducing stress, pain, anxiety, diminishing self-esteem or invading privacy.

Informed consent

One way in which researchers attempt to address some of these issues is through obtaining ‘Informed Consent’ from the subjects. This is when you inform subjects prior to the study of the nature/purpose of the research/evaluation and who is financing it, what you will require from the subjects and what may subsequently happen in terms of the findings. It is important therefore that the information gathered is used only for the purposes stated, and not made available to other people or organisations. Having informed the subjects (and ensured they understand), you then seek their consent to participate, sometimes by getting them to sign a short form to this effect, a copy of which they retain.

The extent to which you can ever ‘fully inform’ subjects of the outcomes of the study is a mute point in that we cannot predict all future ramifications. We are not soothsayers! Additionally, Informed Consent is problematic for certain types of subjects eg. the mentally ill or handicapped. For younger subjects informed consent should be sought from a guardian.

However, it is also often the case in evaluations that subjects are already fully aware of the study and eager to participate which might raise questions over the suitability of using consent forms.

The main point is to consider the context of the study and its subjects in the light of the above considerations to make ‘ethically informed decisions’ prior to conducting the evaluation, and where ambiguities arise seek further advice. Researchers need to ensure that all aspects of the study are conducted in a way which “is respectful of the human rights and needs of the participants” (Batchelor and Briggs 1994, p.949)

Access to records

If client information is to be included in an evaluation, they must agree to this in writing. Individuals must not be identifiable and only aggregate data should be presented in the report.

If you are using client records, permission must also be obtained from the Privacy Committee of South Australia. This committee was set up to ensure there are not intrusions of privacy where government records are concerned. In South Australia, this committee comes under the Attorney General’s Department, but community health workers outside of SA should check whether there is similar legislation in their own state.

Codes and committees

The National Health and Medical Research Council regularly updates a Statement on Ethical Conduct in Research Involving Humans. Guidelines regarding ethical procedures in social research generally are provided by bodies such as: The Australian Sociological Society (TASA), the Market Research Association, and the Social Research Association. Evaluations of projects conducted in a hospital setting will invariably have to be passed by a specific ‘Ethics Committee’ or ‘Review Board’, who will assess the proposed evaluation in the light of the issues raised above. Advice concerning procedures should be sought from the appropriate Ethics Committee prior to commencing the study.

The National Health and Medical Research Council’s “National Statement”

• Attempt to promote unified set of ethical standards for research with humans
• Requires thorough assessment of proposed research by Human Research Ethics Committees established in all institutions/organisations that receive NHMRC funds for research
• Constitution and procedures of Committee defined by National Statement
• Allows cooperation between Committees
• Endorsed by: Ministers of: Health and Aged Care, Education Training and Youth Affairs, Industry Science and Resources; Australian Vice Chancellors’ Committee; Australian Research Council; Australian Academy of the Humanities; Australian Academy of Science; Academy of the Social Sciences in Australia
• Supported by Academy of Technological Sciences and Engineering
• Reference Point for Research Ethics
  

Broad sweep of ethical concerns addressed

• Variety of data types: observations, interviews, records
• Emphasis on: Research for person or “collectives”; Beneficence; Justice; Integrity; Consent; Privacy; Intellectual Property
• Variety of Subject Scenarios: Young people; Intellectual and mental impairment; People dependent on Medical Care; Collectives; People in unequal relationships; Aboriginal and Torres Strait Islanders
• Epidemiological Research
• Range of “medical” research approaches
  

If the evaluation involves Aboriginal people, your proposal should be approved by The Aboriginal Health Research Ethics Committee of SA (AHREC) even if it has already been approved by another ethics committee.

Aboriginal health research

If your project includes undertaking health-related research with Aboriginal people or communities in South Australia, you are required to submit your proposal to the Aboriginal Health Research Ethics Committee of South Australia (AHREC). This is necessary even if approval has been obtained from other institutions such as hospitals or universities. AHREC is a sub committee of the Aboriginal Health Council Inc, which is the peak community-based body for Aboriginal health in South Australia. AHREC is recognised by the National Health & Medical Research Council to be an Institutional Ethics Committee. The committee meets monthly and requests for ethics approval need to be made in writing.

What is an evaluation plan?

An evaluation plan is a short summary of what needs to be evaluated, what information needs to be collected, and how you are intending to collect this information. An evaluation plan is much easier to develop if you have clearly defined goals, objectives and strategies, as they act like the foundation for the evaluation.

Why have one?

An evaluation plan can help you to:

• Make clear what you want to evaluate, when & how
• Guide the information you need to collect along the way
• Identify and approach people in advance who may be vital for the collection of this information
• Work out if ethics approval needs to be sought
• Consult with people about the best ways to collect the information
• Determine what time and resources are needed for the evaluation
• Help ensure the evaluation scale is appropriate to the scale of the project
  

They can also be an essential requirement from funders.

Preparing for an evaluation

In preparing for an evaluation, it is helpful to have:

1. Clearly defined program goals, objectives and strategies
2. Agreement on:
   • How the evaluation will be used
   • Who it is for or who is/are the audience(s)
   • What evaluation questions should be addressed
   • What resources and time do you have for the evaluation

Internally or externally?

Who should do the evaluation?

You can conduct your own evaluation or have an external evaluator do some, or all of it, for you. This decision will depend upon whether the funder of the project requires external evaluation, whether you have evaluation skills within your staff, and whether you have funds to employ an external evaluator.

The following table offers some of the pros and cons of internal and external evaluation (adapted from Feuerstein M. Partners in Evaluation, and Yoland Wadsworth Everyday Evaluation on the Run).

Evaluation indicators overview

Now it’s time to decide what evaluation information you need to collect at different stages.

This section is designed to assist you to identify what could be signs of progress and success (indicators) for your project. It will also explore what evaluation information would be useful to collect.

After a brief introduction PEW will ask you questions, firstly about your strategies and then your goals and objectives. This will help you to come up with ideas about what evaluation information to collect about your activities (indicators of process) and about the changes you are hoping to achieve (indicators of impact/outcome).

The next section will focus on how to collect this information.

What's an indicator?

Indicators are signs of progress and change that result from your project. They provide some guidance about what would be useful evaluation information to collect. Some of this will involve collecting information along the way, which help you to gauge how well things are going, and possibly enable you to make improvements throughout the project. Others will involve collecting information at the end of the project.

 “An indicator is a marker. It can be compared to a road sign which shows whether you are on the right road, how far you have travelled and how far you have to travel to reach your destination. Indicators show progress and help measure change.” (Fuerstein)

Usually indicators are expressed in numbers, eg:

• Number of participants in a program, the proportion of which are male or female, or from different age groups or ethnic groups, or geographical areas
• % of clients satisfied with the information provided
• % of people who have stopped smoking
  

It can also be useful to have indicators that are not numbers, eg:

• Training seminar outline produced by your project
•  Summary of findings from a needs assessment
• Documentation of policies, or networks developed

There are two main types of indicators – ‘process’ and ‘impact/outcome’.

Indicators of process

Indicators of Process provide a sign about how well your activities (strategies) are going. Process indicators often fall into 3 main groups.

They can be indicators of:

1.    Implementation (what has been done), e.g.

• Workshop outlines
• Procedures developed
• Copies of media coverage
  

2.    Reach and scope (who and how many have been involved) e.g.

• Number of participants
• Proportion of ethnic groups, age groups etc.
• Workers and organisations involved
  

3.    Quality (how well things have been done) e.g.

• Proportion of participants who report they are satisfied with materials or information produced, or the service provided
• Certain standards of quality have been met

Developing my plan - process indicators

In this section you will develop signs of success (or indicators) for your project. Using your Blank Project and Evaluation Plan Form (DOCX) which you will have already printed off, the first step involves developing indicators for strategies and then for your goal and objectives.

Developing Process Indicators for your project

At this stage it may be useful to consider the possibilities before prioritising which indicators would be most useful and practical to collect information about.

A good starting point for developing process indicators is to ask yourself

“Looking at the project’s activities (strategies)…what could be signs of success?”

What would be signs (or indicators) that the activities:

• That were planned have occurred?
• Have reached the appropriate people?
• Have been of good quality?

It is a good idea to jot these down and then refine them into specific process indicators for each of your strategies.

If you are happy with your process indicators you now need to come up with some impact/outcome indicators.

Indicators of impact/outcome

Indicators of Impact/Outcome provide a sign of how well you have achieved the changes you were hoping for as a result of your project. They are about measuring change. In other words they are a measure of the extent to which you have achieved your objectives and your longer term goal. Indicators of impact relate to your objectives, and indicators of outcome relate to your goal.

Usually indicators of impact are much easier to come up with because they are about more immediate changes you are seeking. Whereas measuring outcomes refer to large and long term changes. Goals are often written as broad statements of desired change, such as to decrease social isolation in a region, or reduce the incidence of domestic violence. They can also relate to a change in health status, such as reduced mortality, morbidity, or improved wellbeing.

Health income evaluation rarely occurs in small primary health care projects, due to the short term funding, complexity and difficulties of attributing project activities with long term outcomes. See below for more on Impact/Outcome evaluation.

Instead most projects focus on evaluating the impact of their project according to their objectives.

Examples of Impact/Outcome indicators:

• Changes in awareness, knowledge, skills
• Increases in the number of people reached
• Policy changes
• Changes in behaviour
• Changes in community capacity
• Changes in organisational capacity (skills, structures, resources)
• Increases in service usuage
• Improved continuity of care

To view the impact/outcome indicators on our example of a completed Program Evaluation Plan.

Developing my plan - impact and outcomes indicators

In this section you will develop impact/outcome indicators for your project using your Blank Project and Evaluation Plan form (DOCX) which you will have already printed off, this section focuses on your objectives, or the changes you are hoping for as a result of your project.

You will need to consider what could be signs of success in relation to each objective.

What could be signs of success?

Impact indicators need to be able to show some sign of change. They are likely to involve signs of increase/decrease or improvement, or capacity. As signs of change, they require some sort of comparison of what was before, and what was after the project.

Depending on your objectives, impact indicators can include

• Comparing statistics before and after
• Comparing structures or systems before & after
• Participants reporting that change intended by project has occurred (eg. increased knowledge, skills etc. or greater capacity)
  

A good starting point for developing impact indicators is to ask yourself

 “Looking at the project’s objectives… what would be signs of success?”

It is a good idea to jot your ideas down and refine them into specific impact indicators.

View our example of a completed program evaluation plan.

Looking at the options

There are many different methods that can be used to collect information for your evaluation. The strengths and weaknesses of some of those methods are described here:

Deciding on data collection methods

In choosing what method(s) are most appropriate, it may be useful to consider the following:

Will it give you the information you need?

Who will the information be sought from?

• Age, gender, cultural considerations
• Language, literacy and other communication considerations
• Logistics – eg. if people spread over a large geographical area?
• Availability for people who are VERY busy
• Confidentiality considerations
  

Who will gather the information?

• Skills, time available

What are the costs involved?

Developing my plan

Process and impact data collection

Now it’s time to choose what data collection methods you wish to use

Using your Blank Project and Evaluation Plan Form (DOCX) (which you will have already printed off), you need to consider the process indicators and the appropriate methods to gather data for these and then consider the impact indicators and the appropriate methods to gather these.

You may wish to revisit your process and impact indicators in your Project & Evaluation Plan before deciding on the appropriate data collection methods.

Whatever methods you choose, it is important to include some overall or ‘summative’ questions such as:

• What were the greatest achievements?
• What worked best, worst?
• In hindsight what could have been done differently to improve outcomes?

View our example of a completed Program Evaluation Plan.

Feel free to edit your data collection methods at any time if you wish.

Now you can develop your budget.

Evaluation checklist

Some issues to consider about your data:

The suitability of your data collection method

• Is the method acceptable to your subjects?
• Are there any cultural issues to consider?
• Is the method going to tell you what you want to know?
• Do you need to make generalisations about a population? (surveys needed)
• Do you need to explore ‘new’ ideas or areas? (qualitative methods)
• Do you want to do both of these things? (triangulation of methods)
  

Are your findings valid?

• Have you piloted your questionnaire?
• Are you measuring what you are supposed to be measuring?
• Does your questionnaire/topic guide look suitable to your subjects (‘face validity’?)
• Does your questionnaire/topic guide cover all the issues (‘content validity’?)
• Do you need to think about other types of validity? (criterion validity’ and ‘construct validity’)
  

Are your findings ‘reliable’?

• Are you measuring accurately?
• For questionnaires, are they internally consistent?
• If different people are ‘rating’ answers, are they doing this consistently?
  

Can you summarise your findings clearly?

• What were the greatest achievements?
• What worked best/worst?
• In hindsight what could have been done differently to improve outcomes?

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Acquiescence response set (agreeing to everything)
Some respondents have a tendency to frequently endorse the statements or questions in a questionnaire. This is difficult to overcome but where Likert Scales are used, by balancing statements which are favourable toward a topic with statements which are disfavourable, the influence of acquiescence will be suitably reduced.

Action Research
Applied research that join up practitioners with researchers in a research partnership. Emphasis here is on ongoing improvement of practice by the practitioners themselves.

Actuarial Records
Public records about the demographic characteristics of a population.

Applied Research
Research done with the intent of applying results to a specific problem. Evaluation is a form of applied research. This can be conducted as part of an action research approach.

Case
A single unit in a study (e.g. a person, event or setting, such as a day centre).

Case Study
A research method which focuses on the characteristics, circumstances, and complexity of a single case, or a small number of cases, often using multiple methods. The case is viewed as being valued in its own right and whilst findings can raise awareness of general issues, the aim is not to generalise the findings to other cases.

Causal Relationships

A relationship between variables whereby changes in values of one variable (called the 'independent variable') cause changes in the value of another (called the 'dependent' variable).

Census
The gathering of data from all the individuals in a population.

Clinical Trial

An experiment where the participants are patients, usually involving a comparison of a 'project' group (who receive a treatment or intervention) and a 'control' group who do not.

Closed Question
The question is followed by predetermined response choices e.g. multiple choice, Likert Scales and yes/no questions. Many closed questions have 'other' as the last alternative with an 'open' space for respondents to specify their answer in words.

Collectivities
Distinct human groups with their own enduring social structures that link members with a common identity, with common customs and with designated leaders or other persons who represent collective interests in dealing with researchers. Collectivities may include cultural or ethnic groups, and indigenous communities. (see the National Statement on Ethical Conduct in Research Involving Humans, NHMRC, 2001).

Concept
An abstraction representing an object or phenomenon.

Confidentiality
The obligation not to disclose the identity of respondents. Confidentiality can also refer to the obligation of persons to whom private information has been given, not to use the information for any purpose other than that for which it was given.

Consent
The voluntary agreement of a person or group to participate in research. This should be obtained in conjunction with the person or group being given adequate information which has to be fully understood by the subjects; hence 'informed consent'.

Content Analysis
The systematic analysis of observations obtained from records, documents and fieldnotes.

Covert Methods
Where the researcher hides their identity or purpose from the subjects or groups being researched. This can be contrasted with the scenario of 'overt' research where full information is given.

Deception
This occurs when research participants have essential information withheld and/or are intentionally misled about procedures and purposes, including studies where participants are observed without their knowledge or given misleading information about the purposes or outcomes of the research study.

Deduction
Where pre-defined hypotheses are tested against the data gathered to assess the likelihood of them being correct. Contrast with induction.

De-identified (anonymous data)
Data are referred to as 'de-identified', where the identifiers have been removed permanently or the data have never been identified. The term "de-identified" is used frequently, to refer to sets of data from which only names have been removed. However, such data may remain "potentially identifiable" by referring to code numbers. It is also noteworthy that simply removing names does not guarantee confidentiality or anonymity, particularly if the research is with a small group where people may be identified by the opinion that has been recorded.

Empirical
Based on observation.

Ethnography
The study of people in their natural settings; a descriptive account of social life and culture in a defined social system, based on qualitative methods (e.g. detailed observations, unstructured interviews, analysis of documents). This method is used by anthropologists in studies of 'non-western' cultures and ethnographers for studies of sub-cultures within western societies (eg drug cultures, soccer hooligans, sex workers etc) and its institutions (eg the Police). There is much debate concerning the use of covert and overt methods here.

Evaluation
The systematic assessment of the operation and/or outcomes of a program or policy, compared to explicit or implicit standards, in order to contribute to the improvement of the program or policy.

Field Research
Research which takes place in a natural setting. 'Field Work' is often used to refer to the process of gathering primary data.

Focus Group
A qualitative technique developed by social and market researchers in which 6-12 individuals are brought together and interactively give their views and impressions upon a specified topic. These are used to evaluate attitudes towards the topic. Focus groups are usually homogeneous with members being generally of the same age, gender and status to encourage participation.

Formative Evaluation
A type of evaluation conducted during the course of program implementation to provide information to improve the program under study. In this sense formative evaluation is closely linked to 'action research'.

Grounded Theory
The investigator develops conceptual categories from the data and then makes new observations to develop these categories. Hypotheses are derived directly from the data, and may be tested against it. All conclusions must be 'grounded in' and supported by the data.

Harm
That which adversely affects the interests or welfare of an individual or a group. This covers: Physical harm, discomfort, anxiety, pain, psychological disturbance and includes social disadvantage (e.g. ostracism). This can be very difficult to judge and sometimes harm may be experienced as a result of the research, but some time later (eg bad publicity) which renders informed consent problematical.

Hawthorne Effect
When the methods used to gather data have an effect on the data collected. This concern should always be considered bearing in mind how subjects interpret the research and give meaning to it.

Health Behaviours
Any behaviour that is part of maintaining well-being and avoiding illness and disease.

Health Lifestyle
Voluntary health behaviour based on making choices from the alternatives that are available in individual situations.

Identified samples or data
Data that allow the identification of a specific individual. Examples of identifiers may include the individual's name, date of birth or address. Where a 'case study' is conducted people may be identified by their status, ethnicity, age or expressed opinions.

Impact
The net effects of a program. Impact may also refer to program effects for the larger community. Often used interchangeably with "outcome". Incidence Cases (e.g. of disease) which first occur in a population in a defined period of time.

Independent Variable

The presumed cause of an outcome under study: changes in an independent variable are expected to cause changes in the value of a dependent variable (see dependent variable).

Induction
A process whereby data are gathered and subsequently ideas and theories are derived from the findings. Induction harmonises with 'grounded theory' approaches. Contrast with 'deduction'.

Informed Consent
A written or verbal agreement in which potential participants agree to participate in the study after receiving adequate information about the study to make a reasoned decision. (see National Statement on Ethical Conduct in Research Involving Humans, NHMRC, 2001). The two main difficulties here are: 1. Naturalistic research and notably ethnographic research in criminology may wish to use covert methods as the only viable means of obtaining their data, and 2. The extent to which participants have enough information for reasoned decision-making. Difficulties arise with groups such as prisoners, children, NESB groups, the mentally ill etc, and with longer term unexpected outcomes.

Inputs
The resources used to conduct a program.

Instrument
A tool used to measure or study a person, event, or other object of interest. This would include topic guides for focus groups (qualitative instruments) and questionnaires for surveys (quantitative instruments).

Interpretive Approach
The theoretical perspective that social scientists must address the meaning that subjects' give to events and behaviour in order to obtain a full understanding. In evaluations using qualitative methods, the interpretive approach is adopted. In this case it is often the meanings, attitudes, and beliefs that we are addressing.

Interview
A research method which involves a trained interviewer asking questions and recording respondents' replies. These can be recorded by the researcher ticking boxes in closed questions, writing down answers verbatim or as a summary, or by using a tape recorder (usually for in-depth interviews) which allow data analysis to be conducted later.

Justice
That which concerns fairness or equity, often divided into three parts: procedural justice, concerned with fair methods of making decisions and settling disputes; distributive justice, concerned with the fair distribution of the benefits and burdens of society; and corrective justice, concerned with correcting wrongs and harms through compensation or retribution (see National Statement on Ethical Conduct in Research Involving Humans, NHMRC, 2001).

Likert Scales
Where questions appear as statements with respondents being asked to indicate their level of agreement on a 5 point scale from 'strongly agree' to 'strongly disagree'. Statements which are favourable toward a topic should be balanced by the same number of 'disfavourable' statements to address the issue of acquiescence.

Missing Data
Information that is not available for a particular case (e.g. person) for which other information is available. This can be due to subjects missing out the question (intentionally or by accident) or due to a mistake in recording or entering the data (hopefully always by accident!).

Naturalistic Research
Employed by ethnographic researchers using descriptive research in a natural, unmanipulated, social setting. This emphasizes the need to study people in their natural environment as the best means by which to understand them. This is a qualitative approach, but by highlighting the primacy of having to conduct 'research in the natural environment' could lead to criticism of other qualitative methods, such as focus groups, which have to create 'artificial environment' in which to gather data.

Objectives
The specific desired program outcomes.

Observation
A research method, in which the investigator systematically watches, listens to and records the phenomenon of interest.

Open-ended Interview
A form of interviewing that does not use a structured questionnaire with 'closed' questions but rather allows the respondent to shape the direction of the interview, by being encouraged to express their own story from his/her own perspective. Topic guides are used here.

Open-ended Question
A question in a semi-structured questionnaire or topic guide that allows respondents to respond in their own words. Occasionally open-ended questions may appear in a structured interview using a 'closed question' instrument. This is not that common however, due to the difficulties of analysing these quantitatively.

Operationalise
To turn something into a measurable form eg you can operationalise 'height' by asking about height in a questionnaire; you can operationalise an objective by asking questions which address its indicators.

Outcome
The end results of the program. Outcomes may be intended or unintended and be positive or negative.

Participant Observation
A research method in which the investigator takes part in the social phenomenon of interest by participating with a group and observing the interactions between them and between the researcher and subjects to achieve a greater understanding. Used in ethnographic approaches.

Pilot
A small, preliminary test, dress rehearsal or trial run. This should be a mirror image of the research evaluation to be done only on a much smaller scale. Interviews, questionnaires, sampling and initial analysis should all be considered. More associated with quantitative approaches. The results of the pilot are used to improve the program or evaluation procedure being piloted before it is used on a larger scale.

Personal Information
Information by which individuals or collectivities can be identified. This is defined in the Privacy Act 1988 (Commonwealth) as information or an opinion (including information or an opinion forming part of a database), whether true or not, and whether recorded in a material form or not, about an individual whose identity is apparent, or can reasonably be ascertained, from the information or opinion (see National Statement on Ethical Conduct in Research Involving Humans, NHMRC, 2001).

Population
The whole group about which the evaluator wants to draw conclusions. All the members of a population are potential subjects. Usually we cannot ask everyone in a population (conduct a census), so we need to draw a sample. A sample is a subgroup taken from the population that is often meant to be representative of the population.

Preventive Health Behaviours
Actions taken by a person to reduce their risk of developing or worsening illness.

Primary Data
Data gathered by the researcher in the act of conducting research. This is contrasted to secondary data which entails the use of data gathered by someone other than the researcher.

Process Evaluation
A study of what goes on while a program is in progress.

Program
A structured intervention to improve the well being of people, groups, organisations or communities.

Qualitative Research
The approach advocated by the interpretive school as a means to understanding social phenomena. Generally viewed as any kind of research that produces findings not arrived at by means of statistical procedures or other means of quantification, and includes in-depth interviews, observations and participant observation.

Quantitative Research
The approach advocated by the Positivist School. This approach measures social phenomena and obtains numerical values which can be analyzed statistically. Surveys using structured questionnaires and IQ tests are both examples of quantitative research.

Reliability
The questions 'are we measuring accurately?' and 'how stable is our measure?' reflect concerns with the issue of reliability. It is the extent to which the measure is consistent and accurate. A clock which is ten minutes fast can still be reliable in that it accurately and precisely measures the time plus ten minutes. However it is not valid in that what it measures is not really the time.

Research
This involves systematic investigation to establish facts, principles and knowledge.

Research Methods
These are the methods of data collection: focus groups, interview, telephone interviews, postal surveys, diaries, secondary analysis of documents, observation and participant observation etc.

Research Participant
Individual (or groups of individuals) about whom a researcher conducting research obtains data.

Respect for Persons
This has two fundamental aspects 1) respect for the autonomy of those individuals who are capable of making informed choices and respect for their capacity for self-determination; and 2) protection of persons with impaired or diminished autonomy, that is, those individuals who are incompetent or whose voluntariness is compromised. (see National Statement on Ethical Conduct in Research Involving Humans, NHMRC, 2001).

Retrospective Study
The collection of data concerning the past. Interviews which attempt to gain such information often rely on the memory of respondents and may not be reliable.

Sample
A subset of a population.

Sampling
Techniques used to obtain a subset of a population. This includes 'probability sampling' where each subject has a known statistical chance of selection (often used in quantitative studies), and 'non-probability' sampling such as quota, snowball and purposive sampling, where subjects do not have a known statistical chance of selection (used for qualitative sampling).

Sponsor
There are two meanings here: 1. An individual, company, institution or organisation that takes responsibility for the initiation, management, and/or financing of a clinical trail. (see National Statement on Ethical Conduct in Research Involving Humans, NHMRC, 2001) 2. A 'guide' or 'bridge' to link the researcher to a specific group or culture. The 'sponsor' in this case may introduce the researcher to potential participants.

Summative Evaluation
A study conducted at the end of a program (or of a phase of the program) to determine the extent to which anticipated outcomes were produced.

Survey
A method of collecting information from a usually large sample of the population of interest. This is usually a quantitative method which allows statistical inferences to be drawn from the sample about the population.

Theory
A set of logically interrelated propositions and their implications

Topic Guide
A list of themes and/or open questions presented individually to subjects in a focus group or open-ended interview. The idea is to stimulate discussion around each theme presented.

Triangulation
Using multiple methods and/or data sources to study the same phenomenon. The idea here is for the weaknesses in any one method to be compensated for by the strengths of another. The researcher addresses the issue from different methodological positions, rather like taking photographs of the same subject from different angles to reveal a more valid picture of what the object actually looks like.

Validity
In measurement, validity refers to the extent to which a measure captures the dimension of interest. The question: 'are we measuring what we're supposed to be measuring?' reflects the issue of validity. In a questionnaire we would at least address 'face' and 'content' validity in the pilot study. 'Face validity' involves getting a small group of respondents to read the questionnaire to ensure it looks valid in their view. 'Content validity' is determined by a small group of 'experts' in the areas/topics addressed, who ensure all the aspects are covered in the questionnaire.

Variable
An indicator assumed to represent the underlying construct or concept.

Voluntary
Free of coercion, including any sanctions for not taking part.

The following people and organisations helped redevelop PEW (2008)

Project team
Michael Bentley, formerly SACHRU
Rebecca Ramm, formerly SACHRU
Tuesday Udell, formerly SACHRU
Rama Ramanathan, formerly Discipline of Public Health, Flinders University

Website design
Rebecca Ramm, formerly SACHRU
Tuesday Udell, formerly SACHRU

Advisers and contributors
Professor Fran Baum, SACHRU
Gwyn Jolley, SACHRU
Angela Lawless, SACHRU
Paul Aylward, formerly SACHRU

The following people and organisations developed PEW version one

Project team
Charlie Murray, Adelaide Central Community Health Service – Parks (and formerly SACHRU)
Paul Aylward, formerly SACHRU
Richard Cooke, South Australian Drug & Alcohol Services Council (and formerly SACHRU)
Maryanne Martin, Information Officer, formerly SACHRU

Website design
Maryanne Martin, Information Officer, formerly SACHRU
George Aldridge, Cartoonist, Adelaide, South Australia

The funders
Health Promotion SA, Department of Human Services, South Australia
South Australian Community Health Research Unit (SACHRU), The Advisory Committee