Data Exchange 201 – Introduction to program outcomes

Transcript for Data Exchange Training Module 201 – Introduction to program outcomes – December 2018

Welcome to the Introduction to program outcomes webinar.

My name is Rachael, I am with the Data Exchange training team, and it is my pleasure to be taking you through this information session today.

As we begin I would like to acknowledge that “We are meeting today on many traditional lands around the country and I want to acknowledge the traditional owners of those lands and pay our respects to elders past and present.”

This session will provide you with the basic principles of what program outcomes are, including why they are important.

I will demonstrate where to find information on your program description to help develop a program logic and how this links to reporting outcomes in the Data Exchange, using SCORE. SCORE stands for Standard Client Outcomes Reporting.

The first step is to determine what an outcome is.

In thinking about outcomes, there are a lot of different ways to define what they are. Some definitions include that it is a measure, a goal, an objective or the aim of your program.

Outcomes are very important and drive us to improve or remain on a particular course.

When we think about outcomes in regards to the Data Exchange, an outcome is the change in a persons life following their interaction with a funded service.

Outcomes can be positive, neutral or negative, intended or unintended.

Recording the outcome accurately, including negative or neutral outcomes, can demonstrate to government the complexity of service delivery and the difficult situations clients face.

What is the difference between outcomes and outputs?

Outputs are the items produced as a result of a program, such as the number of clients that your organisation has seen, or the number of sessions that a client has received during a reporting period. Outputs are an important part of the story and supports the collection of outcomes data.

Outcomes are about what changed for clients following access to a program.

The shift from outputs to include meaningful information about outcomes, started in 2000 through the Department of Finance’s Outputs and Outcomes guidance, which linked financial management to outcomes. This grew to the establishment of the Data Exchange in 2014, which was designed to reduce multiple reporting methods for families programs in the Department of Social Services. The Data Exchange now provides streamlined and consistent reporting for a number of other government departments through the Community Grants Hub.

The Data Exchange Framework seeks to tell the ‘story’ about grant programme outcomes by breaking them down into a number of linked ‘chapters’—drawing on the principles from Freidman’s Results Based Accountability:

• how much is being done, in terms of the services and help to clients targeted by the program,
• how well it’s being done, from the point of view of clients they are helping, and
• Did we achieve what we expected in terms of resolving the issues clients sought help with, and contributing to positive changes in their circumstances.

It’s important to note that we all operate in a complex service system. Many of these complexities are outside the ability of a program to control.

The Data Exchange outcomes framework is only one piece of the puzzle. Within grants administration, there are other mechanisms to capture contextual information about your services, such as activity work plan reports, or discussions with your Funding Arrangement Manager.

The Data Exchange has two datasets that organisations provide information to.

The first dataset is the priority requirements. These are the mandatory fields that all organisations must complete to add a client. This dataset attempts to answer questions about the number of people are accessing services and where and when the services are being delivered. For example, is the activity being provided to the intended demographic cohort?  This is important information and provides valuable context on the more detailed outcomes information. You can see the fields on screen now.

Name details, date of birth and gender information automatically generates a Statistical Linkage Key (SLK) - 14 characters numbers and letters - which de-identifies client data but enables it to be matched over time and programs. No identifying information is kept. Much of the information you see on screen is captured at the client record level and definitions of these fields can be found in the Protocols document on the Data Exchange website.

Priority requirements collect a lot of output data, how many people, locations etc. This is an important part of the outcomes story, but not the full story.

Along with the priority requirements, we collect partnership approach data within the Data Exchange.

The partnership approach is where the organisation agrees to collect an extended data set and outcome information in return for access to additional reports.

This is an extended dataset comprising of:

• additional client demographic details (Homeless indicator through to NDIS eligibility),
• client needs and referral reasons (reason for seeking assistance and referrals to other services), and most importantly,
• SCORE, which stands for Standard Client/Community Outcomes Reporting

Unless specified in your program guidance information, the extended data set items are optional. This means that organisations can collect any of these additional items if they are relevant to the program that they are delivery.

For example, for an organisation delivering a program to reduce homelessness, the Homeless indicator may be collected and reported on.

Or if you are delivering a program aimed to assist aged care clients to remain independent and living at home you might want to collect Household composition.

If delivering a financial program you might want to collect Main source of income, approximate gross income and income frequency.

Organisations delivering settlement services programs might want to collect Month / Year of first arrival in Australia or visa type and ancestry

Or you might want to collect the reason the client sought assistance or the reason you have referred the client to other services.

You are also able to collect and report the service setting and assessed by fields.

The service setting is the place it took place such as at the organisation’s premise, the clients home, was it over the phone, a mobile service, or at another location. This provides additional context on what happened in the delivery of a service.

As mentioned, these items are optional and can be collected and reported on at any time or for any client. Obviously the more data you collect, the clearer the picture becomes of the outcomes being achieved from your services.

The other partnership approach data items relate to:

• SCORE or outcomes information on the impacts of services that clients are accessing

SCORE, the Data Exchange outcomes framework, supports consistent outcomes measurement through four components that are relevant to the range of programs reporting in the Data Exchange.

SCORE includes a standard set of outcome domains to help quantify the difference programs are making to a client's life.

Organisations are also able to identify how the assessment was made. Was it by the client, practitioner, joint assessment made by the client and practitioner, or a support person who might be accompanying a younger or older client who may not be able to make the assessment themselves.

Organisations can also identify the tool that was used to make the assessment. Was it a validated outcomes tools such as a Kessler K10, or was the assessment made using SCORE directly.

We’ll now go through SCORE in more detail.

SCORE has four main components: Circumstances, Goals, Satisfaction and Community.

The SCORE components are linked to standard domains to make it easy to compare and aggregate client outcomes across the range of programs being reported in the Data Exchange.

Each domain targets a specific area and has a separate set of questions within them, relating to a client outcome. The answers are then rated on a Likert scale of 1 to 5.

Circumstances – is where the funded activity is seeking to measure the change of the impact over a longer term of a clients mental or physical health, material well-being, personal and family safety and situation just to name a few.

Goals – looks at the progress of a client in achieving specific goals, such as changed behaviours, changed confidence to make own decision, changed skills to lessen the impact of a crisis, and is often measured over a shorter term.

Satisfaction – measures the client’s perceptions of the responsiveness and value of the organisation’s ability to meet the client’s needs.

Community – measures changes in group, organisation, and community capacity to address identified needs rather than at an individual level.

The domains are intended to report a specific, stand-alone outcome, that is independent of other outcome domains. This means that each reported SCORE domain needs to be associated with a specific outcome. We suggest you review Appendix B of the Protocols to determine the domains that are most relevant to the programs delivered by your organisation.

Everyone involved in service delivery have the same goal – to improve the wellbeing of individuals and their families. We want to help. This holds true through the range of stakeholders, government and non-government, involved in this work.

We know programs are part of a complex service system, however in the Data Exchange we don’t attempt to measure the complexity in a clients life, only the parts that directly relate to the funded services they are accessing.

Outcomes information helps us all to assess which programs, delivered in which communities, achieve long-term outcomes for clients.

Measuring outcomes gives organisations evidence about what is working and what is not working, to help them adjust service delivery to better meet the needs of clients. 

Measuring outcomes is very beneficial for clients, organisations and government.

The benefits of collecting outcomes data for clients include using SCORE to discuss their goals, finding out about their perception of what has occurred following use of a service, and better targeting service delivery based on their needs.

• The Department has conducted research with clients which showed they value the discussion about what has changed for them as it reassures them that services are interested in their needs and outcomes.
• Clients can also benefit when outcomes measurement is part of the therapeutic process, because it can help them reflect on how far they have come and what areas they still want to work on. 
• Confirm needs are met – provides evidence that the client is achieving their intended outcomes and if not, services can be adjusted to better meet identified client needs.
• The Department’s research also showed many clients are motivated by a sense of altruism; they want to give feedback to help improve service delivery for other people in a similar situation to them.

The benefits or measuring outcomes for organisations are:

• Highlights performance by providing information about the activities and what is and isn’t working including where service gaps may be present or where a program is delivering good results.
• Identifies services that lead to better outcomes – is there a combination of services that work better for clients?
• Monitor impact of initiatives – is your organisation having an impact in your community?
• Provides evidence - Resource for evaluations of what you and your clients are achieving.

Several organisations have provided case studies that indicate outcomes data is useful, in both case management and gaining a deeper understanding of client cohorts accessing their services.

The case studies from Interrelate and Relationship Australia NSW can be found on the Data Exchange website, Policy guidance tab under the Measuring outcomes heading.

The next step in using outcomes to help improve services is to find the program outcomes relevant to the program you are funded to deliver.

A focus on outcomes can bring about a number of benefits, including great results for your clients. Before we get started on a program logic, lets spend a bit of time nutting out where you can find your programs outcomes.

When the program was advertised, you may have seen the Grant Opportunity Guidelines, this contains a range of information relating to the program.

This is on the Community Grant Hub website.  If you didn’t see this, your program aims or outcomes will be in your Grant Agreement or on the funding agencies website. For DSS this is under the program and services tab.

Appendix B of the Data Exchange Protocols Document can be found on the Data Exchange website under the Policy Guidance Tab. Appendix B is a great tool to help work out the outcomes the program is seeking to achieve and also sets out your reporting requirements of your particular program.

If you are still unsure, ask within your organisation or contact your Funding Arrangement Manager for a copy of your funding agreement.

This is an example of a program listed in Appendix B. It is the Family and Relationship Services program.

Its aim as noted under the Description heading is to strengthen family relationships, prevent breakdown and ensure the wellbeing and safety of children through the provision of broad-based counselling and education to families of different forms and sizes.

Once you have identified your program outcomes, you can start to develop what you need to do to achieve those outcomes.

One way to map outcomes is through a program logic. We know that there are other approaches that your organisation may use, however a program logic helps you to tell the story of what you intend to do and how you expect this to impact clients.

This is a really helpful planning tool, bringing out the links that your organisation expects will drive change across the range of client outcomes reported.

Ideally, the program logic is created before a service starts and is something policy areas manage through the funding process. This involves a range of stakeholders contributing to the expected outcomes. Some of you may have already completed one and others may not have needed to.

Let’s look at the elements that make up the program logic starting from left to right:

• Issues: What is the issue or concern of the program?
• Inputs: What resources or ingredients do you need to run the program? How many staff? How much time? Ways of working or new assets?
• Outputs: What will be the tangible products? This includes:
  • Action or Activities: What will you do? What needs to be developed to deliver your program?
  • Participants: How many clients will you see, at what location? Will you produce guidelines?
• Outcomes: What impact will your program have on clients? What’s reasonable to expect to see changes in? In developing the outcomes, be sure that they are directly linked to your program.

The program will have:

• Short-term Outcomes: changes in knowledge, skills, and/or attitudes;
• Intermediate Outcomes: changes in behaviour or action; and
• Long-term Outcomes: changes in condition or circumstances.

Again, it’s reasonable and expected that outcomes may be positive, negative or neutral. You may also see outcomes that you hadn’t thought of in delivering the services. These are the unintended outcomes of the program.

This example is a very high-level example of a program logic that illustrates how a counselling program intends to directly increase family wellbeing and economic engagement for parents.

When working out a program logic, it is best to work back wards and this is called back casting. This will ensure that you are mapping your long term outcomes back to your outputs and not developing any false assumptions. Start by asking what is the program hoping to achieve overall. The question could be “What does success look like?” or ‘what would the community look like if this problem or issue didn’t exist?”

Let’s do that with this example:

• The program aim is to improve child, individual, family and community wellbeing.
• The longer term outcome is families that require less support, are stable and meeting their children’s needs.
• The Intermediate outcome is improved family functioning, regular healthy meals which improves a child’s wellbeing and children attending school and ready to learn which improves their individual and also community functioning.
• The Short term outcome is increased safety and parental capacity with stronger family and community ties.
• The, action is that the organisation sets up an activity, in this case counselling and information is developed. Which is then accessed by participants. The individuals attend counselling sessions and gain referrals to other services. This is the outputs.
• Then we have inputs of funding, staff and ways of working, via partnerships, 

Things may not work in an orderly fashion in reality, and there is likely to be other inputs and actions to get the desired outcome, but this simple diagram allows us to quickly test that there is a logical flow that results in an outcome.

Once you have your program logic, you can articulate the planned outcomes and test that your program is directly affecting the outcomes you expect.

The next step is to test the logic behind your assumptions.

What we do here is connect our outputs through to an outcome and on to a SCORE domain.

You may find that some of your outputs don’t directly link to the outcomes, that would indicate there is a need to consider that outcome and test if that is realistic.

Ask yourself: If I do this, then it will lead to that? So if I develop and deliver financial counselling, it will lead to clients that are able to better budget, or not need to access financial aid.

OK, so once we are sure of the logic behind our program, we can move on to think about how we map this to SCORE.

The easiest way is to draw a line across from the output, to the outcomes, and on to SCORE.

On the screen is the example of linking the short term outcomes of stronger family relationships and increased parental capacity, to the intermediate outcome of Increased family functioning and then to the long term outcome of Increased individual and family wellbeing. This is then linked to the SCORE outcome of Family functioning.

You may find that you are mapping the short term outcomes, others will be more focused on the long term outcomes, as this reflects the type of program you are delivering. We appreciate that you may wish to report on your long-term outcomes, however you can only do this in the Data Exchange where the reporting period is still open and your service is still reporting in the Data Exchange.

So for our example program, we aimed to increase:

• Individual and family wellbeing, which links to the SCORE domain, family functioning;
• Increased economic engagement, which links to the employment domain; and
• More cohesive communities, which links to the community participation and wellbeing domain.

To make it easier to identify which SCORE domains may be relevant to your program, we have worked very closely with the program policy area to develop program-specific guidance in Appendix B of the Protocols.

On the screen is an example of the Family and Relationship Services program from Appendix B.

You will notice that under the Heading ‘What areas of SCORE are most relevant?’ the policy area for this program have identified the SCORE domains that would be most relevant.

All programs in Appendix B have this information listed.

This doesn’t mean that they are the only domains that outcomes can be recorded against. You may have clients that access your services for other issues and you can then record outcomes if you are funded to deliver those other program to reflect the impact the service is having on them as well. This listing in Appendix B is a guide for you as to what areas of SCORE would be most relevant.

As an organisation, it is important that you:

• Understand what outcome your client is trying to achieve by accessing your services
• Assess how the client is at that particular point in time, and
• Record that assessment using an evaluation tool which can be translated to SCORE or capturing the outcome directly in the Data Exchange

The templates to determine your program logic are available on the landing page where this webinar is published on the Data Exchange website. 

Follow the steps that we have discussed and use the handouts to work through this exercise.

If you are unsure of your program’s aim – check your grant agreement, grant opportunity guidelines or Appendix B to obtain this information.

Discuss this process within your organisation to ensure that you are not making any false assumptions and then map the outcomes to the relevant SCORE domains.

We have a range of useful resources on the Data Exchange website to help you in your outcomes journey.

The documents shown on the screen are all found under the Policy guidance tab on the Data Exchange website.

The Guide to measuring outcomes document explains how to do a program logic, so this is a useful resource if you’re going through the program logic exercise. 

You can also contact our helpdesk.

And don’t forget to subscribe for the latest updates on the Data Exchange while you’re there. 

That brings an end to this webinar today.

Thank you for your attention and we hope that you can join us next time.