Data Exchange Training Module 202 – Application of outcomes

Welcome to the Introduction to program outcomes webinar.

My name is Rachael, I am with the Data Exchange training team, and it is my pleasure to be taking you through this information session today.

As we begin I would like to acknowledge that “We are meeting today on many traditional lands around the country and I want to acknowledge the traditional owners of those lands and pay my respects to elders past and present.”

This session will provide you with an insight into some of the factors to consider when measuring and recording outcomes including:

• Choosing an outcomes measurement tool; and
• How SCORE works in the data exchange.

We will review the different reports that are available for organisations that are participating in the partnership approach and how you can use this data to help improve service delivery.

We will now look at the factors to consider when choosing a measurement tool.

When thinking about a useful outcomes measurement tool for your organisation, we recommend that you think about:

• The client cohorts using the tool – will the clients be able to understand and use this tool effectively? For some programs, the use of a culturally appropriate tool may be important. You may also need to consider if it’s suitable for different levels of literacy, age groups, and clients with a disability.
• You should also consider whether an existing tool would meet your organisation’s needs. Are there commonly used tools in your sector that would be suitable?
• Do you need to adapt an existing tool, or make your own?
• If using an existing tool, can it be translated into SCORE? Is it in the Translation matrix on the DEX website, or do you need to translate it yourself? If you’re creating your own tool, think about how it will translate into SCORE, using the guidance in the Data Exchange Translation Matrix document.
• Another important factor to consider is the cost of these tools. Some tools have ongoing licensing fees, or charge for each individual outcomes assessment. There are free tools available, but you should also consider whether there are IT costs in adapting your IT system to implement these tools and translate the results into SCORE.
• If you don’t have a tool, and don’t find something suitable, you can choose to report outcomes through SCORE directly, using it as a proxy measurement tool.

You may find that certain tools are appropriate in some programs and for some clients, but not for others, so consider whether you need multiple tools within your organisation.

The use of tools is an operational decision for organisations, however we recommend that you think about the ability of the tool to measure outcomes. Some are designed for specific purposes, such as risk assessment or needs assessment, that may not appropriately measure outcomes.

You may wish to talk to your peers or network organisations about what tools they are using, or seek advice from peak body organisations. The Child Family Community Australia (CFCA) website, run by the Australian Institute of Family Studies, has some material available on their website about how to choose outcomes measurement tools. We have provided a link to this site: https://aifs.gov.au/cfca/2016/04/14/how-choose-outcomes-measurement-tool

In 2015, the Department of Social Services (DSS) commissioned the Australian Institute of Family Studies (AIFS) to identify the most common instruments that Families and Children activity organisations used to measure outcomes. AIFS reviewed nine instruments for the original Translation Matrix.

DSS committed to review the Translation Matrix to ensure its ongoing relevance for a growing range of program areas that report to the Data Exchange. In 2018, ARTD Consultants undertook a technical review of the instruments presented in the Translation Matrix.

The Department sought feedback and comments from organisations and other stakeholders through two surveys, and face-to-face workshops held in early 2018. In July 2018, organisations were invited to respond to questions raised in a Discussion Paper.

The Department received six submissions from organisations and internal feedback from policy and program areas. The current Translation Matrix encompasses both the review findings and the feedback provided by organisations.

The document has been expanded to provide more information about the background of each tool, and considerations to apply when doing the translation. It also includes templates and guidance for organisations wanting to use their own ‘in-house’ tools and translate them into SCORE. Using these templates can help provide consistency within your own organisation about how results are translated into SCORE.

We’re now going to run through how to translate results from an existing tool into SCORE. In this example, our organisation is using the Kessler Psychological Distress Scale or K10.

The K10 is a basic questionnaire that is often used to figure out whether someone has experienced depression and anxiety.

While we are using K10 as an example – the principles in how we convert the tool into SCORE remain the same if you are using a different evaluation tool.

On the screen is an example of K10 that a practitioner has completed in consultation with the client. The K10 rating was 25.

To convert this rating into SCORE we now consult the Data Exchange translation matrix. The document explains the background to K10, and recommends how to translate the results into SCORE. We can see that:

• this evaluation tool’s outcomes align to the circled SCORE components of Mental health, wellbeing and self-care circumstances domain
• We find where the K10 rating of 25 is – in this case it is in the 22 to 29 range (bottom circle)
• We then go to the SCORE range (the middle circle) and we can see that it translates into a 2 on the SCORE scale.

Let’s look a little deeper on how SCORE works

SCORE is a standardised outcomes framework that aims to capture the changes that have occurred for clients across the broad range of program activities reporting in the Data Exchange.

SCORE stands for Standard Client/Community Outcomes Reporting.

SCORE works by measuring the difference between at least two outcome assessments in the same domain at different points in time.

• The first step is to select the relevant domain(s) in SCORE. For this example we are using the Circumstances component and the Mental health, wellbeing and self-care domain.
• You would make an initial (previously known as a pre) SCORE assessment towards the beginning of a client’s engagement with a service or activity on the provided Likert scale of 1-5.
• You would then make a subsequent (previously known as a post) SCORE assessment at a future point of a client’s engagement with a service or activity.
• Multiple SCOREs should be recorded for long-term clients.

The difference between the initial and subsequent score assessment allows us to look at the amount of change in a client’s outcome after receiving a service or participating in an activity.

Organisations that do not have an existing outcomes measurement tool can choose to use the SCORE Likert scale to measure outcomes for their clients.

To assist these organisations in applying SCORE as part of their everyday business practises, the Department has developed a version of SCORE that explains the domains and Likert scale from a client’s perspective.

This can be used directly with clients, with questions from the clients point of view.

The Additional guidance for using score with clients document which is located on the Data Exchange website under the Policy guidance tab explains this in further detail.

On screen is an example of this for the Mental health, wellbeing and self-care domain.

Again, the client would select the response that best reflects where they are at the time the assessment is being made.

For example, the client feels that they have some difficulty in managing their work pressures and feel this is having a moderate impact on their mental health. This rates as a 2 for SCORE and this number is recorded when a SCORE is added for this client.

Outcomes are the changes in an individual, group or family’s life following interaction with a funded service. These occur over time and are not always immediately visible. This means regular reporting of this data is critical.

We also know it is important to build trust with clients, and for this reason we recommend that the initial SCORE assessment is completed sometime towards the beginning of service delivery, such as during the first month. For some services this can be during intake, others will need more time.

The subsequent assessments can be undertaken at any time, depending on the intensity and length of service delivery.

For some programs and clients it can be tricky to identify the end point but we recommend a final assessment where there is an end point if that is possible. In a number of programs there is a natural end point, others may not know when the client will drop out.

We suggest undertaking an assessment once a change is observed or the client indicates a change has occurred in their goals or circumstances. Undertaking subsequent assessments with the client can be valuable in determining if the client has achieved what they expected, and may be used in their case management.

Due to the varying nature of programs reporting in the Data Exchange, flexibility is built into the SCORE process. This allows you to undertake assessments when it fits into service delivery practices.

For example, if your service is delivering Emergency Relief, you may complete the initial and subsequent assessment in the one session, both before and after you’ve provided a food parcel or financial support.

An organisation delivering a financial literacy program may make an assessment of a person’s financial goals or their savings behaviour at the beginning and the end of the financial literacy program.

Likewise, a program delivering settlement services will likely meet with clients a number of times potentially across years, so they may record several subsequent assessments every few months, until the client no longer requires that service.

Overwhelmingly, outcomes measurement works best when it’s integrated into everyday practices. The key to making it work is planning for it to occur at logical points in your service delivery practices. This is where you see incremental changes over time in clients circumstances or behaviours.

As we have already mentioned, you can collect outcomes using a variety of methods.

An organisation can use a validated evaluation tool, they can adapt a tool to suit their needs, or they can use SCORE directly.

Organisations are able to identify the method in the Data Exchange of how the assessment was made, using the ‘Assessed by’ field. Please note this field is entirely voluntary; however it can provide valuable context about how a SCORE assessment was made.

As explained on this slide, if an organisation is using a validated outcomes tool, they can select the ‘validated outcomes tool’, and then specify who made the assessment. Was it made by the client in say a self-evaluation? Was it by the practitioner? Was it a joint assessment made by the client and the practitioner having a discussion together? Or was the SCORE assessment made by a support person such as a parent or carer for someone who is unable to make their own assessment?

Organisations that are using SCORE directly, or who don’t have a validated tool, should choose the ‘SCORE directly’ options and then specify who made the assessment.

Please note that if an organisation develops its own tool, this would not automatically constitute a ‘validated outcomes tool’, unless it had been formally evaluated, as is the case for clinical assessment tools used by qualified practitioners. A ‘validated tool’ is an instrument that has been psychometrically tested for:

• reliability (the ability of the instrument to produce consistent results)
• validity (the ability of the instrument to produce true results), and
• sensitivity (the probability of correctly identifying a client with the condition).

If you have any questions about this field, please contact the Data Exchange Helpdesk. We will now look at some steps to assist you in measuring outcomes successfully.

• The first step is to make sure you use the resources on the Data Exchange website. There is a lot of helpful information about how to measure outcomes, summarising the key concepts we have discussed in this webinar.
• Through a program logic, identify the connection or direct link between your service and the programs intended outcomes.
• Check that there are no false assumptions between what you are doing and what you expect it to achieve for clients.
• Review the SCORE domains and identify the outcomes that your service is seeking to achieve with clients
• Consider the tools that your service uses. There are a number of validated instruments that have been translated into SCORE, via the Translation Matrix.
• Integrate these tools into service delivery so that it becomes the norm when you are delivering your services.
• We recommend submitting or inputting data regularly over time to the Data Exchange. This way you can update it regularly to ensure that your data is correct.
• Use the reports to test what is submitted, and that it reflects what you expected. Look at the Client outcomes reports to determine what is happening for clients.
• Most importantly, organisations tell us that the shift to outcomes measurement is a powerful tool for front line staff in being able to see the impact they are making in a client’s life. This supports case management and provides a deeper understanding of what is happening for your clients.
• Outcomes measurement itself is just data, the power is in using this data to generate insights that make improvements that benefit clients.

We are now going to look at how you can interpret and use the data that you have entered. The first step is to access your reports.

Go to the Data exchange website and select the log in button. You will come to the MyDEX Dashboard landing page.

Scroll down to click on the Access MyDEX reports. This will take you through to the data in your organisations reports.

A reminder that your reporting permissions will reflect the permissions that have been granted to you by your Data Exchange organisation administrator. For example, if you have been given permission to either edit or view data for one outlet and one program only, then this is the data that will be visible to you in the reports that you access.

The reports landing page will now display which will allow you to selection either Standard reports or the Partnership reports.

If your organisation has not opted into the partnership approach then the partnership reports tile will be greyed out and not accessible.

There are a number of reports available to organisations.

All organisations have access to the two standard reports, the Organisation Overview and Organisation Data Quality Reports.

Organisations that have opted into the Partnership reports can access a further four reports. These reports are most useful for organisations wanting insights into their outcomes data.

When looking at the reports available to you, you will see that there is normally a lot of data. To make this data useful to you and your organisation it is useful to:

• Define the questions you are seeking answers for. This will guide you on the reports to look at. We will walk through a scenario of this shortly.
• Consider the context your services are delivered in - what is happening at the community or local area where your clients are.
• Look for patterns and trends to help see what is occurring for your clients. Focusing on single data points such as gender only, will not allow you to see the higher level groupings and trends.
• This information helps to understand the context and the bigger picture.

The Data Exchange reports will tell you what is happening with your program and clients. The results can lead to a discussion within your organisation as to why this might be occurring.

With the reports you can:

• Monitor the impact of your service – is your program delivering the results that you had anticipated? if not, why not?
• Find the combination of services that lead to the best results. – is there a combination of services that clients use in conjunction with one another that lead to the best results.
• See if the service is meeting client need – you have evidence if you are targeting the right client cohort and what outcomes they are achieving by accessing your services
• Where your clients are coming from to see you – is this expected, are they travelling from areas that are unexpected? Some clients may not be accessing their nearest service, and this is interesting to consider. You may not be aware of a closer service.
• Gain insights into client demographics – who are your clients, where do they come from, what is their background and is this expected?
• Apply your insights to your services and use this information in your continuous improvement processes.

We are going to go through these steps very briefly in the next few slides.

Before you start using your data to generate insights, it’s important to check that the data is of a high quality.

First, use the Organisation overview report, to check if the data captured reflects the clients you have seen:

• Apply the appropriate filters – do you want to look at a particular reporting period? Or a particular program/activity or outlet?
• The attendance and sessions sheet will display how many clients attended and how many sessions were recorded in the selected reporting period. Does this match your organisations records?
• The Client demographics sheet will then give you a breakdown of your clients background – their age, their gender, Indigenous and disability status, etc. Is this expected? Do you have clients where their ages or other details seem incorrect?

Once you have checked the Organisation Overview report and noticed there might be errors, you can use the data quality report to find out the clients where those errors are occurring:

Some handy sheets are:

• The Birth date details sheet
• The Not stated details sheet and
• The unidentified client details sheet.

These sheets will provide the client ID in your system that has an error or anomaly. This makes it easier for you to fix your data quality.

The ‘How can I fix this?’ hyperlink on these sheets will link you to the training resources page so that you can locate task cards to help you in this process. You can also speak to the Data Exchange Helpdesk for information about how to improve data quality.

We can now start to turn our data into insights.

The question we are raising could be: What impact has your program had on clients’ circumstances and the outcomes being achieved in a particular area?
Next consider the context:

What is happening in the community or local area for those clients that might be affecting their situation and outcomes.

The community profiles report will assist with this

• Filter to the area that you want to review
• Select the relevant sheets to see what is happening at a state or community level
• If you are looking at financial counselling the Financial resilience and Material Wellbeing sheets will prove useful
• If you are looking at parenting services, the Family functioning sheet will prove useful
• Also look at other sheets for information that may prove useful including the demographic information available.
• Then apply your local knowledge or your organisation’s knowledge to what is happening in the selected area as this will provide further insight.
• Is the area made up of farmers and is there a severe drought occurring?
• Has the area lost a major industry and unemployment is high?
• Does the area have a high crime rate?
• Is there a high Aged pension rate?

Next check the client outcomes report to view the impact of your services. Filter to your:

• Program/Activity/Outlet
• Reporting period
• Client type – note that this report is only for domains associated with individual clients, a report for unidentified clients is being developed.
• Select the client demographics – do you want to see the impact of your services on clients who are a particular age, gender, Indigenous or disability status, CALD status?

Select the overview sheet in the client outcomes report

• This will provide you with the number of clients that have attended your sessions and the number of clients that have had an assessment? Is this number expected?
• It will also provide you with a percentage of clients that have had a positive outcome in the SCORE component areas of circumstances, goals and satisfaction.

Then select the relevant score component sheet

• For circumstances and goals - What has been the average change? Is this expected?
• For satisfaction – what is the average satisfaction score? Is this expected? It is too high or too low?
• Are the outcomes that your clients are achieving stronger in a particular outlet or a particular program or a particular SCORE domain such as Family functioning or mental health and wellbeing?

Next look for patterns and trends

What patterns are apparent from the data?

• Is there are a particular client demographic attending.
• Is the attendance from a certain area?
• Is this what you expected?
• Is this happening across other outlets and programs?

Looking for patterns and trends will help you see what is occurring for your clients. Focusing on single data points such as gender only, will not allow you to see the higher level groupings and trends.

This information adds to your understanding of the service delivery context and the overall picture of how your service is impacting your clients and region.

The next steps is to apply your insights to your services and use this information in your continuous improvement processes.

Can improvements be made to services in other outlets and programs?

Is further training for staff required? Are additional services required in a particular area to help improve outcomes for clients?

Share your insights with your frontline staff, team and management, to discuss what you are observing in the data. Let them know how they are going, celebrate success and work on creating a culture of continuous improvement.

We hope that the information we have just reviewed is helpful to you. As you can see, there is a range of insights you can gain from the reports, and what will be of most interest you will depend on your organisation’s priorities and the types of programs you are delivering.

Before we finish, one final reminder that more information on outcomes, SCORE and reports can be found on the data exchange website. Don’t forget to subscribe to our mailing list, for the latest updates on the Data Exchange while you’re there.

Once you have reviewed the information and need further assistance, you can also contact our Helpdesk. That brings an end to this webinar today.

Thank you for your attention and we hope that this webinar has provided you with some insight to apply the outcomes data being captured in your service delivery.