What goes into an explanation?

At a glance

You need to consider how to provide information on two subcategories of explanation:

• process-based explanations which give you information on the governance of your AI system across its design and deployment; and
• outcome-based explanations which tell you what happened in the case of a particular decision.

There are different ways of explaining AI decisions. We have identified six main types of explanation:

• Rationale explanation: the reasons that led to a decision, delivered in an accessible and non-technical way.
• Responsibility explanation: who is involved in the development, management and implementation of an AI system, and who to contact for a human review of a decision.
• Data explanation: what data has been used in a particular decision and how.
• Fairness explanation: steps taken across the design and implementation of an AI system to ensure that the decisions it supports are generally unbiased and fair, and whether or not an individual has been treated equitably.
• Safety and performance explanation: steps taken across the design and implementation of an AI system to maximise the accuracy, reliability, security and robustness of its decisions and behaviours.
• Impact explanation: steps taken across the design and implementation of an AI system to consider and monitor the impacts that the use of an AI system and its decisions has or may have on an individual, and on wider society.

In more detail

• What do we mean by ‘explanation’?
• Process-based vs outcome-based explanations
• Rationale explanation
• Responsibility explanation
• Data explanation
• Fairness explanation
• Safety and performance explanation
• Impact explanation

What do we mean by ‘explanation’?

The Cambridge dictionary defines ‘explanation’ as:

“The details or reasons that someone gives to make something clear or easy to understand.”

While this is a general definition, it remains valid when considering how to explain AI- assisted decisions to the individuals affected (who are often also data subjects). It suggests that you should not always approach explanations in the same way. What people want to understand, and the ‘details’ or ‘reasons’ that make it ‘clear’ or ‘easy’ for them to do so may differ.

Our own research, and that of others, reveals that context is a key aspect of explaining decisions involving AI. Several factors about the decision, the person, the application, the type of data, and the setting, all affect what information an individual expects or finds useful.

Therefore, when we talk about explanations in this guidance, we do not refer to just one approach to explaining decisions made with the help of AI, or to providing a single type of information to affected individuals. Instead, the context affects which type of explanation you use to make an AI-assisted decision clear or easy for individuals to understand.

You should remember which audience you are aiming your explanation at, such as:

• staff whose decisions are supported by the AI system and who need to relay meaningful information to an individual affected by the AI-assisted decisions;
• those affected by the decision, with particular thought given to:
  • vulnerable groups; and
  • children; or
• auditors or external reviewers who are charged with monitoring or overseeing the production and deployment of the system.

Each group may require different levels of detail within the explanation they receive. The level of knowledge that the explanation recipient has about the subject of the explanation will affect the detail and language you need to use.

You should also take into account the transparency requirements of the GDPR, which (at least in cases of solely automated AI decisions) includes:

• providing meaningful information about the logic, significance and envisaged consequences of the AI decision;
• the right to object; and
• the right to obtain human intervention.

Where there is a “human in the loop” you still have to comply with the transparency requirements. In these situations, you should consider information about the decisions or recommendations made by the system and how this informs the human decision.

As a result of our research and engagement we identified six different types of explanation. You can combine these into your explanation in various ways depending on the specific decision and the audience you are clarifying that decision about. You may not need to supply a decision recipient with information about all of these explanation types. However, you should consider what information will be required by all affected individuals, as well as the context in which that decision will be made, and plan your explanation accordingly.

You should also keep in mind that your system will need to be appropriately explainable to others who are, or may be, involved in the decision process. This could include, for example, implementers using the system or auditors checking up on it. These explanation types are designed to help you do this in a concise and clear way.

Process-based vs outcome-based explanations

Before we explore the six explanation types, it is useful to make a distinction, which applies to all of them, between process-based and outcome-based explanations.

The primary aim of explaining fully automated or AI-assisted decisions is justifying a particular result to the individual whose interests are affected by it. This means:

• demonstrating how you and all others involved in the development of your system acted responsibly when choosing the processes behind its design and deployment; and
• making the reasoning behind the outcome of that decision clear.

We have therefore divided each type of explanation into the subcategories of ‘process’ and ‘outcome’:

• Process-based explanations of AI systems are about demonstrating that you have followed good governance processes and best practices throughout your design and use.

  For example, if you are trying to explain the fairness and safety of a particular AI-assisted decision, one component of your explanation will involve establishing that you have taken adequate measures across the system’s production and deployment to ensure that its outcome is fair and safe.

• Outcome-based explanations of AI systems are about clarifying the results of a specific decision. They involve explaining the reasoning behind a particular algorithmically-generated outcome in plain, easily understandable, and everyday language. 

  If there is meaningful human involvement in the decision-making process, you also have to make clear to the affected individual how and why a human judgement that is assisted by an AI output was reached.

  In addition, you may also need to confirm that the actual outcome of an AI decision meets the criteria that you established in your design process to ensure that the AI system is being used in a fair, safe, and ethical way. 

Rationale explanation

What does this explanation help people to understand?

It is about the ‘why?’ of an AI decision. It helps people understand the reasons that led to a decision outcome, in an accessible way.

What purposes does this explanation serve?

• Challenging a decision

  It is vital that individuals understand the reasons underlying the outcome of an automated decision, or a human decision that has been assisted by the results of an AI system. If the decision was not what they wanted or expected, this allows them to assess whether they believe the reasoning of the decision is flawed. If they wish to challenge the decision, knowing the reasoning supports them to formulate a coherent argument for why they think this is the case.

• Changing behaviour

  Alternatively, if an individual feels the reasoning for the decision was sound, they can use this knowledge to consider how they might go about changing their behaviour, or aspects of their lifestyle, to get a more favourable outcome in the future. If the individual is already satisfied with the outcome of the AI decision, the rationale explanation may still be useful so that they may validate their belief of why this was the case, or adjust it if the reasons for the favourable outcome were different to those they expected.

What you may need to show

• How the system performed and behaved to get to that decision outcome.
• How the different components in the AI system led it to transform inputs into outputs in a particular way, so you can communicate which features, interactions, and parameters were most significant.
• How these technical components of the logic underlying the result can provide supporting evidence for the decision reached.
• How this underlying logic can be conveyed as easily understandable reasons to decision recipients.
• How you have thought about how the system’s results apply to the concrete context and life situation of the affected individual.

Rationale explanations might answer:

• Have we selected an algorithmic model, or set of models, that will provide a degree of interpretability that corresponds with its impact on affected individuals?
• Are the supplementary explanation tools that we are using to help make our complex system explainable good enough to provide meaningful and accurate information about its underlying logic?

What information goes into rationale explanations

As with the other types of explanation, rationale explanations can be process-based or outcome-based.

Process-based explanations clarify:

• How the procedures you have set up help you provide meaningful explanations of the underlying logic of your AI model’s results.
• How these procedures are suitable given the model’s particular domain context and its possible impacts on the affected decision recipients and wider society.
• How you have set up your system’s design and deployment workflow so that it is appropriately interpretable and explainable, including its data collection and pre-processing, model selection, explanation extraction, and explanation delivery procedures.

Outcome-based explanations provide:

• The formal and logical rationale of the AI system – how the system is verified against its formal specifications, so you can verify that the AI system will operate reliably and behave in accordance with its intended functionality.
• The technical rationale of the system’s output – how the model’s components (its variables and rules) transform inputs into outputs, so you know what role these components play in producing that output. By understanding the roles and functions of the individual components, it is possible to identify the features and parameters that significantly influence a particular output.
• Translation of the system’s workings – its input and output variables, parameters and so on – into accessible everyday language, so you can clarify, in plain and understandable terms, what role these factors play in reasoning about the real-world problem that the model is trying to address or solve.
• Clarification of how a statistical result is applied to the individual concerned. This should show how the reasoning behind the decision takes into account the specific circumstances, background and personal qualities of affected individuals.

The GDPR also refers to providing meaningful information about the logic involved in automated decision-making under Articles 13, 14 and 15.

In order to be able to derive your rationale explanation, you need to know how your algorithm works. See Step 3 of Part 2 of this guidance for more detail about how to do this.

How can the guidance can help me with this?

See Part 2 (Explaining AI in practice) for more information on extracting the technical rationale and translating this into understandable and context-sensitive reasons.

Responsibility explanation

What does this explanation help people to understand?

It helps people understand ‘who’ is involved in the development and management of the AI model, and ‘who’ to contact for a human review of a decision.

What purposes does this explanation serve?

• Challenging a decision

  Individuals in receipt of other explanations, such as rationale or fairness, may wish to challenge the AI decision based on the information provided to them. The responsibility explanation helps by directing the individual to the person or team responsible for carrying a human review of a decision. It also makes accountability traceable.

• Informative

  This explanation can also serve an informative purpose by shedding some light on the different parts of your organisation involved in the design and deployment of your AI decision-support system.

What you may need to show

• Who is accountable at each stage of the AI system’s design and deployment, from defining outcomes for the system at its initial phase of design, through to providing the explanation to the affected individual at the end.
• Definitions of the mechanisms by which each of these people will be held accountable, as well as how you have made the design and implementation processes of your AI system traceable and auditable.

What information goes into responsibility explanations

Process-based explanations clarify:

• The roles and functions across your organisation that are involved in the various stages of developing and implementing your AI system, including any human involvement in the decision-making. If your system, or parts of it, are procured, you should include information about the providers or developers involved.
• Broadly, what the roles do, why they are important, and where overall responsibility lies for management of the AI model – who is ultimately accountable.
• Who is responsible at each step from the design of an AI system through to its implementation to make sure that there is effective accountability throughout.

Outcome-based explanations:

Because a responsibility explanation largely has to do with the governance of the design and implementation of AI systems, it is, in a strict sense, entirely process-based. Even so, there is important information about post-decision procedures that you should be able to provide:

• Cover information on how to request a human review of an AI-enabled decision or object to the use of AI, including details on who to contact, and what the next steps will be (eg how long it will take, what the human reviewer will take into account, how they will present their own decision and explanation).
• Give individuals a way to directly contact the role or team responsible for the review. You do not need to identify a specific person in your organisation. One person involved in this should have implemented the decision, and used the statistical results of a decision-support system to come to a determination about an individual.

How can the guidance help me with this?

See Part 3 of this guidance (What explaining AI means for your organisation) for more information on identifying the roles involved in explaining an AI-assisted decision. See Part 2 of this guidance (Explaining AI in practice) for more details on the information you need to provide for this explanation.

Data explanation

What does this explanation help people to understand?

Data explanations are about the ‘what’ of AI-assisted decisions. They help people understand what data about them, and what other sources of data, were used in a particular AI decision. Generally, they also help individuals understand more about the data used to train and test the AI model. You could provide some of this information within the fair processing notice you are required to provide under Articles 13 and 14 of the GDPR.

What purposes does this explanation serve?

• Challenging a decision

  Understanding what data was input into an AI decision-support system will allow an individual to challenge the outcome if they think it was flawed (eg if some of the input data was incorrect or irrelevant, or additional data wasn’t taken into account that the individual thinks is relevant).

• Providing reassurance

  Knowing more about the actions you took when collecting and preparing the training and test data for your AI model will help to reassure individuals that you made appropriate and responsible choices in the best interests of developing an understandable, fair and accurate AI decision-support system.

What you may need to show

• How the data used to train, test, and validate your AI model was managed and utilised from collection through processing and monitoring.
• What data you used in a particular decision and how.

What information goes into data explanations

Process-based explanations include:

• What training/ testing/ validating data was collected, the sources of that data, and the methods that were used to collect it.
• Who took part in choosing the data to be collected or procured and who was involved in its recording or acquisition. How procured or third-party provided data was vetted.
• How data quality was assessed and the steps that were taken to address any quality issues discovered, such as completing or removing data.
• What the training/ testing/ validating split was and how it was determined.
• How data pre-processing, labelling, and augmentation supported the interpretability and explainability of the model.
• What measures were taken to ensure the data used to train, test, and validate the system was representative, relevant, accurately measured, and generalisable.
• How you ensured that any potential bias and discrimination in the dataset have been mitigated.

Outcome-based explanations:

• Clarify the input data used for a specific decision, and the sources of that data. This is outcome-based because it refers to your AI system’s result for a particular decision recipient.
• In some cases the output data may also require an explanation, particularly where the decision recipient has been placed in a category which may not be clear to them. For example, in the case of anomaly detection for financial fraud identification, the output might be a distance measure which places them at a certain distance away from other people based on their transaction history. Such a classification may require an explanation.

How can the guidance help me with this?

See the data collection section in Part 2 (Explaining AI in practice) for more on deriving this explanation type.

In Part 3 (What explaining AI means for your organisation) we provide some further pointers on how to document and demonstrate responsible data management practices across the design and implementation of your AI model.

Fairness explanation

What does this explanation help people to understand?

The fairness explanation is about helping people understand the steps you took (and continue to take) to ensure your AI decisions are generally unbiased and equitable. It also gives people an understanding of whether or not they have been treated equitably themselves.

What purposes does this explanation serve?

• Trust

  The fairness explanation is key to increasing individuals’ confidence in your AI system. You can foster trust by explaining to an individual how you avoid bias and discrimination in the AI-assisted decisions you make and by proving that they were not treated differently than others like them.

• Challenging a decision

  It also allows individuals to challenge a decision made using an AI system. An individual might feel the explanation you provide actually suggests they were treated unfairly.

What you may need to show

An explanation of fairness can relate to several stages of the design, development and deployment of AI systems:

Dataset fairness: The system is trained and tested on properly representative, relevant, accurately measured, and generalisable datasets (note that this dataset fairness component will overlap with data explanation). This may include showing that you have made sure your data is:

• as representative as possible of all those affected;
• sufficient in terms of its quantity and quality, so it represents the underlying population and the phenomenon you are modelling;
• assessed and recorded through suitable, reliable and impartial sources of measurement and has been sourced through sound collection methods;
• up-to-date and accurately reflects the characteristics of individuals, populations and the phenomena you are trying to model; and
• relevant by calling on domain experts to help you understand, assess and use the most appropriate sources and types of data to serve your objectives.

Design fairness: It has model architectures that do not include target variables, features, processes, or analytical structures (correlations, interactions, and inferences) which are unreasonable or unjustifiable. This may include showing that you have done the following:

• Attempted to identify any underlying structural biases that may play a role in translating your objectives into target variables and measurable proxies. When defining the problem at the start of the AI project, these biases could influence what system designers expect target variables to measure and what they statistically represent.
• Mitigated bias in the data pre-processing phase by taking into account the sector or organisational context in which you are operating. When this process is automated or outsourced, show that you have reviewed what has been done, and maintained oversight. You should also attach information on the context to your metadata, so that those coming to the pre-processed data later on have access to the relevant properties when they undertake bias mitigation.
• Mitigated bias when the feature space was determined (ie when relevant features were selected as input variables for your model). Choices made about grouping or separating and including or excluding features, as well as more general judgements about the comprehensiveness or coarseness of the total set of features, may have consequences for protected groups of people.
• Mitigated bias when tuning parameters and setting metrics at the modelling, testing and evaluation stages (ie into the trained model). Your AI development team should iterate the model and peer review it to help ensure that how they choose to adjust the dials and metrics of the model are in line with your objectives of mitigating bias.
• Mitigated bias by watching for hidden proxies for discriminatory features in your trained model, as these may act as influences on your model’s output. Designers should also look into whether the significant correlations and inferences determined by the model’s learning mechanisms are justifiable.

Outcome fairness: It does not have discriminatory or inequitable impacts on the lives of the people it affects. This may include showing that:

• you have been explicit about the formal definition(s) of fairness you have chosen and why. Data scientists can apply different formalised fairness criteria to choose how specific groups in a selected set will receive benefits in comparison to others in the same set, or how the accuracy or precision of the model will be distributed among subgroups; and
• the method you have applied in operationalising your formalised fairness criteria, for example, by reweighting model parameters; embedding trade-offs in a classification procedure; or re-tooling algorithmic results to adjust for outcome preferences.

Implementation fairness: It is deployed by users sufficiently trained to implement it responsibly and without bias. This may include showing that you have appropriately prepared and trained the implementers of your system to:

• avoid automation bias (over-relying on the outputs of AI systems) or automation-distrust bias (under-relying on AI system outputs because of a lack of trust in them);
• use its results with an active awareness of the specific context in which they are being applied. They should understand the particular circumstances of the individual to which that output is being applied; and
• understand the limitations of the system. This includes understanding the statistical uncertainty associated with the result as well as the relevant error rates and performance metrics.

What information goes into fairness explanations

This explanation is about providing people with appropriately simplified and concise information on the considerations, measures and testing you carry out to make sure that your AI system is equitable and that bias has been optimally mitigated. Fairness considerations come into play through the whole lifecycle of an AI model, from inception to deployment, monitoring and review.

Process-based explanations include details about:

• your chosen measures to mitigate risks of bias and discrimination at the data collection, preparation, model design and testing stages;
• how these measures were chosen and how you have managed informational barriers to bias-aware design such as limited access to data about protected or sensitive traits of concern; and
• the results of your initial (and ongoing) fairness testing, self-assessment, and external validation – showing that your chosen fairness measures are deliberately and effectively being integrated into model design. You could do this by showing that different groups of people receive similar outcomes, or that protected characteristics have not played a factor in the results.

Outcome-based explanations include:

• details about how your formal fairness criteria were implemented in the case of a particular decision or output;
• presentation of the relevant fairness metrics and performance measurements in the delivery interface of your model. This should be geared to a non-technical audience and done in an easily understandable way; and
• explanations of how others similar to the individual were treated (ie whether they received the same decision outcome as the individual). For example, you could use information generated from counter-factual scenarios to show whether or not someone with similar characteristics, but of a different ethnicity or gender, would receive the same decision outcome as the individual.

How can the guidance help me with this?

See Part 2 (Explaining AI in practice) for more information on building fairness into the design and deployment of your AI model. See also Part 3 (What explaining AI means for your organisation) for information on how to document what you have done to achieve fairness.

Safety and performance explanation

What does this explanation help people to understand?

The safety and performance explanation helps people understand the measures you have put in place, and the steps you have taken (and continue to take) to maximise the accuracy, reliability, security and robustness of the decisions your AI model helps you to make. It can also be used to justify the type of AI system you have chosen to use, such as comparisons to other systems or human decision makers.

What purposes does this explanation serve?

• Reassurance

  Individuals often want to be reassured that an AI system is safe and reliable. The safety and performance explanation helps to serve this purpose by demonstrating what you have done to test and monitor the accuracy, reliability, security and robustness of your AI model.

• Informative

  If an individual receiving an explanation of an AI-assisted decision is technically knowledgeable or proficient, this explanation will allow them to assess the suitability of the model and software for the types of decision being made. This explanation helps you to be as transparent as you can with people about the integrity of your AI decision-support system.

• Challenging a decision

  Individuals can make an informed choice about whether they want to contest an AI decision on the basis that it may be incorrect for them, or carried out in an unsafe, hazardous, or unreliable way. This is closely linked with challenging a decision on the basis of fairness. 

What you may need to show

Accuracy: the proportion of examples for which your model generates a correct output. This component may also include other related performance measures such as precision, sensitivity (true positives), and specificity (true negatives). Individuals may want to understand how accurate, precise, and sensitive the output was in their particular case.

Reliability: how dependably the AI system does what it was intended to do. If it did not do what it was programmed to carry out, individuals may want to know why, and whether this happened in the process of producing the decision that affected them.

Security: the system is able to protect its architecture from unauthorised modification or damage of any of its component parts; the system remains continuously functional and accessible to its authorised users and keeps confidential and private information secure, even under hostile or adversarial conditions.

Robustness: the system functions reliably and accurately in practice. Individuals may want to know how well the system works if things go wrong, how this has been anticipated and tested, and how the system has been immunised from adversarial attacks.

What information goes into safety and performance explanations

Process-based explanations include:

For accuracy:

• How you measure it (eg maximising precision to reduce the risk of false negatives).
• Why you chose those measures, and how you went about assuring it.
• What you did at the data collection stage to ensure your training data was up-to-date and reflective of the characteristics of the people to whom the results apply.
• What kinds of external validation you have undertaken to test and confirm your model’s ‘ground truth’.
• What the overall accuracy rate of the system was at testing stage.
• What you do to monitor this (eg measuring for concept drift over time).

For reliability:

• How you measure it and how you went about assuring it.
• Results of the formal verification of the system’s programming specifications, ie how encoded requirements have been mathematically verified.

For security:

• How you measure it and how you went about assuring it, eg how limitation have been set on who is able to access the system, when, and how.
• How you manage the security of confidential and private information that is processed in the model.

For robustness:

• How you measure it.
• Why you chose those measures.
• How you went about assuring it, eg how you’ve stress-tested the system to understand how it responds to adversarial intervention, implementer error, or skewed goal-execution by an automated learner (in reinforcement learning applications).

Outcome-based explanations:

While you may not be able to guarantee accuracy at an individual level, you should be able to provide assurance that, at run-time, your AI system operated reliably, securely, and robustly for a specific decision.

• In the case of accuracy and the other performance metrics, however, you should include in your model’s delivery interface the results of your cross-validation (training/ testing splits) and any external validation carried out.
• You may also include relevant information related to your system’s confusion matrix (the table that provides the range of performance metrics) and ROC curve (receiver operating characteristics)/ AUC (area under the curve). Include guidance for users and affected individuals that makes the meaning of these measurement methods, and specifically the ones you have chosen to use, easily accessible and understandable. This should also include a clear representation of the uncertainty of the results (eg confidence intervals and error bars).

How can the guidance help me with this?

See Part 2 (Explaining AI in practice) for more information on ensuring the accuracy, reliability, security and robustness of your AI system. See also Part 3 (What explaining AI means for your organisation) for information on how to document what you have done to achieve these objectives.

Impact explanation

What does this explanation help people to understand?

An impact explanation helps people understand how you have considered the effects that your AI decision-support system may have on an individual, ie what the outcome of the decision means for them. It is also about helping individuals to understand the broader societal effects that the use of your system may have. This may help reassure people that the use of AI will be of benefit. Impact explanations are therefore often well suited to delivery before an AI-assisted decision has been made. See Task 6 of Explaining AI in practice for guidance on when to deliver explanations.

What purposes does this explanation serve?

• Consequences

  The purpose of the impact explanation is primarily to give individuals some power and control over their involvement in an AI-assisted decision made about them. By understanding the possible consequences of the decision (negative, neutral and positive) an individual can better assess their willingness to take part in the process, and can anticipate how the outcomes of the decision may affect them.

• Reassurance

  Knowing that you took the time to consider and manage the potential effects that your AI system has on society can help to reassure individuals that issues such as safety, equity, and reliability are core components of the AI model they are subject to. It also helps individuals to be more informed about the benefits and risks of AI decision-support systems, and therefore, more confident and active in the debate about its development and use.

What you may need to show

• Demonstrate that you have thought about how your AI system will potentially affect individuals and wider society. Clearly show affected individuals the process you have gone through to determine these possible impacts.

What information goes into impact explanations

Process-based explanations include:

• Showing the considerations you gave to your AI system’s potential effects, how you undertook these considerations, and the measures and steps you took to mitigate possible negative impacts on society, and to amplify the positive effects.
• Information about how you plan to monitor and re-assess impacts while your system is deployed.

Outcome-based explanations:

Although the impact explanation is mainly about demonstrating that you have put appropriate forethought into the potential ‘big picture’ effects, you should also consider how to help decision recipients understand the impact of the AI-assisted decisions that specifically affect them. For instance, you might explain the consequences for the individual of the different possible decision outcomes and how, in some cases, changes in their behaviour would have brought about a different outcome with more positive impacts. This use of counterfactual assessment would help decision recipients make changes that could lead to a different outcome in the future, or allow them to challenge the decision.

How can the guidance help me with this?

See Part 2 (Explaining AI in practice) for more information on considering impact in how you select an appropriately explainable AI model. See also Part 3 (What explaining AI means for your organisation) for information on how to document this.