Taxonomy of Explainable ML

(Explainability Fact Sheets)

Kacper Sokol

Topics

• Dimensions of Explainability
• [F] Functional Requirements
• [U] Usability Requirements
• [O] Operational Requirements
• [S] Safety Requirements
• [V] Validation Requirements
• Examples
• Wrap Up

Dimensions of Explainability

Social and technical explainability desiderata spanning five dimensions

1. functional – algorithmic requirements
2. usability – user-centred properties
3. operational – deployment setting
4. safety – robustness and security
5. validation – evaluation, verification and validation

(Sokol and Flach 2020)

👥   Audience

• 👩‍🔬   Researchers (creators)
• 👨‍💻   Practitioners (users):
  engineers & data scientists
• 🕵️‍♀️   Compliance Personnel (evaluators):
  policymakers & auditors

⚙️️   Operationalisation

• Work Sheets:
  design & development
• Fact Sheets:
  assessment & comparison
• Checklist:
  inspection, compliance, impact & certification

🧰   Applicability

• Explainability Approaches (theory)
• Algorithms (design)
• Implementations (code)

Running Example: Counterfactual Explanations

Had you been 10 years younger,
your loan application would be accepted.

[F] Functional Requirements

• F1 Problem Supervision Level
• F2 Problem Type
• F3 Explanation Target
• F4 Explanation Breadth/Scope
• F5 Computational Complexity

• F6 Applicable Model Class
• F7 Relation to the Predictive System
• F8 Compatible Feature Types
• F9 Caveats and Assumptions

F1 Problem Supervision Level	unsupervised semi-supervised supervised reinforcement
F2 Problem Type	classification probabilistic / non-probabilistic binary / multi-class multi-label regression clustering

F6 Applicable Model Class	model-agnostic model class-specific model-specific
F7 Relation to the Predictive System	ante-hoc (based on endogenous information) post-hoc (based on exogenous information)

F5 Computational Complexity	off-line explanations real-time explanations
F8 Compatible Feature Types	numerical categorical (one-hot encoding)
F9 Caveats and Assumptions	any underlying assumptions, e.g., black box linearity

F3 Explanation Target	data (both raw data and features) models predictions
F4 Explanation Breadth/Scope	local – data point / prediction cohort – subgroup / subspace global

[U] Usability Requirements

• U1 Soundness
• U2 Completeness
• U3 Contextfullness
• U4 Interactiveness
• U5 Actionability
• U6 Chronology

• U7 Coherence
• U8 Novelty
• U9 Complexity
• U10 Personalisation
• U11 Parsimony

U1 Soundness	How truthful it is with respect to the black box?	(✔)
U2 Completeness	How well does it generalise?	(✗)
U3 Contextfullness	“It only holds for people older than 25.”
U11 Parsimony	How short is it?	(✔)

U6 Chronology	More recent events first.
U7 Coherence	Comply with the natural laws (mental model).
U8 Novelty	Avoid stating obvious / being a truism.
U9 Complexity	Appropriate for the audience.

U5 Actionability	Actionable foil.	(✔)
U4 Interactiveness	User-defined foil.	(✔)
U10 Personalisation	User-defined foil.	(✔)

[O] Operational Requirements

• O1 Explanation Family
• O2 Explanatory Medium
• O3 System Interaction
• O4 Explanation Domain
• O5 Data and Model Transparency

• O6 Explanation Audience
• O7 Function of the Explanation
• O8 Causality vs. Actionability
• O9 Trust vs. Performance
• O10 Provenance

O1 Explanation Family	associations between antecedent and consequent contrasts and differences causal mechanisms
O2 Explanatory Medium	(statistical / numerical) summarisation visualisation textualisation formal argumentation
O3 System Interaction	static – one-directional interactive – bi-directional

O4 Explanation Domain	original domain (exemplars, model parameters) transformed domain (interpretable representation)
O5 Data and Model Transparency	transparent/opaque data transparent/opaque model
O6 Explanation Audience	domain experts lay audience

O7 Function of the Explanation	interpretability fairness (disparate impact) accountability (model robustness / adversarial examples)
O8 Causality vs. Actionability	look like causal insights but aren’t
O9 Trust and Performance	truthful to the black-box (perfect fidelity) predictive performance is not affected

O10 Provenance

predictive model data set predictive model and data set (explainability trace)

[S] Safety Requirements

• S1 Information Leakage
• S2 Explanation Misuse
• S3 Explanation Invariance
• S4 Explanation Quality

S1 Information Leakage	Contrastive explanation leak precise values.
S2 Explanation Misuse	Can be used to reverse-engineer the black box.
S3 Explanation Invariance	Does it always output the same explanation (stochasticity / stability)?
S4 Explanation Quality	Is it from the data distribution? How far from a decision boundary (confidence)?

[V] Validation Requirements

• V1 User Studies
• V2 Synthetic Experiments

V1 User Studies	Technical correctness Human biases Unfounded generalisation Usefulness
V2 Synthetic Experiments

Examples

👩‍🔬   Researcher’s   🎩

• 🔍 only works with predictive models that output numbers (F2 Problem Type)
  • Is 🔍 intended for regressors?
  • Can 🔍 be used with probabilistic classifiers?

👩‍🔬   Researcher’s   🎩    

• 🔍 only works with numerical features (F8 Compatible Feature Types)
  • If data have categorical features, is applying one-hot encoding suitable?

👩‍🔬   Researcher’s   🎩    

• 🔍 is model agnostic (F6 Applicable Model Class)
  • Can 🔍 be used with any predictive model?

👩‍🔬   Researcher’s   🎩    

• 🔍 has nice theoretical properties (F9 Caveats and Assumptions)

  The explanation is always [insert your favourite claim here].

  • This claim may not hold for every black-box model (model agnostic explainer)
  • The implementation does not adhere to the claim

👨‍💻   Engineer’s   🎩

• 🔍 explains song recommendations (O7 Function of the Explanation)
• 🔍 explains how users’ listening habits and interactions with the service influence the recommendations (O10 Provenance & U5 Actionability)

👨‍💻   Engineer’s   🎩    

• How does 🔍 scale? (F5 Computational Complexity)
  • Required to serve explanations in real time
  • Will the computational complexity of the algorithm introduce any lags?

👨‍💻   Engineer’s   🎩    

• Music listeners are the recipients of the explanations (O6 Explanation Audience)
  • They are not expected to have any ML experience or background (U9 Complexity)
• They should be familiar with general music concepts (genre, pace, etc.) to appreciate the explanations (O4 Explanation Domain)

👨‍💻   Engineer’s   🎩    

• The explanations will be delivered as snippets of text (O2 Explanatory Medium)
• They will include a single piece of information (U11 Parsimony)
• They are one-directional communication (O3 System Interaction & U4 Interactiveness)

🕵️‍♀️   Auditor’s   🎩

• Are the explanations sound (U1) and complete (U2)?
  • Do they agree with the predictive model?
  • Are they coherent with the overall behaviour of the model?
• Are the explanations placed in a context? (U3 Contextfullness)
  • “This explanation only applies to songs of this particular band.”

🕵️‍♀️   Auditor’s   🎩    

• Will I get the same explanation tomorrow? (S3 Explanation Invariance)
  • Confidence of the predictive model
  • Random effects within the 🔍 algorithm

🕵️‍♀️   Auditor’s   🎩    

• Does the explainer leak any sensitive information? (S1 Information Leakage)
  • →explanation←
    “Had you been older than 30, your loan application would have been approved.”
  • →context←
    “This age threshold applies to people whose annual income is upwards of £25,000.”
• Why don’t I “round up” my income the next time? (S2 Explanation Misuse)

🕵️‍♀️   Auditor’s   🎩    

• Was 🔍 validated for the problem class that it is being deployed on? (V2 Synthetic Validation)
• Does 🔍 improve users’ understanding? (V1 User Studies)

LIME Explainability Fact Sheet

Wrap Up

Challenges

• The desiderata list is neither exhaustive nor prescriptive

• Some properties are incompatible or competing – choose wisely and justify your choices
  • Should I focus more on property F42 or F44?
  • For O13, should I go for X or Y?
• Other properties cannot be answered uniquely
  • E.g., coherence with the user’s mental model

• The taxonomy does not define explainability

• We will try to define explainability in a separate module

Summary

• Explainability is characterised by a broad range of diverse properties
• Striking the right ballance may be challenging
• Desiderata include social and technical aspects of explainability
• Having a readily available list of properties helps to better design XML systems

Bibliography

Sokol, Kacper, and Peter Flach. 2020. “Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 56–67.

Questions

kacper.sokol@rmit.edu.au
k.sokol@bristol.ac.uk