Documentation	Build Status	DOI

Explainable AI in Julia.

This package implements interpretability methods for black-box classifiers, with an emphasis on local explanations and attribution maps in input space. The only requirement for the model is that it is differentiable¹. It is similar to Captum and Zennit for PyTorch and iNNvestigate for Keras models.

This package supports Julia ≥1.6. To install it, open the Julia REPL and run

julia> ]add ExplainableAI

Let's explain why an image of a castle is classified as such by a vision model:

using ExplainableAI

# Load model and input
model = ...                      # load classifier model
input = ...                      # input in batch-dimension-last format

# Run XAI method
analyzer = SmoothGrad(model)
expl = analyze(input, analyzer)  # or: analyzer(input)

# Show heatmap
heatmap(expl)

# Or analyze & show heatmap directly
heatmap(input, analyzer)

By default, explanations are computed for the class with the highest activation. We can also compute explanations for a specific class, e.g. the one at output index 5:

analyze(input, analyzer, 5)  # for explanation 
heatmap(input, analyzer, 5)  # for heatmap

Analyzer	Heatmap for class "castle"	Heatmap for class "street sign"
InputTimesGradient
Gradient
SmoothGrad
IntegratedGradients

Depending on the method, the applied heatmapping defaults differ: sensitivity-based methods (e.g. Gradient) default to a grayscale color scheme, whereas attribution-based methods (e.g. InputTimesGradient) default to a red-white-blue color scheme. Red color indicates regions of positive relevance towards the selected class, whereas regions in blue are of negative relevance. More information on heatmapping presets can be found in the Julia-XAI documentation.

Warning

ExplainableAI.jl used to contain Layer-wise Relevance Propagation (LRP). Since version v0.7.0, LRP is now available as part of a separate package in the Julia-XAI ecosystem, called RelevancePropagation.jl.

Analyzer	Heatmap for class "castle"	Heatmap for class "street sign"
LRP with EpsilonPlus composite
LRP with EpsilonPlusFlat composite
LRP with EpsilonAlpha2Beta1 composite
LRP with EpsilonAlpha2Beta1Flat composite
LRP with EpsilonGammaBox composite
LRP with ZeroRule (discouraged)

Check out our talk at JuliaCon 2022 for a demonstration of the package.

Currently, the following analyzers are implemented:

• Gradient
• InputTimesGradient
• SmoothGrad
• IntegratedGradients
• GradCAM

One of the design goals of the Julia-XAI ecosystem is extensibility. To implement an XAI method, take a look at the common interface defined in XAIBase.jl.

In the future, we would like to include:

• PatternNet
• DeepLift
• LIME
• Shapley values via ShapML.jl

Contributions are welcome!

Adrian Hill acknowledges support by the Federal Ministry of Education and Research (BMBF) for the Berlin Institute for the Foundations of Learning and Data (BIFOLD) (01IS18037A).