TY - JOUR
T1 - Supporting Deep Neural Network Safety Analysis and Retraining through Heatmap-Based Unsupervised Learning
AU - Fahmy, Hazem
AU - Pastore, Fabrizio
AU - Bagherzadeh, Mojtaba
AU - Briand, Lionel
N1 - Publisher Copyright:
© 1963-2012 IEEE.
PY - 2021/12/1
Y1 - 2021/12/1
N2 - Deep neural networks (DNNs) are increasingly important in safety-critical systems, for example, in their perception layer to analyze images. Unfortunately, there is a lack of methods to ensure the functional safety of DNN-based components. We observe three major challenges with existing practices regarding DNNs in safety-critical systems: 1) Scenarios that are under-represented in the test set may lead to serious safety violation risks but may, however, remain unnoticed; 2) characterizing such high-risk scenarios is critical for safety analysis; 3) retraining DNNs to address these risks is poorly supported when causes of violations are difficult to determine. To address these problems in the context of DNNs analyzing images, we propose heatmap-based unsupervised debugging of DNNs (HUDD), an approach that automatically supports the identification of root causes for DNN errors. HUDD identifies root causes by applying a clustering algorithm to heatmaps capturing the relevance of every DNN neuron on the DNN outcome. Also, HUDD retrains DNNs with images that are automatically selected based on their relatedness to the identified image clusters. We evaluated HUDD with DNNs from the automotive domain. HUDD was able to identify all the distinct root causes of DNN errors, thus supporting safety analysis. Also, our retraining approach has shown to be more effective at improving DNN accuracy than existing approaches.
AB - Deep neural networks (DNNs) are increasingly important in safety-critical systems, for example, in their perception layer to analyze images. Unfortunately, there is a lack of methods to ensure the functional safety of DNN-based components. We observe three major challenges with existing practices regarding DNNs in safety-critical systems: 1) Scenarios that are under-represented in the test set may lead to serious safety violation risks but may, however, remain unnoticed; 2) characterizing such high-risk scenarios is critical for safety analysis; 3) retraining DNNs to address these risks is poorly supported when causes of violations are difficult to determine. To address these problems in the context of DNNs analyzing images, we propose heatmap-based unsupervised debugging of DNNs (HUDD), an approach that automatically supports the identification of root causes for DNN errors. HUDD identifies root causes by applying a clustering algorithm to heatmaps capturing the relevance of every DNN neuron on the DNN outcome. Also, HUDD retrains DNNs with images that are automatically selected based on their relatedness to the identified image clusters. We evaluated HUDD with DNNs from the automotive domain. HUDD was able to identify all the distinct root causes of DNN errors, thus supporting safety analysis. Also, our retraining approach has shown to be more effective at improving DNN accuracy than existing approaches.
KW - DNN explanation
KW - DNN functional safety analysis
KW - Deep neural network (DNN) debugging
KW - heatmaps
UR - http://www.scopus.com/inward/record.url?scp=85107204451&partnerID=8YFLogxK
U2 - 10.1109/TR.2021.3074750
DO - 10.1109/TR.2021.3074750
M3 - Article
AN - SCOPUS:85107204451
SN - 0018-9529
VL - 70
SP - 1641
EP - 1657
JO - IEEE Transactions on Reliability
JF - IEEE Transactions on Reliability
IS - 4
ER -