CHITRA: Clustering Hidden-layer Interpretations through Technical Ranking and Attribution

Abstract

Developing transparent and reliable AI systems demands a deep understanding of how neural networks make decisions. This project introduces CHITRA, a novel algorithm designed for the analysis of neural network hidden layers. CHITRA combines cosine similarity, Singular Value Decomposition (SVD), and K-means clustering to identify and group similar neurons. Its unique approach involves clustering these neurons and tracking their activation paths across different inputs to deduce their function logically.

Existing interpretability techniques like SHAP and LIME often struggle with scalability and consistency in larger, more complex networks. In contrast, experiments using CNNs and RNNs on the MNIST and CIFAR-10 datasets showed that CHITRA outperformed these methods in terms of quantitative metrics, including accuracy, precision, and recall, as well as a human-centered interpretability score.

 

To quantitatively evaluate CHITRA’s performance in identifying neuron function, we established a ‘ground truth’ for a small, manually-labeled subset of neuron clusters. This involved a detailed, manual analysis of the features each cluster was known to detect. For each cluster, we systematically presented the model with specific inputs and recorded its activation patterns. For example, if a cluster consistently showed high activation exclusively for images of the digit ‘5’, we manually labeled it as a ‘5-detector’. This rigorous process created a reliable set of ground-truth labels against which we could compare CHITRA’s predictions.

Our analysis demonstrated that CHITRA’s predictions aligned with these ground-truth labels, showing a notable improvement in quantitative metrics over existing methods. In addition, the algorithm also excelled in qualitative evaluations, such as the human interpretability assessment. Given the past incidents of poorly trained models causing misdiagnoses, significant financial losses, and even fatalities, CHITRA provides a critical tool for researchers to fully understand their models, thereby preventing such incidents in the future.