
Our third study of the research blog looks at a study that developed an artificial Intelligence model to recognise upper GI bleeding on digital subtraction angiography imaging. (1)
Study Objective
The study aimed to evaluate an artificial intelligence (AI) model in identifying active bleeding. The study focused on digital subtraction angiography (DSA) images of mesenteric and celiac arteries in patients with upper gastrointestinal bleeding (UGIB). The aim was to focus on automating the detection of extravasation (active bleeding), which would assist interventional radiologists during angiographic procedures.
Methods Used
Data Collection:Â The study was a retrospective single centre study, involving DSA images from angiographic procedures performed between 2018 and 2022. A total of 587 images from 142 patients were included. 302 were labeled as normal and 285 showing active bleeding.
AI Model:Â A convolutional neural network (CNN) using the EfficientNet-B5 architecture was used. The model was pre-trained on a ImageNet dataset and the model was fine-tuned with the DSA images. The images were labeled by two senior radiologists (including one abdominal radiologist) and then split into training (80%) and validation (20%) subsets.
Performance Metrics:Â Model performance was evaluated using a variety of metrics including but not limited to: area under the curve (AUC), accuracy, sensitivity, specificity, F1 score, positive predictive value (PPV), and negative predictive value (NPV). Cross-validation (fivefold) was used.Â
Results
Dataset: 587 DSA images were processed, with 302 labeled as normal and 285 with active bleeding. The average Cohen’s kappa score for inter-rater reliability was 0.775, indicating strong agreement between the two radiologists.
AI Model Performance:Â The model achieved an AUC of 98.8% for the training cohort and 85.0% for the validation cohort. The average classification accuracy was 87.6% for training and 77.3% for validation. Sensitivity at the Youden's index cutoff was 85.4%, and specificity was 81.2%. Other key metrics included an NPV of 85.6%, PPV of 78.1%, and F1 score of 81.2%.
Time Efficiency:Â The AI model classified images at an average rate of 166 ms per image.
Visualization:Â Class activation maps (CAMs) were used to visually represent the areas in the images most influential in the model's decision-making process.
Conclusion
This study demonstrated that the AI model could accurately identify images with active bleeding during mesenteric and celiac artery angiography. With high sensitivity and reasonable specificity, the AI model has the potential to support interventional radiologists in making faster and more accurate decisions during procedures.
Strengths of the Study
The use of CAMs provided visual interpretability of the AI model's decision process, improving understanding of the model’s outputs.
The Cohen’s kappa score of 0.775 indicated reliable image labelling by radiologists initially.Â
Limitations of the Study
Retrospective Design:Â The study was retrospective.
Dataset Limitation:Â The model was trained and validated using images from a single centre, which could limit the generalisability to other institutions.Â
Exclusion of Motion-Artifacts: The exclusion of images with motion artefacts, while ensuring high-quality data, may affect the model’s robustness in a real world setting, where such artefacts are commonplace.
External Validity:Â The model's performance was not tested on external datasets, which may limit applicability in broader contexts.Â
Opportunities for Future Research
Larger, Multi-centre Studies:Â Future studies could involve larger, more diverse datasets from multiple centres to assess the model's generalisability.Â
Incorporating Motion-Artifact Images:Â The model should be trained and tested on images with motion artefacts to evaluate its performance in real-world clinical conditions.
AI-Enhanced Decision Support:Â The model could be integrated into clinical workflows for real-time assistance during angiography, and then its effectiveness in reducing procedural time and improving patient outcomes could be assessed.Â
External Validation: Prospective studies across multiple institutions could help validate the AI model’s external applicability.Â
Overall, this study demonstrates the promising role of AI in aiding interventional radiology for UGIB, though further validation and refinement are needed for broader implementation. Thank you for reading and see you next time on our research blog!Â
(1) Barash, Y., Livne, A., Klang, E., Sorin, V., Cohen, I., Khaitovich, B. and Raskin, D., 2024. Artificial intelligence for identification of images with active bleeding in mesenteric and celiac arteries angiography. CardioVascular and Interventional Radiology, 47(6), pp.785-792.
Comments