AI-Driven 3D Visualization for Intracranial Tumour Analysis

This project delves into the application of 3D visualization techniques, enhanced by artificial intelligence, to aid in the detection and analysis of intracranial tumours. The study utilizes radiological images, including MRI and CT scans, and employs image segmentation and registration techniques to create detailed 3D models of the brain and tumours. The integration of AI, particularly deep learning algorithms, enhances the accuracy of segmentation and provides quantitative assessments of radiographic characteristics. The research highlights the benefits of 3D visualization in overcoming the limitations of 2D medical images, improving diagnostic accuracy, and assisting surgeons in surgical planning. The project also explores the use of AI in identifying biological features and automating the segmentation process, ultimately improving the efficiency and effectiveness of medical image processing. Various algorithms and techniques such as FCM algorithm, SLANT-27 whole brain segmentation method and DeepMedicUS are discussed and compared. The goal is to develop image guidance and visualization methods to overcome the complexities in brain tumour surgery and improve patient outcomes.

3D visualization in aiding intracranial tumour through radiological images
and image segmentation by using image registration and AI
Highlights
The highlights of the work are as follows:
1) The area on which this project focuses is Virtual Reality and Augmented Reality. Virtual reality
(VR) and augmented reality (AR) act as the opposite sides of a same coin. Both of these are there to
extend sensitive environment of any person by the medium of reality via the help of technology.
Virtual Reality depends on the alternative way of setting in order to provide experience while
Augmented Reality is there to improve the existing elements by adding more layers of meaning.
These are the thriving technologies and as per the reports of Augmented/Virtual Report of the year
2017, the VR/AR market would be reaching $108 billion of revenue by the year 2021 while as per the
overview given by () this business will reach a market value of $80 in the year 2025.
2) Radiological images and image segmentation will also be discussed upon. Computerised
tomography (CT), magnetic resonance imaging (MRI), ultrasound and positron emission tomography
(PET) are some of the examples of modalities related to imaging. A radiologist provides views on the
images that are produced by a scanner (Lee & Wong, 2019). Segmentation refers to extracting
specific region of the images as these are required in partitioning image data to associated elements.
Image segmentation gives quantitative data about some relevant anatomy such as to determine the
volume or the size and many more. This enables 3D visualization of a specific structure with accuracy
by making use of surface triangulation and volume rendering.
3) Extraction of 3D models by making use of various image registration techniques and AI will also
be included.
4) The main topic if the paper is combining the above said techniques that is “3D visualization in
aiding intracranial tumour through radiological images and image segmentation by using image
registration and AI”
Abstract
Background and Aim
The advancements in the sphere of imaging in medical science are a revolution for the
medicines. It is helpful in determining the presence or the level of severity of any disease that has an
impact on the clinical care of patients or the related outcomes in a specific study. Three dimensional
techniques has been a great help in intracranial tumour through the help of radiological images and
image segmentation technique. Accuracy is a vital factor in segmentation and thus 3D visualization
makes the technique more appropriate and perfect. The use of artificial intelligence in this sphere will
also be a part of the study.
1

Methodology
Artificial Intelligence algorithms have specifically deep learning have shown remarkable
progress in the field of image recognition. In radiological practice the trained physicians used to
visually assess the medical images for detecting, characterizing and monitoring diseases. The AI
implementation in image segmentation enhances the process of recognizing patterns in the data that
has been imaged thus providing quantitative assessments of radiographic characteristics. It is difficult
to find the exact location of intracranial tumours based on the 2D medical images thus the 3D
technology of visualizing medical images is being used to obtain accuracy in the field.
Introduction
Guidance and visualization of images are a vital part in the sphere of modern surgery and this
helps the surgeons in performing surgical procedures. In this paper the focus is on intracranial tumour
detection by the help of radiological images. 3D technology in visualization has enhanced the process.
To understand the system, intracranial tumour needs to be understood. The images help locating the
tumour and these images help in surgery. In brain surgery, the brain changed shape when the skull is
opened and this kind of deformation of the brain is termed as brain shift. The boundaries of these
tumours are not easy to identify thus developing image guidance as well as visualization methods in
case of the surgery of tumour will help in overcoming the complicacies in this specific type of
surgery. Medical visualization of the organs is essential for accuracy in the diagnosis being made and
surgery requires perfect 3D visualization of parts of the brain. The detection as well as 3D
visualization of parts of the brain and the tumours by the MRI method requires a lot of time and it is
also prone to errors. The system helps in detecting and presenting 3D reconstruction model of brain
and tumours. This in turn aids the radiologist to diagnose as well as analyse the brain tumours.
Segmentation and visualization technique overcome the complicacies in the 3D volume segmentation
methods such as some fine details missing. Three phase segmentation reduces the probability of the
errors in the process of segmentation. The method involved detects contours for brain, tumour and
skull. These contours are then stacked one over the other and two techniques are made use of to find
out the 3D visualization models. The results of the above said methods are impressive in nature.
Tumour refers to collection of the abnormal cells in any of the organs of the body. Brain tumour
detection is difficult and in order to detect the place of tumour knowledge and experience on the field
of radiology is required in case of medical imaging. MRI is the image diagnostic modality that is
widely used having the ability to provide characterization on a range of parameters in the living
subject providing exquisite spatial resolution. Segmentation is one of the essential but difficult steps
in the classification as well as analysis in the field of medical imaging. Artificial intelligence can be
made use of in identification of the biological features and process of segmentation. These methods
are of high value in the process of Medical Image Segmentation. Features of segmentation depend on
2

factors such as type of diseases and the features of the images. Recent advancements in techniques of
artificial intelligence such as machine learning, processing of image, recognition of pattern result in
considering the need of MIS to enhance the diagnosis. The informations that are desired for in case of
biological objects relates to basic features and thus is essential to apply image processing techniques
for the process of segmentation and visualization of the medical images.
Huo, Y., Xu, Z., Xiong, Y., Aboud, K., Parvathaneni, P., Bao, S., ... & Landman, B. A. (2019). 3d
whole brain segmentation using spatially localized atlas network tiles. NeuroImage, 194,
105-119.
Medical images have important role in the process of diagnosis and treatment of ailments.
Processing of such images increases the efficiency as well as effectiveness of processes. Segmentation
is considered to be as the initial step in case of medical image processing and this is made use of in
extracting Region of Interest (ROI) from a certain image. The author discusses about its effectiveness
and segmentation algorithm named Fuzzy C-Means (FCM) algorithm. FCM is time consuming in
case of processing the 3D models. This specific problem can be resolved by making use of parallel
programming by the use of Graphics Processing Unit. The article provides a hybrid parallel
implementation of the algorithm for the extraction of the volume objects form the medical files.
Algorithm proposed is effective in improving the performance five times as compared to the
sequential version. The article considers image segmentation that is the process involved in
partitioning digital image into certain set of different regions and this is said to be as a vital step in
many applications. Image segmentation is used which is referred to as early pre-processing steps
which can separate all the existing different areas that represent different tissues. This finds its
application in the extraction of Region of Interest that aids in providing directions to the attention of
the physician resulting in abnormalities in the tissues of the body for example tumours.
Figure 1: The proposed SLANT-27 whole brain segmentation method
3

Demiray, B., Rackerseder, J., Bozhinoski, S., & Navab, N. (2019). Weakly-Supervised
White and Grey Matter Segmentation in 3D Brain Ultrasound. arXiv preprint
arXiv:1904.05191.
The article compares 2D visualization with that of 3D visualization in case of image
processing application in medical. The adoption of the 3D models in the process of medical imaging
is becoming popular. The 2D Fuzzy algorithm is extensively being used in the process of
segmentation of the medical images owing to its effectiveness. There were many extensions that have
been proposed throughout the years. The article provides its readers with modified versions of the
algorithm for the process of segmenting the 3D medical volumes which in rare case are implemented
in 3D medical image segmentation. An alternative way of implementing this algorithm by making use
of GPU has also been included in the article. Efficiency is the main problem in using the FCM
algorithm for the process of medical imaging in case of the 3D models. A hybrid case of parallel
implementation of the algorithm is made use of in extracting objects form the medical files. This
specific algorithm is validated by making use of real medical data along with the phantom data
simulations. Accuracy in segmentation of the predefined datasets and the real patient datasets are the
primary factors for validating system. The time of processing of the sequential and parallel
implementations is analyzed illustrating the efficiency of each of the implementations.
Figure 2: DeepMedicUS with three pathways at different input resolutions
Mazurowski, M. A., Buda, M., Saha, A., & Bashir, M. R. (2019). Deep learning in radiology: An
overview of the concepts and a survey of the state of the art with focus on MRI. Journal
of Magnetic Resonance Imaging, 49(4), 939-954.
The article discusses on the usage of radiology in capturing the images of brain tumours. The
authors are of the opinion that clinical imaging is able to capture volumes of information but majority
of these radiological data get reported in qualitative terms. The use of radiomics in the field of neuro-
oncology helps in improvising the concept of biology and related treatment in brain tumours. This is
4

done by the extraction of the quantitative features from the clinical imaging arrays. These specific
data are in turn mined along with the methods of machine learning. The data then are validated in the
form of quantitative imaging biomarkers helping in characterizing the dynamics of intra-cranial
tumours in the course of the treatment. In recent times the image-based models of computation are
therefore becoming a vital technology that helps in identifying, analysing as well as validating the
quantitative features that have been extracted. The article discusses on the prevalent methodologies in
the field of radiomics that can possibly be used as the markers of prediction in the sphere of
therapeutic planning, diagnosis and prognosis in case of brain tumours in adults. The article also
discusses on the possible challenges emerging from data that are computationally generated by the
radiomic models.
Figure 3: A diagram of typical architecture of a convolutional neural network
Huang, R. Y., Bi, W. L., Griffith, B., Kaufmann, T. J., la Fougère, C., Schmidt, N. O., ... &
Nassiri, F. (2019). Imaging and diagnostic advances for intracranial
meningiomas. Neuro-oncology, 21(Supplement_1), i44-i61.
The article discusses on the imaging technology for intracranial meningiomas. It also considers the
advances that have been made in this field. The authors highlight the fact that in this particular sphere
that is ventriculography, imaging was done only when mass was doubted. The results of this imaging
used to more suggestive rather than being definitive. The article speaks about the journey that took
almost more than a century to reach the position where medical science can depend on imaging
technology to diagnose intracranial tumours such as menigioma. The use of radiology and 3D
imaging has been of great help in not just identifying the location of the tumours but also their types.
The modern techniques of radiology are continuously improving the power in the imaging
technology. Previous techniques could detect and monitor tumours but at present the techniques used
can extract biological informations from the parameters of radiology. The article explains
meningiomas describing it to be intracranial extra-axial masses that are attached to dura matter of the
brain.
5

Figure 4: (A) Noncontrast CT (B) T1-weighted gadolinium-enhanced MRI (C) T2-weighted
MRI of partially calcified right falcine meningioma
Kao, P. Y., Chen, J. W., & Manjunath, B. S. (2019). Improving 3D U-Net for Brain Tumor
Segmentation by Utilizing Lesion Prior. arXiv preprint arXiv:1907.00281.
The article discusses on the 3D MRI brain tumour segmentation by making use of auto
encoder regularization. The automatic process involved in segmentation of brain tumours from the 3D
magnetic resonance images is essential for diagnosing, treating as well as monitoring the disease.
Manually delineating requires good knowledge of anatomy and the process is expensive as well as
time consuming. Owing to its dependence on humans it can also give erroneous results. Here comes
the play of Artificial Intelligence in the segmentation process. The article refers to the segmentation
based on the 3D MRIs that are based on the architecture of encoder and decoder. The segmentation
process included in this article follows the encoder and decoder based CNN architecture. The encoder
extracts the features of the images and the decoder reconstructs the mark of segmentation.
Figure 5: The workflow diagram
6
Register to
MNI space of
1mm
Extracting the
binary masks of
necrosis
Extracting the
binary masks of
edema
Extracting the
binary masks of
enhancing tumour
Applying
element-wise
summation to
binary masks
Applying
element-wise
summation to
the binary
masks
Applying
element-wise
summation to
the binary
masks
Heat
map of
necro-
sis
Heat
Map of
edema
Heat
Map of
tumour

Mazurowski, M. A., Buda, M., Saha, A., & Bashir, M. R. (2019). Deep learning in radiology: An
overview of the concepts and a survey of the state of the art with focus on MRI. Journal
of Magnetic Resonance Imaging, 49(4), 939-954.
The article discusses about glioma that is a common type of intracranial tumour.
Segmentation needs to be accurate in case of the study of sub cortical brain structures. A large amount
of human labour is required in case of obtaining manual segmented Magnetic Resonance Imaging
(MRI) data that limits the use of quantitative data in the clinical practices. In this article, the authors
try to look into this particular problem and thereby consider an alternative that is 3D Convolutional
Neural Network on the basis of the automatically segmented gliomas. The difficulty in this process is
that in different patients the structure, location and shape of the gliomas differ. The article compares
the other 3D based models with that of 3D CNN. The authors introduce modern approach for the
segmentation of brain tumour in the MRI scans. DenseNet was developed for image classification
problem. The proposed model has been shown to have great potential for the process of MRI
segmentation or other such medical segmentation tasks. The article addresses the challenges that pop
up during the process of brain tumour segmentation by making use of the MRI scans. Class imbalance
problem is recognized as the major problem in this case.
Figure 6: An illustration depicting the difference between traditional and deep learning
machine learning
Sahiner, B., Pezeshk, A., Hadjiiski, L. M., Wang, X., Drukker, K., Cha, K. H., ... & Giger, M. L.
(2019). Deep learning in medical imaging and radiation therapy. Medical physics, 46(1),
e1-e36.
The article discusses on the segmentation process of brain tissues form the 3D medical images. This is
of great use for diagnosis of brain diseases. Segmentation also includes progression assessment as
well as monitoring of the neurological conditions. Manual segmentation is time consuming as well as
laborious. It provides subjective data while in case of automatic segmentation, there exists various
challenges such as complicated anatomical environment of brain along with the larger variations of
7

the tissues of brain. The authors address to this problem by proposing novel voxelwise residual
network that includes a set of training schemes. Manual segmentation of the structures of the brain
from the 3D images is laborious as well as time consuming and requires sophisticated and sound
knowledge in anatomy of brain. Manual segmentation is error-prone as it is low in reproducibility.
Thus automated segmentation methods are desirable as this provide consistent measurements along
with quantitative analysis.
Figure 7: CNN-extracted and conventional features: ways of combination
Prevedello, L. M., Halabi, S. S., Shih, G., Wu, C. C., Kohli, M. D., Chokshi, F. H., ... & Flanders,
A. E. (2019). Challenges related to artificial intelligence research in medical imaging and
the importance of image analysis competitions. Radiology: Artificial Intelligence, 1(1),
e180031.
The recent advancements along with the future perspectives of the machine learning
techniques provide some significant applications in the sphere of medical imaging. Machine learning
can make improvisations in the different steps of radiology. The article discusses on the facts of
radiological works such as scheduling of order and triage, detection and interpretation of findings of
data, clinical decision support systems, and quality control in the various examinations and
radiological reporting. In this particular article the authors highlight various applications of artificial
intelligence along with machine learning in the sphere of diagnostic radiology. The future impacts and
the natural extension of the techniques that related to radiological practices have also been discussed.
More and more availability of the imaging data from the machine learning techniques have been of
great significance in medical imaging. Medical imaging has progressed in post processing tasks for
example; segmentation, quantification and image registration. A medical imaging paradigm that is
intelligent is driven by data and medical images provide informations that aid in clinics. Extractions
8
Pretrained CNN
features
SVM Classifier 1
ROC analysis
and evaluation
Ensemble
Classifier
ROI image
Analytically
extracted
Features
SVM Classifier 2

of those data that are clinically relevant from the medical images require being accurate in case of
image registration and segmentation. Thus there are studies that make use of the techniques of
machine learning approaches for segmenting the medical images.
Continuous Output from
Deep Learning Algorithm
Cancer Benign Disease
0 0 5
0.1 2 103
0.2 6 90
0.3 5 21
0.4 5 8
0.5 8 5
0.6 15 8
0.7 20 5
0.8 25 4
0.9 11 1
1.0 3 0
Total 100 250
Table 1: Results from the hypothetical research study evaluating performance of hypothetical
deep learning algorithm in the differentiation of cancer.
S Tandel, G., Biswas, M., G Kakde, O., Tiwari, A., S Suri, H., Turk, M., ... & K Madhusudhan,
B. (2019). A review on a deep learning perspective in brain cancer
classification. Cancers, 11(1), 111.
In the article an overview has been provided in neurosurgical care. Background Machine
Learning is the branch of the artificial intelligence allowing computers to learn from the large
complex data sets without being programmed in an explicit way. In the day to day chores of medical
research and clinical care the technology is in its emerging stage. The complicacy in the diagnostic
and the therapeutic modalities that are made use of in neurosurgery generate volumes of data that in
ideal cases suits Machine Learning models. This article discusses on the systematic review which
effectively explores machine learning’s inherent potential in assisting and improving neurosurgical
care. Machine learning being a branch of artificial intelligence has the potential to enable computer
based algorithms to study and improve large databases without any such need of explicit
programming. This particular branch of artificial intelligence has become popular as here the
9

computers have the required computational power in building the complicated models that can help in
processing and learning from the unstructured data.
Figure 8: Work Flow Diagram of ML-based algorithms
Dos Santos, D. P., Giese, D., Brodehl, S., Chon, S. H., Staab, W., Kleinert, R., ... & Baeßler, B.
(2019). Medical students' attitude towards artificial intelligence: a multicentre
survey. European radiology, 29(4), 1640-1646.
The authors research on Artificial Intelligence demonstrates there are myriad applications found in the
field of medical images which pushed it forward. AI can automatically recognize complex methods
which provides quantitative rather than qualitative analysis and pattern matching however historically
trained doctors practiced radiology by visually assessing images that were medical. In the article the
author describes a general understanding of AI methodology, especially tasks that are image-based.
The authors have gracefully explored how these methods impacts multiple facets of radiology. There
is a general focus on the demonstration of these methods and advancements in applications on
oncology. The authors also describes about the challenges that are faced while clinical
implementation of radiology and AI merged together and perspectives on the advancements of these
domains. There are methodologies ranging from convolutional networks that are neural to auto
encoders that are vibrational. This research article also plots the performance level of Artificial
Intelligence as compared to a human intelligence starting from earliest computer science use and
breaking into the future.
10
Texture
Contrast
Classifiers
Brain MRI
Image Features
Performance
Parameters
Curvature Based
Shape Based
Extraction of
Feature
Classification Training

Figure 9: MRI brain images, pink circle- tumour region
Anaraki, A. K., Ayati, M., & Kazemi, F. (2019). Magnetic resonance imaging-based brain tumor
grades classification and grading via convolutional neural networks and genetic
algorithms. Biocybernetics and Biomedical Engineering, 39(1), 63-74.
The authors, researches objective of the paper is to show the tumor grade classifications and the
segmentation. The authors mention that the motive of the study is for the requirement of the early
detection of brain tumor and the grade the motivation. Physicians always need a quantification of the
area where the tumor has occurred, even after the tumor appears clear during the MRI (Magnetic
Resonance Testing). This area is where the technique like digital image processing is used along with
machine learning aids are used between the computer and the radiologist. This technique involves
diagnosis, prior and post-surgical procedures that synergies between the human intelligence and
machine intelligence. The diagnostic accuracy can be achieved by the hybrid methodology of
analyzing using both AI and human intelligence and provides great assistance to radiologists for
further understanding. In this article the author talks about the retrospect and the current occurrences
in human brain that is tumor infected MR images which includes astrocytoma. The methods that are
used to clinically examine the tumors and grading them can be integrated into protocols of standard
imaging techniques and elucidated. The state of art assessments and the future developments are
disserted along with the trends.
Figure 10: Three different grades of Gliomas axial brain images
11

Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J. S., ... & Davatzikos, C.
(2017). Advancing the cancer genome atlas glioma MRI collections with expert segmentation
labels and radiomic features. Scientific data, 4, 170117.
The authors, describe that Gliomas are a category of the central nervous system tumors and it
has various sub-regions. The radiographic imaging is the gold standard labeling which is used for the
purpose of computational studies as well as clinical studies. This process as the authors describe it
also includes radio genomic and radiomic analysis. Doe all the pre-operative multi-model magnetic
resonance imaging segmentation labels are released labels at the end (MRI) (n=243) is the multi-
institutional collections of glioma which is collected in the Cancer Genome Atlas (TCGA). There is a
publicly available (TCIA) The Cancer Imaging Technique. The authors also explain that the scans that
are pre-operative could be identified in both TCGA-GBM, n=135 (gliblastoma) and TCGA-LGG,
n=108 (low grade glioma) that are collected using an assessment that is radiological. The sub-regions
of glioma was put into picture and was manually looked upon by an expert certified neuroradiologist
in the automatic state-of-art. Functions related to radiomic features were manually extracted using
methods of revised-labels. There is a performance evaluation of the segmentation methods by the use
of computer-aided tools which effectively compare the state-of-the-art methodologies. The new
prognostic assessments including predictive as well as diagnostic assessments are done by the direct
utilization of the TCGA/TCIA glioma collections.
Figure 11: Automatic Brain Tumour Segmentation using CNNs (Cascaded Anisotropic)
12