Millions of people throughout the globe suffer from pneumonia, a potentially fatal respiratory illness. Effective treatment and patient outcomes depend on a prompt and correct diagnosis of pneumonia. Diagnosing pneumonia is mostly dependent on medical imaging tests like X-rays and CT scans. To improve the diagnostic precision and productivity, this research introduces a unique multimodal and transfer learning model for deep convolutional neural network (CNN)-based pneumonia identification in medical pictures. Our proposed approach takes full use of multimodal data by integrating findings from many imaging modalities into a single framework for enhanced diagnostic accuracy. The detection accuracy is improved by combining X-ray and (Computed tomography) CT scan pictures, which provide complimentary information. In addition, we use transfer learning to take advantage of pre-trained models, enabling the network to pick up pertinent characteristics and patterns from massive datasets, which aids in the diagnosis of pneumonia in medical images.