Tracking monocular colonoscope in the GastroIntestinal (GI) tract is challenging as the obtained images suffer from deformation, blurred textures, and significant changes in appearance. These drawbacks greatly restrict the tracking ability of conventional geometry-based methods, which are heavily dependent on the performance of corner points extraction from the image. Even though end-to-end Deep Learning (DL) can overcome these issues, limited labeling data is a roadblock to the state-of-the-art DL-based method. To handle these drawbacks, we propose a novel approach to combine the DL-based method with the traditional geometry-based approach to achieve better localization with small training data. In this work, a DL network is trained with the images of the pre-operative endoscopy/colonoscopy. Siamese architecture is introduced to perform the zone labeling of the image based on the anatomical segmentation with expert knowledge. Then, using the image in the therapeutic intervention, our method predicts the 6 degrees of freedom scope pose and recover geometric reference to the images from the pre-operative endoscopy/colonoscopy. The DL network predicts the zone of the testing image, and the pre-generated triangulated map points within the zone in the training set are registered with the bundle adjustment algorithm. The proposed hybrid method is tested on the synthetic data sets and the real-world in-vivo data sets. Further, the results achieved through various experiments validate that the proposed method outperforms traditional geometry-based only or DL-based only localization techniques.