Learning Local Descriptor for Comparing Renders with Real Images - IMT Mines Albi-Carmaux Access content directly
Journal Articles Applied Sciences Year : 2021

Learning Local Descriptor for Comparing Renders with Real Images

Pamir Ghimire
  • Function : Author
  • PersonId : 1095876
Igor Jovančević
Jean-José Orteu


We present a method to train a deep-network-based feature descriptor to calculate discriminative local descriptions from renders and corresponding real images with similar geometry. We are interested in using such descriptors for automatic industrial visual inspection whereby the inspection camera has been coarsely localized with respect to a relatively large mechanical assembly and presence of certain components needs to be checked compared to the reference computer-aided design model (CAD). We aim to perform the task by comparing the real inspection image with the render of textureless 3D CAD using the learned descriptors. The descriptor was trained to capture geometric features while staying invariant to image domain. Patch pairs for training the descriptor were extracted in a semisupervised manner from a small data set of 100 pairs of real images and corresponding renders that were manually finely registered starting from a relatively coarse localization of the inspection camera. Due to the small size of the training data set, the descriptor network was initialized with weights from classification training on ImageNet. A two-step training is proposed for addressing the problem of domain adaptation. The first, “bootstrapping”, is a classification training to obtain good initial weights for second training step, triplet-loss training, that provides weights for extracting the discriminative features comparable using l2 distance. The descriptor was tested for comparing renders and real images through two approaches: finding local correspondences between the images through nearest neighbor matching and transforming the images into Bag of Visual Words (BoVW) histograms. We observed that learning a robust cross-domain descriptor is feasible, even with a small data set, and such features might be of interest for CAD-based inspection of mechanical assemblies, and related applications such as tracking or finely registered augmented reality. To the best of our knowledge, this is the first work that reports learning local descriptors for comparing renders with real inspection images.
Fichier principal
Vignette du fichier
Learning-Local-Descriptor-for-Comparing-Renders-with-Real-Images.pdf (4.03 Mo) Télécharger le fichier
Origin : Publisher files allowed on an open archive

Dates and versions

hal-03197287 , version 1 (13-04-2021)





Pamir Ghimire, Igor Jovančević, Jean-José Orteu. Learning Local Descriptor for Comparing Renders with Real Images. Applied Sciences, 2021, 11 (8), pp.1-15/3301. ⟨10.3390/app11083301⟩. ⟨hal-03197287⟩
119 View
124 Download



Gmail Facebook X LinkedIn More