Abstract:

Information regarding what an object is made of – its material – can provide crucial clues for image understanding. If a robot, for instance, detects soft dirt or a smooth metal surface ahead, it can adjust its movement in advance.  Recognizing materials solely from images, however, has proven to be a difficult problem. In this talk, I will present our research geared towards visual material recognition. I will first discuss about a generative approach, in which we aim to decompose the image into its building blocks–geometry, illumination, and reflectance–so that we can later use the reflectance estimate to deduce the material. I will show how the space of real-world reflectance can be faithfully encoded with a novel reflectance model and be exploited to estimate reflectance in complex real-world environments. I will then discuss a discriminative approach in which we directly try to classify each pixel of an image into different materials. For this, we introduce a novel intermediate representation, called visual material traits, that represent the appearance of material properties like “smooth” and “shiny,” and use them to recognize materials locally without any knowledge of the object. Finally, I will show some preliminary results on using material as visual context for image understanding.