Image data, in order to be turned into information, require interpretation. Thereby image understanding is the process of scene reconstruction, the description and mental representation of the content of imaged, and potentially complex, realities.
Image understanding thereby goes beyond single feature extraction. Instead, it aims at a complete description of the image content, i.e. the reconstruction of a real-world scene. In the early days of digital image processing, image understanding was mainly confined to identifying and labelling image primitives. Today, advanced mapping keys and hierarchical classification schemes to analyse EO data, include composite and complex target classes. Thereby ‘full’ scene description means reaching from signal processing to a symbolic representation of the scene content. This entails the relationships of real‐world objects in different scales and spatio-temporal aspects.
Describing a scene, visually or computer-aided or mixed, depends on a conceptual framework comprising (a) the underlying research question within (b) a specific field of application and (c) pre‐existing knowledge and experience of the operator. Obtaining insights from imagery requires general knowledge about the expected scene content and domain expertise. The field of image understanding is interlinked with image (pre-)processing, computer vision, and artificial intelligence (AI). Image processing conditions the data material and enhances the interpretation source. Computer vision including pattern recognition providing knowledge representation, expert systems. AI is mainly concerned with automation processes, be it via knowledge transfer to an automated system or machine / deep learning.
In analogy to the human mind, image understanding is the computational process of extracting information from images, i.e. locating, characterizing, and recognizing objects and other features in the depicted scene. However, image understanding is not a linear, but rather a cyclic process and takes place during the pre-processing and data assimilation steps. For example, cloud masks on EO images is an early product of image understanding, prior to many pre-processing tasks.
In a typical GEOBIA workflow, the process of image understanding can be illustrated by the following steps: Starting from the subset of a real‐world scene captured on an image first step may entail scaled representations by grouping neighbouring pixels on several hierarchical sales. The multi‐scale segmentation provides a set of nested objects with geospatial and spectral properties to be used in the classification process.
With object hypotheses in mind the object relation modelling can be realized by encoding expert knowledge into a rule system. This setp aims at categorizing the image objects by their spectral and spatial properties and their mutual relationships. Hereby, an object‐centred view is accomplished. This representation of the image content should meet the conceptual reality of the interpreter or user. Knowledge is stepwise adapted and improved through progressive interpretation and modelling. Experience grows, as knowledge will be enriched by analyzing unknown scenes and the transfer of knowledge may incorporate or stimulate new rules.