Overview

History of Images in Forensic Science

Images have always played an important role in forensic science and normally we refer to photos or digitally stored images. During the last decade, photography and digital imaging have increased in importance, as computers provide us with the tools to analyze images.
An image can be defined as a ‘visually recognizable pattern’. The human visual system has the ability to interpret the pattern presented on the retina and we use what we have learned for recognizing what we see (Fig. 1). We are rather good at analyzing images and now we want computers to be at least as good as we are. That is not an easy task. The key components we use for dealing with visual informations are image acquisition, preprocessing, processing, analyzing, interpretation and visualization of the analysis result, storage, transmission and database search.
In essence every contact leaves a trace, which is one of the fundamental principles of forensic science. Images that have been directly captured at a crime scene can be used as evidence. In this case the trace consists of the radiated and reflected photons from the objects at the scene. These photons are passed through the optics, captured onto the sensor surface, quantified into an electric signal processed through the electronic devices, to give a signal that can be displayed or converted into a magnetic track on a videotape (Fig. 2).

Figure 1 A poor image of a crocodile enhanced from a negative, in which we usually recognize its typical textured skin.

Figure 2 Images digitized from a surveillance system recording.
Images are widely used in other forensic applications, for instance in the examination of firearms, toolmarks, shoeprints, fingerprints and questioned documents. However this can be considered as indirectly captured traces from the crime scene. There are many specialized imaging techniques and analytical methods for the purpose, referringto their respective forensic applications.
Initially, sketches were used for illustratinga crime scene. Conventional photographic techniques were introduced in the middle of the nineteenth century and have been used ever since. Although digital imag-inggives the impression of beinga relatively new technique, it actually dates back more than 30 years and has passed through several phases and driving forces. One of the driving forces was the introduction of digital images as a new tool. During World War II ‘electronic and digital calculating machines’ appeared, and this started the era of computers. One great advantage was that these machines could be used for large two-dimensional data structures, such as images. Another driving force was the expansion of the scientific fields. New solutions led to breakthroughs and formed new scientific areas. Optical character recognition, blood cell counting and target recognition in aerial photographs are some examples.
The digital images were initially generated from other media, but duringthe 1970s the first digital satellite image was produced. It was quite a complex task to view, store and handle these large amounts of data. In the mid 1980s there was a breakthrough for both research and applications, which provided systems for processingand analyzingdigital images and image sequences. The success was followed by a backlash, caused by unrealistic optimism and by underestimation of the complexity of the human visual system.
Then the use of image processing systems at forensic laboratories began, which provided methods to deal with surveillance photos and videos. The main purpose was to acquire and handle images and to provide printouts. Image enhancement methods were also introduced.
In the late 1980s, tools for image analysis and image-based measurement became available; these were refined duringthe 1990s to become more precise, complex and user-friendly. Today, any computer can be equipped with hard- and software to deal with digital images at a reasonable price and computer processingtime. The research has also reached a more mature status, partly by merging several scientific fields, as image processing and computer graphics. In the late 1990s software specialized for forensic application became available. This overview of methods and applications only mirrors the state-of-the-art as new tools and techniques will continuously be introduced.

Image Acquisition Techniques

Imaging refers to the process of capturing data by imaging devices. Images can be analog, digital or hybrid. Examples of analog images are photographs, optical images and holograms. The most common TV and video standards are examples of the combination of hybrid techniques, as there are a discrete number of TV-lines but each TV-line is an analog representation of the scene.
The imaging techniques have varied over the years from being artistic, photographic and magnetic to electronic. It is therefore no surprise that evidential images come in a variety of formats.
Photographs can be considered as one important class, and digital or digitized images another.
In all image acquisition stages information is more or less lost or distorted and artifacts might be introduced.

Photographic techniques

Conventional photographic techniques are important for documentation and for developing and securing traces, by providing high quality photos. Light composition is important in the imaging stage, and by varying the intensity and direction of the light the trace can be enhanced. Photographic techniques also have the capability to extend the observations outside the visual range, by varying the wavelength of the lighting source from UV and IR regions and transform the trace into the visual range. In contrast to many digital images, the high resolution provides the ability to enlarge interesting regions from positives and negatives. Photos can be developed from the negatives with varying techniques, for example enhanced contrast, intensity or color mapping for better visibility.
Another possibility is to electrically charge the trace, such as for dust particles to visualize shoeprints.

Digital imaging techniques

A digital image consists of a discrete number of picture elements, so called pixels. Each pixel carries information from the depicted scene, digital representation of intensity and usually also color. The pixels are ordered in a rectangular grid in the image, normally referred to as rows and lines. Digital images can be acquired by scanning photos or negatives, transmitted from digital still cameras, digitized video, data stream from digital camera and video or copied data files. There are several ways to achieve and distribute data files, such as downloading from the Internet or by using other telecommunication drivers.
Most imaging techniques result in high quality images, but to analyze evidential images often means dealing with low quality images and unknown imaging devices. It is therefore essential to be aware of typical device characteristics. This can be quite a task due to the large number of ways of achieving a digital image and the variety of imaging devices, copying techniques and telecommunication channels. Some major problem areas will be discussed as the effects of poor imaging can fruitfully be used in the investigation process.
Sharpness and resolution mainly characterize the quality of an acquired image. Useful terms are, the transfer functions of the optics and imaging systems, respectively (MTF) and (OTF), used to measure how a known input signal is affected by the components of the imaging system. Another measurement is the point spread function (PSF), which provides an alternative measure of the optical quality. Other functions used are the contrast sensitive function, and the signal to noise ratio (SNR). For evidential images these parameters are mostly not controllable and they therefore remain unknown. At present there are no well-established methods for quantifying the image quality acquired from unknown systems.
The visual information is affected at several levels, such as by photon effects, blur caused by motion, lens distortion, thermal noise, jittering in video signals or distorted image ratio induced by frame grabbers. Shot noise and thermal noises are normally not the main problems, provided the recording environment is normal. When charge coupled device sensors (CCD-sensors) are used, high temperature or low illumination seriously affect the image quality.
Low spatial resolution is considered the main problem compared to the quality of photos. In some way this can be compensated for by the large amount of images generated in a video recording. Digital still cameras continuously offer improved resolution, and additional data, such as date, time of exposure and type of camera. Comments might be stored as text and in complementary audio files. Presently there is a wide range of image formats and coding schemes used for digital still cameras and digital video, some are compatible and some are not. Comparing the image size and color depth to the true size of the image file might indicate if unknown compression and restoration techniques have been used. There is also the possibility that enhancement functions available in many cameras and videos have been used.
Image processing methods used for corrections of distortions and artifacts require a great deal of caution and knowledge. When images are used for identification of persons or objects by superimposing two digital images, it is essential to investigate and if necessary correct for distortion before the analysis. Otherwise the shapes and contours might remain affected, which actually are the features used for identification. Correction techniques are used to compensate for distortions caused by the optics, such as removal of radial lens distortion. Distortion of the image aspect ratio from a digitizing board is another error that can be corrected quite easily.
Videotapes from surveillance systems are perhaps the most common source of evidential images. The quality of surveillance system recordings indicates the problems and weaknesses of recording philosophy. The images often suffer from low resolution, poor lighting conditions, blur caused by optics and motion, and physical stress on videotapes. These are currently the main problems rather than the lack of scientific analytical methods. Distorted or lost synchronization pulses cause further problems. For this purpose, time-based correctors or similar devices stabilizing the synchronization pulses are valuable. To avoid further degradation of the videotape, the first step of the analysis is normally to digitize the sequences of interest. This is important in order to keep up the quality and thereby retain the fine details.
There is a similar chain of stages from digital images to printed paper copies affecting the image information. This will not be discussed further here, but it might be important to bear in mind for the preparation of material for court presentations. This particularly holds for the processing of color and the effects of raster and printer techniques.

Methods and Applications

Analytical methods are used to extract the information that is not directly obvious to the eye. For this purpose additional information is derived from the images, such as numerical values or processed images.
The processed images or image features are used to associate an image directly captured at the crime scene, to the corresponding features of a suspect. The interpretation of the analysis result is evaluated to finally determine the association of a suspect with a crime scene. Science provides powerful tools, bearing in mind that complex solutions are complex to use and require cautious implementation, validation and interpretation of the result.
Images can be processed and analyzed either by biological systems, e.g. by the human visual system, or by technical systems. Technical image processing can be divided into analog and digital methods, which can further be either linear or nonlinear (Table 1). Enhancement of photos by photographic techniques is an example of analog and linear processing. Digital images in combination with science enable the use of sophisticated nonlinear processing and analytical techniques. Mathematics is the basic platform, with fundamentals as algebra, calculus, statistics, transformation and optimization theory. Other relevant scientific areas are briefly explained in a following section.
The dominant casework methods are image enhancement techniques and person or object identification (Table 2). Examples where image information is valuable are identification of persons involved in robberies, fraud or any other crime captured on surveillance systems, identification of pedophiles, distribution of child pornography on photo, video, or by telecommunication channels such as the Internet. There are also other useful features of the evidence to be found in the storage medium. The processing of audio signals is closely related to that of visual images. Apart from images and audio there might be electrical signals, magnetic tracks, physical and chemical features or data in text files and file headers. These features give information valuable for tracing imaging devices as well as editing, copying, production and distribution order. Images are also important to illustrate and document the analytical methods and procedures. Court presentation is one such area of importance.

Authenticity

By investigating the authenticity the aim is to determine whether the information of an image medium has been technically altered. There are many software packages and video-editing systems for commercial use that provide functions for editing images and image sequences. Methods used for copyright protection can be used to examine image authenticity. Imaging devices might also use coding schemes in the acquisition process, such as JPEG (Fig. 3), which are useful in the investigation process. The artifacts and distortions vary with optics, imaging device, format, coding technique and noise patterns. The altering of image context often causes irregularities in distortion patterns, and these abnormalities are used to determine authenticity (Fig. 4).

Table 1 Image processing and analytical methods

Biological systems	Our human visual system (as one example)	Highly complex processing of visual information
Technical systems	Analogue processing	Linear methods l
		Non-linear methods ) low complexity
	Digital processing	Linear methods
		Non-linear methods ) low to high complexity processing

Table 2 Image analysis methods

Imaging	Recording devices
	Distribution of image
Image authentication	Technical alteration and manipulation
Image processing	Enhancement
	Enlargement
	Restoration
	Processing with multiframes
	Correction of moving or defocused objects
Person identification	Identification from facial images
	Craniofacial identification and facial reconstruction
Object recognition	Identification from images
Image measurement	2D measurements, 3D reconstructions
	Calibrated or uncalibrated cameras
Technical investigations	Image information
	Physical features
	Electrical signals
	Magnetic fields
	Reconstruction of damaged image media
Other methods	Image interpretation
	Course of event, simulations
	Image transmission and database search

Figure 3 Artifacts from image compression caused by a JPEG coding scheme in the right image.
There is a lack of automatic methods for analyzing authenticity, which today is a mostly manual task. Other features than the image can also be used, further described in the section of technical investigation of image storage media and recording devices. To determine if a certain camera has been used to expose certain negative slides, the shape and size of the bar between the exposures on the negatives can be analyzed (Fig. 5), as well as the film transfer mechanism and the ‘toolmarks’ induced onto the negative. Recently methods for identification of digital still cameras have been presented.

Image processing

The main purpose of image processing is to produce images that are easier to interpret, visually or technically. Enhancement methods dominate and are used to extract additional information and improve the visual quality to better suit the eye. These methods can be used to deal with motion distortion or defocused objects. Automatic processing of image sequences, enlargement, restoration of lost data files or damaged image media are other methods. Enlargement of images from video is not as easy as for photographs (Fig. 6), due to low spatial resolution. The use of neighboring pixels and also neighboring images in a sequence has proved to be successful for enlargement as well as for enhancement, referred to as super-resolution, mosaicing and frame fusing. Data can also be recovered from erased data files or damaged image medium, but recovery of polluted or damaged videotapes is a time-consuming manual work.

Person identification

The identification of persons by anthropometry is not a new science. The anthropometric method of Alphonse Bertillion is well known, but nowadays computer-aided methods have led to a breakthrough in this area, and have extended the analysis not only to the anthropometrical measurements and features, e.g. iris pattern, face, ears, fingerprints, hands, feet, or the posture, but also to the behavior of the human body in motion. Facial identification by image super-imposition, three-dimensional (3D) measurements and comparison of a 3D-reconstruction of faces or skulls to facial images is the most frequently used image-based method (Table 3).
Identification from facial features can be based on visual and metric comparison, computer-based 2D or 3D comparison and alignment or by automatic pattern recognition methods. When identifying a person from facial images it is essential to be aware of the functionality of the human visual system, as there are many ways to fool our brain, which for example has the capability to fill in uncompleted segments. Automatic pattern recognition is currently less useful for forensic purposes as it requires a highly controllable imaging phase, which is not the case for acquiring evidential images. Masks, glasses, beards, aging (Fig. 7), and variations in weight cause problems for both manual and automatic methods.

Figure 4 B, an altered image of A.ln C, a nonlinear mapping of the grayscale has enhanced the abnormalities caused by the alteration.

Figure 5 A, B Variations in the spacing between two exposures on a negative slide.
Image analysis is further used for face reconstruction techniques.
Many commercially available systems use biometrics data to verify a claimed identity. For instance, coded iris data stored at smart cards can be used to authenticate a cardholder’s identity. These methods are not fully applicable for the same reasons as for automatic face recognition.

Figure 6 A less successful enlargement of an image from a surveillance video by bilinear interpolation due to low spatial resolution of surveillance system.

Object recognition

Object recognition defines the process of detecting and identifying objects to determine if a physical item is identical to an object in an image. For this purpose visual methods, superimposing digitized images or metrology-based methods are used. As for 3D facial identification, creating a 3D volume of the physical exhibit can simplify the comparison, although it is not commonly used at this time.
Investigations often concern vehicles from videotapes. The shape, texture and color are used to determine class similarities and individual features such as dents and scrape marks, are used for identification. Methods for automatic interpretation of license plates are commercially available, based on image processing and pattern recognition methods. This requires high quality images. Otherwise, image-processing techniques can be applied to enhance the visibility of the characters, though not always successfully, mainly because at low spatial resolution.

Table 3 Methods for person identification

Visual comparison
Comparison using microscopes
Metric comparison, manual or semiautomatic measurements of	Absolute measurements
the facial features and geometrical distances	Relative measurements
Computer-based methods for 2D facial images	Perspective alignment
	2D superimposition techniques
Automatic face recognition algorithms	Statistical methods
	Learning based systems
Computer-based methods for 2D and 3D facial image comparison
Computer-based methods for entirely 3D facial images

Image measurements

Image measurements can provide information about the course of events and the size of items and persons in a crime scene. The height of a person might for example add information useful in an investigation. Measurement techniques are based on photogram-metry and 3D-reconstruction and are successfully used to extract data from 2D images. 3D-reconstruction techniques are also used for crime scene documentation and analysis. The choice of methods depends on the parameters available; fixed or mobile camera position; fixed optics or zoom; known or unknown place of photography/video recording; if there is a single image, sets or sequences and if there are one or several camera views.
As an image is only a 2D representation of a real scene, additional information is needed for measuring in the scene. One way is to use calibrated cameras and metric values to provide the data needed, but calibration requires a visit to the crime scene. Known item sizes in combination with geometric image correction techniques can be used in the case of noncalibrated cameras.
Reverse projection is a useful technique for single camera systems that require fixed cameras and optics, which surveillance systems normally provide. An additional recording is required where the same recording system and camera position is used, for scaling the measured values into metric values. This also takes into consideration any lens distortion. By applying this method to an image sequence, the uncertainty of the measured height due to various body postures can be reduced (Fig. 8).
The 3D-reconstruction technique usingcalibrated cameras provides rather accurate measurements, if the place of the crime can be located. This technique is based on the use of more than one camera view and a calibrated camera to obtain complementary data. Metric values are also needed for scaling. In practice, the scene is reconstructed by calculating and comparing the distances from objects in the scene, the image planes and the camera positions. In the case of variable focal length or unknown camera position this method is preferred.

Figure 7 Variations of a person’s face overtime. A, June 1999; B, 1975; C, 1986; D, 1994; E, 1995.

Figure 8 Measuring the height of a robber by reverse projection from a digitized surveillance video recording. The height was calculated from an image sequence and the path of the robbers movement in the bank have been tracked.
Some methods allow for the use of only noncalibrated cameras. In this case there must be well-known item sizes in the scene for scaling of the measurements. Additional correction techniques are often used in combination.
The difference between measuring simple rigid objects and the complex measuring of the nonrigid human body is well worth noting as accuracy strongly depends on the person’s position and dynamic behavior. Shoes, clothes and masks further affect measured values.

Technical investigation of image storage media and recording devices

In many cases there is a need for tracing the image source to determine a suspect’s connection to a crime scene. Methods for identifying recording devices, to determine copying, recording and editing order of a video recordingand to determine the time and place of production of the videotape might clarify such connections to evidential images. The recording devices leave traces in the image storage medium why storage medium itself provides a set of useful features. These methods can for example be used to find out whether a specific video camera has been used to record child pornography videotapes or in order to trace technical alteration.
For this purpose the image information can be analyzed as well, to determine if and how the image content or image sequences have been altered. Deviations and variations in video signals, irregularities on charged coupled device (CPP) sensor surfaces, variations and noise patterns caused by the recording mechanism in a video camera can give some clues. For digital devices there is sometimes additional information in image file headers, data files, or time codes supplied by cameras. The audio signals can be used to trace the type of video recording device, for example audio format and number of audio channels.
From the physical features of the videocassette information such as manufacturing, time of production and place of production can be determined by using the printed information on the cassette houseing, box and labels. The same holds for CDs, although there might be more information in the data blocks assigned to manufacturing information. The magnetic tracks can be used to determine the authenticity of a video recording or to determine copying, recording and editing order in a similar way as for magnetic audiotapes.
Electrical signals provide further information about the recording devices, and methods for trouble shooting television and video are useful. A commonly used method is to analyze the head switch position (Fig. 9), which can be used to identify recording devices and to determine the copying order. Other signals are the dihedral error, that is the position of the video-head on the drum of the video tape recorder, the position of the CTL-pulse versus the video signal, the RF-envelope and the FM-carrier of the luminance signal. These signals can be measured for videotapes and compared to the corresponding measures from a questioned video tape recorder.
Characters inserted in the video sequence can also be useful information. The characters differ in functionality, size, shape and position among cameras, which holds for conventional and digital still cameras as well as for conventional and digital video cameras.

Related techniques

Images in large databases, e.g. a digital reference library, as well as in combined databases are today essential for rapid and automatic search. Such databases can contain reference libraries of child pornography. The automatic search engines are based on features defined by the image texture, color and object size for example. The algorithms are often statistical or learning-based and might also provide rapid on-line search.

Figure 9 A horizontally and vertically delayed picture where the head switch position can be seen.
Image transfer has become an important area, as time in many cases is a critical parameter. Telecommunications for information exchange, such as image transmission to a remote database for on-line comparison of the images in the database, is often needed.

Scientific Areas

The pixels in a digital image are not randomly ordered as are the picture elements in a photograph (Fig. 10). Each pixel is addressed to a certain color and position in the image, and to time in an image sequence. The number of pixels in the image related to the true size of the object is referred to as image resolution. Pixels can further be used to define clusters of pixels, where contours, shapes, colors, texture and pixel motion are used to define how the pixels are related. These pixel clusters further provide for analysis of the various items in the depicted scene by semantic analysis. Adding the time, not only analysis of single images but also image sequences, which extend the dimensionality. These are the basic low level features from which the scientific areas have evolved into higher level analysis.
Image processing is normally referred to as enhancing poor images by creating images that are easier to interpret. Lately semantic analysis has expanded image processing into new exciting areas.
Image analysis refers to the science of extracting information from images, although values other than images are the output. The aim can also be to reduce the large amount of data for computational reasons. The output can be numerical, for example the object positions in an image. The analysis can be on a low level where each pixel is treated separately, or on higher levels, semantic or context-based image analysis.

Figure 10 Picture elements in a photo versus pixels in a digital image.
Computer vision can mainly be seen as the science of modeling imaging systems for processing and analysis. Computer vision differs from image processing and analysis in the way that the goal is to analyze the scene content, rather than just the 2D-image information.
The basic goal of Image coding is to deal with efficient ways to represent images for transmission and storage purposes involving data compression. Image compression techniques are based on the fact that our eye is rather insensitive to minor changes of geometry, contrast and resolution in an image. As this information can be considered redundant, it can also be reduced without seriously affecting the visual quality of an image. Examples of image compression techniques are the various JPEG and MPEG standards and methods based on wavelet or fractal coding schemes.
Computer graphics are often referred to as the process of generating images. These methods can successfully be used for image alteration in combination with image processing. Computer graphics are used for simulation, illustration, reconstruction and documentation.
Photogrammetry is the process of measuring from images that provides techniques for measuring a 3D scene from 2D images. This field can be combined with computer vision, image processing and analysis for reconstruction of a scene and the objects in the scene.
There are two major subgroups of automatic image interpretation methods: statistical and learning-based techniques. Statistics can be defined as the identification of structure in an image, and the handling of uncertainty. Statistical interpretation often requires a priori knowledge, such as face geometry in the case of automatic identification of an individual from facial images. Learning-based systems require sets of images for training purpose. Methods, such as artificial neural network and fuzzy logic, fall into this category. Common to all automatic recognition methods is that the variations between the classes or individuals to be identified must be larger than variations within the class. The result strongly depends on the choice of representation model.
Pattern recognition can be used to detect, recognize and identify objects, automatically based on the mentioned interpretation methods. Methods used in biometrics, for example, aim to fully automate interpretation of image features such as faces, iris patterns and fingerprints.
A common subject for discussion is ‘Who tells the truth, the eye or the technology?’ This question is especially relevant for forensic image analysis. The human visual system is enormously complex, and superior to any computer vision system, for example in dealing with motion and dynamic changes. In spite of this, there are areas where the computer can be used as a complement to the eye: to enhance the visibility (Fig. 11), and provide numerical values. There are also ways to fool the brain. This makes perception of visual information an important field for dealing with evidential images, even if the definition of neuroscience is still in the making. The study of the brain will most likely pinpoint and provide solutions to the problems with visual interpretation of images, focused on the areas of visual systems, coupled to behavioral and cognitive neuroscience.
The process of exploring, transforming and view-ingdata as images, referred to as visualization, can be used to illustrate the result of image processing or to plot numerical values from an analysis. Image storage, image transmission, database search engines, data fusion, sensor fusion and signal processing are other related areas of importance, as well as physics, optics, electronics and computer science.

Conclusion

Dealing with visual evidence requires methods for extracting all possible information, as the image quality is frequently very low. Evidence from other methods, such as DNA or fingerprints, often yield a better result. In some cases the evidence consists of only a worn out, noisy video recording, and the outcome of forensic image analytical methods might then add clues to the identification process. These methods can also be used for cases that could not have been solved by traditional methods or would require considerable time and resources.
Videotapes from surveillance system recordings are the dominate image source and, no doubt, the major problem source. Currently, the best way of achieving image enhancement and restoration is to improve surveillance system technology. Even if the enhancement techniques are continuously developed, the information lost at the imaging stage is impossible to fully recover.
No doubt images will remain important evidence, although the storage medium will probably alter by bringing images closer to the more general concept of ‘digital evidence’. It is obvious that there is further need for highly specialized techniques for image analysis that can help in the daily work in forensic laboratories.

Figure 11 Variations of resolution and contrast.