As early as 1907, Sergey Mikhailovich Prokudin-Gorskii, a Russian chemist and photographer, was convinced that color photography would revolutionize the future. His color portrait of Leo Tolstoy (author of War and Peace and Anna Karenina, among other titles), a widely acclaimed piece of work at the time, became a golden ticket that granted him the privilege of presenting his work to Tsar Nicholas II 1909. The Tsar so enjoyed the presentation that Prokudin-Gorskii was granted funds to document the entirety of Russia in color.
Prokudin-Gorskii’s formula for color photography is simple. The photographer records three black-and-white exposures of the desired scene onto a glass plate using red, green, and blue filters. Aligning the three images should produce an RGB (color) image! (Prokudin-Gorskii envisioned accomplishing this with projectors.) With this in mind, he assembled a collection of over 10,000 photos over the course of 10 years (now known as the Prokudin-Gorskii Collection), traveling all over Russia.
Prokudin-Gorskii never lived to see his negatives transformed into color images due to the political climate of the period and events such as World War I and the Russian Revolution. In fact, a portion of the photos were confiscated, hidden, or given away upon his departure from the country in 1918. Outside of the Library of Congress collection, no other photos from Prokudin-Gorskii’s collection have been recovered.
Luckily, photos from the Prokudin-Gorskii Collection are now public, courtesy of the Library of Congress. This project aims to experience Prokudin-Gorskii’s grand vision by taking the images he left behind and using image processing techniques to put them together according to his intended procedure. In doing so, we can get a glimpse of the world as he saw it—in color—a century ago!
So, how does one begin to align three grayscale images? To create the color photos, I followed this general procedure:
read in the image
split image into thirds and call them b, g, and r, respectively
if aligning from pixel values:
call ncc_align
else if aligning from edges:
call sobel_align
combine adjusted r and g images and combine with b to form 3D matrix with color channels
save image
The ncc_align method can be described as follows:
if image is small, perform single-scale alignment:
search over a grid of possible x and y shifts
score each combination using the NCC algorithm
return the combination with the highest NCC score and the shifted image
else if the image is big, use an image pyramid:
if image is small enough:
return the results of single-scale alignment
else:
do image pyramid with: 1) reduced image, 2) increase search range
apply the x-y shift returned, scaled up, to current image
do single-scale alignment on current image
return shifted image and overall x-y shift values
The purpose of the Normalized Cross Correlation algorithm is to normalize images so that vary brightness or exposure plays less of a role in determining whether a certain x-y shift is a good match. We typically do this by subtracting each image by its mean value and dividing by its standard deviation.
But, this relatively straight-forward method is no match for large images. Such photos would take much too long to process. Luckily, we have the image pyramid. The image pyramid is a technique often used for processing large images. It represents the image at multiple scales (in this case, a factor of 2) and processing is done sequentially starting from the smallest (roughest) image and going down the pyramid, updating the shift recommendations as we go.
I should note that, in single-scale alignment, my algorithm ignores a specified percentage of the edge pixels in order to reduce the effect of messy borders. None of the Prokudin-Gorskii images line up exactly, so it is counterproductive to force the algorithm to attempt to make the edges match. (You can see evidence of this in the colorful borders of the images below.)
Simply applying NCC with single-scale alignment and the image pyramid seemed to do fairly well on most images except for the photo of Emir of Bukhara.
Left: The original B, G, R grayscale images (from top to bottom) taken by Prokudin-Gorskii. Right: An attempt at aligning the photo with NCC.
This is likely because the red, green, and blue grayscale images to be matched do not have the same brightness values. As such, using the raw pixels as a tool for alignment is not enough to produce a proper color image. Indeed, if we examine the grayscale images for Emir (the order is blue, green, red), it’s very clear that the image taken with the red filter is far brighter than the others. Further, the value of Emir’s clothing, in comparison to the rest of the background, changes with each image. In the blue picture, Emir’s clothing appears much lighter than his surroundings, while in the red image, his clothing appears to be darker.
It should also be noted that, while NCC as a technique, works well for images that may vary due to differences in lighting and exposure conditions (this is due to its normalization of the images involved), its performance degrades for low-contrast or noisy photos.
In order to fix the problem of Emir of Bukhara, I turned to edge detection. Edge detection is essentially using mathematical methods to identify points in a digital image where the brightness changes abruptly. The result is that we are able to identify where the various lines are in an image! In this case, I used the Sobel filter provided by the scikit-image filters library.
given grayscale images r, g, and b, run all of them through the sobel filter
perform NCC alignment (from Method 1) on the edge images:
apply the recommended x-y shifts to the original r and g grayscale images
The technique was a success. Ladies and gentlemen, I now present to you, in full color, Emir. Of. Bukharaaaaaaaa!
This has been an incredibly fun project and there are so many other things that could be attempted to better these images (white balancing, edge cropping, etc.) But, I’ll bow out now and leave you with some more sobel-aligned images from the Collection. Thanks for reading!
Source Acknowledgements: Header icon courtesy of Nikita Kozen. Bell icon courtesy of Stanislav Levin.