I created this two-part series which takes documentary filmmakers step by step through my entire documentary finishing workflow. I attempt to demystify the enigmatic process of conforming your picture-locked timeline for a finishing company or colorist. I also discuss my round-trip finishing workflow when working with a restoration artist.
You can download the free Documentary Finishing Workflow diagram here:
Abstract: How effective are state-of-the-art AI-based colorization models?
Results: AI-based colorization models are fast becoming successful and it’s only a matter of time before we have models which provide realistic and consistent results. (the DeepRemaster model even colored the pupils of people’s eyes!)
Definition: Although the model I’ve used is called DeepRemaster, this footage has not been “remastered”. Colorization is a “revision” of the media and does not reflect the creative intent of the filmmaker. I’ve only run a light dust busting filter and deblocking algorithm over the film prior to input.
Disclaimer: It’s important to note that flesh tone generated by a computer is treading within a sensitive cultural realm. The “tones and hues” represented in this test are not meant to accurately represent flesh tone, but rather, provide a generalized colorization. As I point out in further detail in my results below, there is much work to do and I’m hoping that real-world testing like this will provide useful feedback for the humans that train the machines which then create the models.
I’ve been both curious and skeptical of the automated colorizing models available for video for the past four or five years. Colorized and frame interpolated archival enhancement videos have become viral hits on YouTube garnering tens of millions of views. This is fascinating considering the films were shot over a century ago (STILL GOT IT!!!) The topic has become a popular sub-genre ever since prosumer software like Topaz Video Enhance and Gigapixel AI were released a few years ago. All of a sudden, everyone started using AI to “upscale” their archival media not really understanding just how badly these tools were decimating the original detail of the archival. Admittedly, I was initially fascinated myself but after using it a few times, I realized it was throwing out a ton of detail and making people look like inter dimensional creatures made of plastic. Strangely, most of our audience these days don’t seem to notice the artifacts inherent with these processes. Peter Jackson’s recent series The Beatles: Get Back gave us a glimpse at just how much detail film studios are willing to throw out so long as AI tools give archival the appearance of looking “better”. HBO’s recent documentary George Carlin’s American Dream is a good example of just how far filmmakers can push the uncanny valley. But I digress…
Colorizing models like DeOldify do seem to make an effort to paint flesh tone on faces and other exposed body parts to a reasonable degree, but the rest of the image elements take on a random assortment of muted hues. Ironically, the models impart the hand painted characteristics of colorized photos of yore. At first glance these AI models appear to colorize the image, however, upon closer inspection this assumption falls apart. In this article, I’ll show you an example of some tests that I’ve done recently using an archival film called The Cry of Jazz, which I’ve sourced from the Prelinger Archive. I’ve used this particular film for testing because it has a range of skin tones, it’s in the public domain, and the source quality is relatively decent. It’s certainly not a high quality scan of the original print but it hasn’t been encoded into oblivion.
If you don’t have a powerful Linux or Windows machine to install and run virtual environments for models like DeOldify, you can simply login to your Google account and run a Colab session for free. Google Colab is a cloud-based computing platform available for public use which provides free access to powerful GPUs. Once you’ve logged into your google account, you can visit the DeOldify Colab and begin testing.
You’ll need to export a frame from your source archival video as a JPEG or PNG, then colorize it using the DeOldify model. You can run the model locally on your machine or you can run via Colab. These frames will be important for the video colorization process later. For this test, I exported JPEG images that were taken from the first frame of each video clip I want to colorize.
*Note that the model colorized her pupils to a beautiful chestnut hue! Wow. Very cool detail. (though we have no idea what her eye color was in real life)
*Note that the model colorized the hardwood floor the color of wood. Very cool. But the model failed to colorize the woman’s right arm and hand. The man’s white shirt is perfectly clean with a neutral white point. Even the painting tacked to the wall has been given a bit of an artful colorization.
Once you have colorized your frame grabs, you can now use them as reference images for the corresponding video clips you’d like to colorize. You’ll need to upload them along with your archival video clip before you can run the DeepRemaster model in Colab.
DeepRemaster will now reference these colorized stills and apply a matching colorization to the corresponding video clip.
Needless to say, the initial results are stunning. We’ve certainly come a long way in the fast five years with this technology, however, DeepRemaster is a significant step forward. There are a few issues which seem to keep coming up and I’ve listed them below. I have not done any sort of quantitative testing to validate these observations.
RESTORATION
One issue I currently have with DeepRemaster is that the model also seems to be removing what it believes are unwanted artifacts. I would prefer to use my own methods to restore the image prior to colorization. I am, however, still learning to use the parameters of the model so this is something I might be able to address.
FLESH TONE
Darker skin tones don’t seem to be rendering as well as lighter skin tones. I’m not implying that this artifact is the result of the model not being trained with enough darker skin tones. This might be an issue with the model’s ability to segment and effectively colorize areas of the image with lower luminance values. Perhaps there is some sort of sweet spot for flesh tone exposure which I’m not taking advantage of. I have not color graded these source images before or after the fact. I’ve only cleaned up the dust and scratches a bit before colorizing. This might also simply be due to the specific nature of this test material. I need to run other media through the model to see if this is consistent.
MAGENTA SHADOWS
As I mention above, DeepRemaster has trouble rendering realistic shadows (not always but often). This becomes particularly problematic as the darker image areas have a magenta wash to them. You can see in the tests that the darker areas shift toward the line of purples and because of this, there is a global magenta feel to the results.
INCONSISTENT IMAGE SEGMENTATION
The model often does not recognize things like hands and legs when they take on abstract angles/perspectives. You’ll notice that somethings people’s faces and limbs drift in and out of color. I would not say that this is due to compression artifacts or large macroblocking as I did deblock and clean other compression artifacts out of the source image before testing. This is certainly one of the major issues with the current model since if there’s ever to be massive industry adoption for this technology, consistency will be paramount. The good news is that this is a popular field of research and the models keep improving. It will only be a matter of time before we get there.
COMPRESSED HIGHLIGHTS
You’ll notice that the highlights seem to be a bit hotter in the colorized material than in the source image. To be fair, this may be my fault. I need to play around with various input rendering transforms to see what it prefers. Was the model trained on archival? If so, was this archival’s color primaries and gamma kept at Rec.601 or was it transformed to Rec.709/Bt.1886? I need to read the documentation to find out a bit more. But I would prefer the model not make any gamma changes to my image if I’m going to add this methodology into my pipeline.
Take a peak under the hood of my process to restore a heavily compressed archival film using the open source Python script-based image processing tool Vapoursynth.
Read More
I convinced ten cinematographers to bring their cinema cameras to a qualitative camera test. Here are the results.
Read More
For the love of God, please stop using shuttle drives. Just use the Resolve media manager to shrink ray your Premiere timeline down to a tiny portable file and send to your colorist!!! In this article, I show you how!
Read More
My ongoing experiments with incorporating the Python-based image processing software, VapourSynth, into my color management and restoration pipeline has resulted in compiling a stable package which Windows users may use for further experimentation.
Read More
MCM Creative in NYC reached out to Magic Lantern and asked if it was possible to clean up and upscale his feature doc Shelter Island prior to an Amazon streaming distribution sale. Unfortunately, the original media was no longer available and the master had already gone through various unknown encodes.
Read More
Typically, filmmakers use the deinterlacing tool provided for free within their NLE or Davinci Resolve. This article will demonstrate why this approach can be destructive to the image and will offer a better alternative.
Read More
A collection of photos taken with a cell phone at full magnification yields interesting results. Although the discrete cosine transform compression still has a significant effect in the image, by magnifying the image in camera prior to the encode, these images take on a hybrid of pixilation and compression decimation. There is also a slight halation ghosting around the higher contrast edges which get encoded into a pleasant glow.
Read More
Colors, generally speaking, are instinctively associated with emotions. Working as a colorist and having to manipulate color everyday – I feel that most directors and cinematographers intuitively ask for specific colors to enhance specific emotions on special scenes without even realizing it. Here is a little help on how to actually use color in film making to activate – or not – a physical reaction in your audience.
Read More
Depending on your budget, you may or may not be intimidated by the prospect of hiring a colorist for your film. Fear not! Many post finishing houses can be flexible with you and can often find a solution to fit your needs. Finishing labs will often have a range of creative professionals with varying experience levels.
Read More
Eric Jenkins - Documentary filmmaker
I’ve been shooting and color grading Sony cameras for the last fifteen years and I can honestly say that the FX9 is a game changer for the company. By oversampling a 4k image using a 6k sensor, Sony is achieving multiple things at once. Due to the nature of a Bayer sensor, most 4k cameras on the market aren’t true 4k. Each RGB pixel on a sensor is spaced in such a way that the term “pixel resolution” is misleading.
Read More
In the mid-twentieth century as color film was quickly becoming the preference for middle-class white citizens, Kodak began to hire studio models for their product brochures. One of the first women to model in Kodak’s testing studio was named Shirley Page. She somehow set the standard for Kodak’s concept of beauty, and with that a new tradition was born.
Read More
OZY - Creators - Colorist: Zachary Halberd
The first thing I look for when browsing the raw media coming from a client prior to a color grade is the quality of the skin tones. Skin tones will tell you most of what you need to know when assessing the technical quality of a digital negative. Exposure, white balance, tint, mid-tones, saturation, noise level, black saturation, and even sharpness.
Read More
Elti Meshau — Unsplash
While these images were not accompanied with an expert witness or proof of reasonable scientific accuracy, we the jury were left to assume that the relative viewing condition at the time of the murder was such that a woman peering through a window from the third floor could see an African American man under a hoodie. Baloney!
Read More
Most of you who are reading this article are likely to have been around when the United States switched off analogue television broadcasting in the summer of 2009. The great migration to digital cut most of us off from daily interaction with cosmic noise as advanced appliances began to use sophisticated methods of filtering out unwanted interference.
Read More