Thanks for trying or purchasing Eff-it!
It’s a great tool for automatically bleeping out words in Premiere and this quick start guide should get you up and running in no time.
Photoshop’s Generative Fill Isn’t Great, But It Works Well at Fixing Other GenAI Images
One problem with generative AI is that it’s difficult to get exactly what you want. You can often get something that’s good enough but more often than not, you get 90% of the way to what you want and getting the AI to make the correct changes to get to a 100% is daunting. (which is why I’m a little skeptical about GenAI for video. For generic B-roll stuff, sure maybe, but wrangling the correct prompts for a 30 second video that needs to be exactly this or that is going to be difficult to say the least. It’s hard enough for a single still image.) Photoshop’s Generative AI (called Generative Fill) is pretty subpar when compared to some of the more cutting edge ones (DALL-E, Stability AI, etc) for creating images from scratch. However, what it does pretty well is extending images. i.e. If you’ve got an image that you want wider or more head room than it was originally shot with. OR… if you’ve created something with another AI tool, like DALL-E, as I’ve done here. DALL-E gave me more or less what I wanted but without much of a tail. I spent another 20 minutes or so trying to get DALL-E to give me this fish with a tail before giving up. It really wanted to redo the entire image. So it got frustrating. This is where Photoshop’s GenAI turned out to be useful. To be fair, they market it as more of a way to extend/improve existing images than creating stuff from scratch. It can create from scratch but the results often aren’t great. But when it comes to extending images, there’s a big advantage to being in Photoshop… selections! You can make the canvas wider, select the empty area to the side and type in ‘extend image’. Boom. Now of course it gave me some other variations that didn’t work at all, but doesn’t matter. It gave me a great variation that did work. Also, prompting something like ‘extend image with skeleton of an angler fish’ didn’t work. It was the simpler prompt ‘extend image’ that did the trick. (Prompts are weird and a whole art unto themselves. Figuring out what the AI is going to respond to takes a LOT of trial and error. And then you still need to get it to do what you want.) I then selected the other side and it created that easily. You can see slight seams where the image was extend. When having Photoshop create the extensions, I tried both selecting the area by itself and selecting a little of the original image (including feathering it). It didn’t really make much difference. You got slightly different imagery but the seams tended to show up no matter what. The tail was the worst problem however. There was an obvious change in style from the original to the Photoshop extension. So I selected just that bit and ran Content Aware Fill a few times to cover up the seam. And that worked reasonably well despite CA Fill not being AI. It’s just sampling from other parts of the image. Selecting the seam and running Generative Fill (prompt: ‘remove seam’) on it created three variations. Two of the three didn’t work but the third one arguably looks better than CA Fill. But they’re both pretty good. So just realize CA Fill can help touch up slight imperfections as well. Getting DALLE, Midjourney or whatever to give you exactly what you want can be difficult. If you get most of the way there, but are having trouble prompting those tools to fill in the details, Photoshop’s Generative Fill may be able to touch things up or extend the image more easily. Here’s the final image:
Only Beauty Box 5.x Supports Metal GPUs and Apple Silicon
Beauty Box 5.0 and higher supports Metal and Apple Silicon (M1, M2, etc.). This includes the upcoming Beauty Box 6. However, Beauty Box 4.0 does not support Metal GPU rendering on Macintosh. It uses the older OpenCL technology for GPU processing. (on Windows, 4.0 works fine) Premiere Pro/After Effects 2022 and later dropped support for OpenCL rendering, and only supports Metal on the M/Silicon chips and Intel Macs. This means Beauty Box 4.0 does not support GPU rendering in the current Intel builds of After Effects or Premiere. It doesn’t work at all on Silicon Macs. If you’re experiencing slow rendering in Adobe products on a Mac with Beauty Box 4.0 or it’s not showing up at all, that’s probably why. So if you have 4.0 and have an Intel Mac, you’ll probably want to upgrade to 5.0. If you have a M/Silicon Mac you’ll need to upgrade. 5.0 was released before the Silicon chips and that’s the only version of Beauty Box that’s been re-written for those chips. On Windows, Beauty Box 4.0 should still work fine. Both OpenCL and CUDA (for Nvidia) are still supported by Premiere and After Effects. If you’re experiencing slow render times in 5.0 on Intel, double check that Hardware rendering is set to Metal. (on Apple Silicon Macs, it is always set to Metal and you can’t change it) In both Premiere and After Effects go to File>Project Settings>General to change If this is not why you’re having a problem with Beauty Box, try these articles or contact support: Reasons Plugins Might Not Show in the Effects Menu Use GPU can be turned off in the plugin or the Beauty Box ‘About’ dialog.
THE PROCESS OF STORYCRAFT
(This is a guest blog post by editor/director Kyle Koch) I was recently asked about how to craft/edit narrative projects and thought this would be a cool share for my colleagues.~~~~~~~~~~~~~~~~~~~~~~~~ THE PROCESS OF STORYCRAFTI have about 30+ years under my belt and much has been doc cutting. I can give a few tips. STORY IS EVERYTHING Craft a journey that hooks and keeps the audience watching. Weave ever rising plateaus of insight and intrigue with your interviews and location sound ups. CONTENT DRIVES THE PROCESSIn other words, if the story relies heavily on historical footage, you’ll need to craft story around those assets. Is the content esoteric, philosophical, spiritual? You may need to create support visuals that only come to light from the content itself. Either way, story and timing is the foundation for all the imagery that paints the picture of the narrative. BE HONEST WITH THE GENREIs it a story if humour, drama, both? When is it taking itself too seriously? When not seriously enough? Through the stages of post, this may change. It’s important to have a solid sense of what the genre is before you start. TRANSCRIPTS ARE KEYBeing time efficient with a large amount of material is challenging. Decisions on how to process assets can exponentially impact the time it takes to craft the story. I always start with transcripts of my interviews. Using the script/outline as my window upon which to view the content, I identify themes, keywords/phrases, colour code them, and add markers. Finding special moments where the delivery by an interviewee is particularly strong is critical. KNOW YOUR CONTENT One of the most important things I do to help me craft story is make sound files that I can use to listen to source material or the edit while I’m walking the dog, driving, etc – anything but being in front of the computer. I’ll add markers to the file with my phone (or take screenshots). Ideally, use Frame to add comments so that you can upload them to your sequence as markers. FIND THE GOLDMake selects of your content by using stringouts (source sequences). Raise good shots/content up in the video tracks (v1 is the base, v2 good, v3 great, v4 awesome). This process will allow you to get to know the materials and move the project forward as you’re watching assets. TEXT DRIVEN VIDEO CUTS ARE AMAZING!!!Jim Tierney of Digital Anarchy is one of the first code warriors to give us this superpower his tool Transcriptive. 5 years ahead of the curve! Adobe has on-boarded much of the functions now, but those Jim has been a pioneer for transcription based editing. Certainly, Transcriptive offers functionality that is deeper than what is included with an Adobe CC account. I’ll be attending the upcoming AdobeMax in Los Angeles the week of October 9-20, 2023. The event has a bunch of good in-person and on-line sessions for all sorts of topics including the creative process, AI augmentation, team building, content authentication (DRM), etc. If you’re attending, please say hello. Feel free to reach out to me directly:kwk@TrueNorthEntertainment.com – I look forward to additional insight/tips from my pals in production! ? ~~~~~~~~~~~~~~Kyle’s has been crafting content for 25+ years. He owns a creative agency, True North Entertainment and is Admin of the largest professional editors group dedicated to Adobe workflow solutions. http://www.truenorthentertainment.com/https://www.facebook.com/groups/adobepremierepro/,
Skin Detail Smoothing and 4K
Beauty Box’s settings are resolution dependent. This means the same settings you have for HD may not work for 4K. On a basic level, it’s similar to applying a blur. A Gaussian Blur of 1.0 might be too much for a low res, 640×480 image, but might be almost unnoticeable on a 4K image. Also, the ‘right’ settings may depending on the framing of the shot. Is the footage a tight close up where the talent’s face fills most of the frame? Or is it pulled back to show three or four actors? The settings that are ideal for one of those examples, probably won’t be ideal for the other one. The default settings for Beauty Box are really designed for HD. And even for HD they may be a bit heavy, depending on the footage. Often they aren’t the ideal settings for 4K (or 12K or whatever). So in this post we’ll talk about what to do if you have 4K footage and aren’t getting the look you want. Mainly I want to focus on Skin Detail Smoothing, as I think that plays a bigger role than most people think. AND you can set it negative! Skin Detail Smoothing As you might expect from the name, this attempts to smooth out smaller features of the skin: pores and other small textures. It provides sort of a second level of smoothing on top of the overall skin smoothing. You generally want this set to a lower value than the Smoothing Amount parameter. If it’s set too high relative to Smoothing Amount, you can end up with the skin looking somewhat blurry and blotchy. This is due to Skin Detail Smoothing working on smaller areas of the skin. So instead of the overall smoothing, you get a very localized blur which can look blotchy. So, first off: Set Skin Detail Smoothing to a lower value than Smoothing Amount. (usually: there are no hard and fast rules with this. It’s going to depend on your footage. But most of the time that’s a very good rule of thumb. Negative Skin Detail Smoothing With 4k and higher resolutions it’s sometimes helpful to have a slightly negative value for Skin Detail Smoothing. Like -5 or -10. The smoothing algorithms occasionally add too much softness and a slightly negative value brings back some of the skin texture. In the example, the area around her nose gets a bit soft and using a negative value, IMO, gives it a better look. The adjustment is pretty subtle but it does have an effect. You may have to download the full res images and compare them in Photoshop to truly see the difference. (click on the thumbnails below to see the full res images) This definitely isn’t the case for all 4K footage and, as always, you’ll need to dial in the exact settings that work for your footage. But it’s important to know that Skin Detail Smoothing can be set negative and sometimes that’s beneficial. Of course, I want to emphasize SLIGHTLY negative. Our Ugly Box free plugin makes use of negative Skin Detail Smoothing in a way that won’t make your footage look better. If you set it to -400… it’s good for Halloween but usually your clients won’t like you very much.
Testing A.I. Transcript Accuracy (most recent test)
Periodically we do tests of various AI services to see if we should be using something on the backend of Transcriptive-A.I. We’re more interested in having the most accurate A.I. than we are with sticking with a particular service (or trying to develop our own). The different services have different costs, which is why Transcriptive Premium costs a bit more. Gives us more flexibility in deciding which service to use. This latest test will give you a good sense of how the different services compare, particularly in relation to Adobe’s transcription AI that’s built into Premiere. The Tests Short Analysis (i.e. TL;DR): For well recorded audio, all the A.I. services are excellent. There isn’t a lot of difference between the best and worst A.I… maybe one or two words per hundred words. There is a BIG drop off as audio quality gets worse and you can really see this with Adobe’s service and the regular Transcriptive-A.I. service. A 2% difference in accuracy is not a big deal. As you start getting up around 6-7% and higher, the additional time it takes to fix errors in the transcript starts to become really significant. Every additonal 1% in accuracy means 3.5 minutes less of clean up time (for a 30 minute clip). So small improvements in accuracy can make a big difference if you (or your Assistant Editor) needs to clean up a long transcript. So when you see an 8% difference between Adobe and Transcriptive Premium, realize it’s going to take you about 25-30 minutes longer to clean up a 30 minute Adobe transcript. Takeaway: For high quality audio, you can use any of the services… Adobe’s free service or the .04/min TS-AI service. For audio of medium to poor quality, you’ll save yourself a lot of time by using Transcriptive-Premium. (Getting Adobe transcripts into Transcriptive requires a couple hoops to jump through, Adobe didn’t make it as easy as they could’ve, but it’s not hard. Here’s how to import Adobe transcripts into Transcriptive) (For more info on how we test, see this blog post on testing AI accuracy) Long Analysis When we do these tests, we look at two graphs: How each A.I. performed for specific clips The accuracy curve for each A.I. which shows how it did from its Best result to Worst result. The important thing to realize when looking at the Accuracy Curves (#2 above) is that the corresponding points on each curve are usually different clips. The best clip for one A.I. may not have been the best clip for a different A.I. I find this Overall Accuracy Curve (OAC) to be more informative than the ‘clip-by-clip’ graph. A given A.I. may do particularly well or poorly on a single clip, but the OAC smooths the variation out and you get a better representation of overall performance. Take a look at the charts for this test (the audio files used are available at the bottom of this post): All of the A.I. services will fall off a cliff, accuracy-wise, as the audio quality degrades. Any result lower than about 90% accuracy is probably going to be better done by a human. Certainly anything below 80%. At 80% it will very likely take more time to clean up the transcript than to just do it manually from scratch. The two things I look for in the curve is where does it break below 95% and where does it break below 90%. And, of course, how that compares to the other curves. The longer the curve stays above those percentages, the more audio degradation a given A.I. can deal with. You’re probably thinking, well, that’s just six clips! True, but if you choose six clips with a good range of quality, from great to poor, then the curve will be roughly the same even if you had more clips. Here’s the full test with about 30 clips: While the curves look a little different (the regular TS A.I. looks better in this graph), mostly it follows the pattern of the six clip OAC. And the ‘cliffs’ become more apparent… Where a given level of audio causes AI performance to drop to a lower tier. Most of the AIs will stay at a certain accuracy for a while, then drop down, hold there for a bit, drop down again, etc. until the audio degrades so much that the AI basically fails. Here are the actual test results: TS A.I. Adobe Speechmatics TS Premium Interview 97.2% 97.2% 97.8% 100.0% Art 97.6% 97.2% 99.5% 97.6% NYU 91.1% 88.6% 95.1% 97.6% LSD 92.3% 96.9% 98.0% 97.4% Jung 89.1% 93.9% 96.1% 96.1% Zoom 85.5% 80.7% 89.8% 92.8% Remember: Every additonal 1% in accuracy means 3.5 minutes less of clean up time (for a 30 minute clip). So that’s the basics of testing different A.I.s! Here are the clips we used for the smaller test to give you an idea of what’s meant by ‘High Quality’ or ‘Poor Quality’. The more jargon, background noise, accents, soft speaking, etc there is in a clip, the harder it’ll be for the A.I. to produce good results. And you can hear that below. You’ll notice that all the clips are 1 to 1.5 minutes long. We’ve found that as long as the clip is representative of the whole clip it’s taken from, you don’t get any additional info from the whole clip. An hour long clip will product similar results to one minute, as long as that one minute has the same speakers, jargon, background noise, etc. Any questions or feedback, please leave a note in the Comments section! (or email us at cs@digitalanarchy.com)
Removing flicker from concert videos (or anything with stage lights)
LED lights are everywhere and nowhere moreso than concerts or other performances. Since they are a common source of flicker when shot with a video camera, it’s something we get asked about fairly regularly. Other types of stage lights can also be problematic (especially in slow motion), but LED lights are the more common culprit. You can see this clearly in this footage of a band. It’s a slow motion clip shot with an iPhone… which will shoot a few seconds of regular speed video before switching to slo-mo. So the first five seconds are regular speed and the next five are slo-mo: To remove the flicker we’re using our plugin Flicker Free, which supports After Effects, Premiere, Final Cut Pro, Resolve, and Avid. You can learn more about it and download the trial version here. The regular lights are fine, but there are some LED lights (ones with the multiple lights in a hexagon) that are flickering. This happens in both the regular speed and slow motion video portions of the video. You’ll notice, of course, the flickering is slower in the slo-mo portion. (mixed frame rates can sometimes be a problem as well but not in this case) Usually this is something Flicker Free can fix pretty easily and does so in this case, but there are a few variables that are present in this video that can sometimes complicate things. It’s a handheld shot (shaky), there are multiple lights, and there are performers (who, luckily, aren’t moving much). Handheld shot: The camera is moving erratically. This can be a problem for Flicker Free and it’s something the Motion Compensation checkbox was specifically designed to deal with (in addition to the Detect Motion settings). However, in this case, the camera isn’t moving quickly which is where this really becomes a problem. So we can get away with only having Detect Motion on. Also… with stage performances there is also often a lot of movement of the performers. Not a problem here, but if there is a lot of performer movement, it’s likely you’ll really need to turn Motion Compensation on. Motion Compensation increases the render time, so if you don’t need to turn it on, then it’s best not to. But some footage will only be fixable with it on, so if the default settings aren’t working, turn on Motion Compensation. As is often the case the default settings, the preset Rolling Bands, works great. This is very common with LED lights as they produce a certain type of flicker that the preset/default works very well on. Multiple Lights: It’s possible to have multiple lights in the scene that flicker, and do so at different rates. Flicker Free can usually handle this scenario, but sometimes you need to apply two instances of Flicker Free. If you do this, it’s highly recommended not to use Motion Compensation and either turn Detect Motion off or set it to Fast. If you have Motion Compensation on and use two instances of FF, you’ll get exponentially longer render times and you might run out of memory on the GPU causing a crash. Slow Motion: Slo-mo footage can really slow the flickering down, requiring you to max out Time Radius. Again, this is a setting that can increase render times, so lower values are better if you can get away with them and it fixes the flicker. This clip was fairly easy. Only one light was an LED light and flickering, So the default settings worked great. If the default settings don’t work there are a few other presets to try: Stage Lights, Projection Screen, etc. But, even if those don’t work right off the bat, hopefully this gives you some tips on how to fix even the most challenging videos of performances.
Transcriptive OnBoarding: Where to Find Everything
Welcome to Transcriptive Rough Cutter! This page/video will give you an overview of the UI, so you know where to find everything! The five minute video above will give the quickest explanation. But for those that prefer reading, skip past the video and all the info is below it. Happy Transcribing! (and so much more 🙂 We’re going to go over the different areas of Transcriptive Rough Cutter, so you know where to find stuff, but we’ll leave deeper explanations for other tutorials. So let’s get started. The first thing to know is the Clip Mode switch. If this is on. Then Transcriptive is going to be looking at the project panel and as you select different clips, it’s going to show the transcript for that clip. If you have this turned off, it will be in sequence mode and the transcript will be for whichever sequence is currently active. So as we switch to different sequences, it will load the transcript for that sequence. Next up is getting the transcript itself. Now, if you already have a transcript in text format, you can click on the import dialog and you can import the transcript and the variety of different ways. But if you need to get the transcript, then you click on Transcribe. And this will allow you to select which speech service to go with. Alignment is also here. So if you’ve imported a text based transcript that does not have timecode, alignment will analyze the audio and add timecode to your imported text file. Now you can also select the language. Glossary allows you to type in names or terminology that the A.I. might not necessarily know, and they can definitely improve accuracy. If you have a lot of terms that are maybe unusual. So Transcribe dialogue is incredibly important. The Auto Load button that’s next to Transcribe tells Transcriptive Rough Cutter to automatically load the transcript for whatever you click on. So if I’m down in the project panel and clicking on different clips, Transcriptive will load the transcript for those clips. If you want to lock it to just one clip/sequence, say you’re using a sequence and you always want to see the transcript for that sequence… turn on Sequence Mode (Clip Mode = Off) and no matter what we do this transcript will be shown if Auto Load = Off. So as I move to different sequences or clips, the transcript for the sequence that was shown when we turned Auto Load off, is now always shown. And of course, you can edit text. it works similar to a word processor, except every word has timecode. You can just click on words and start typing away. Because every word has timecode, as you click on different words, it’s going to move to different points on the Sequence/Clip timeline. You can also change speakers by clicking on the speakers and selecting from the dropdown menu. On the right hand side, you’ll see three icons. This is where you add Comments or Delete entire blocks of text or Strike Through the text for use with the Rough Cut feature. And that brings us to the main menu where you can Manage Speakers. You can click on that. It will allow you to add speakers, change the name of speakers, and all that will then be reflected in the dropdown you see in the Text Edit window. Sync Transcript to Sequence Edit is an important item. If you edit your sequence and delete stuff, it will become out of sync with the transcript. The transcript will have stuff that’s no longer in your sequence. If you select Sync Transcript To Sequence Edit, Transcriptive will go through your sequence and rebuild the transcript to match what you’ve edited out. You can also Batch Files. Which is very important if you have lots of clips that you want to get transcribed. Batch is a very easy way of processing as many clips as you want at the same time. There are multiple ways you can do it, Batch Project will transcribe anything selected in the Project panel and with Batch Files/Folder, you select files from the operating system. (kind of like importing files into Premiere) If you need to do a lot of transcribing this is a VERY important feature. At the bottom of the Transcriptive Rough Cutter panel you have the ability to Export Files like a text file of the entire transcript. And… There’s the Rough Cut button, which is a key feature of Transcriptive Rough Cutter. It will take a transcript that you have edited and build a sequence based on that transcript. So if you delete text, it will delete that portion of the clip or sequence from the new Rough Cut. So this is a feature that requires a bit of explaining, so I definitely encourage you to check out the in depth tutorial on Rough Cut. You also have the ability to search. Search is one of the most powerful features of Transcriptive Rough Cutter, along with Power Search, which is the other panel that ships with transcript. Here you can search the entire transcript that’s in the Transcriptive Rough Cutter Panel. You can also replace words, but the real power comes from Power Search, which can search the entire project. So if you’re looking for something and you’re not quite sure what transcript or clip it’s from. You can type in the term and get a bunch of search results, much like any other search engine. And when you click on those results, it’s going to open up that clip and jump right to where that happens. So we can click on these other ones, it’ll work for sequences as well. And then when you load that up, that transcript will appear in Transcriptive Rough Cutter. And since there’s nothing else like this anywhere in Premiere itself, this is a really powerful way of making use of your transcripts. If you’d
The Rule of Thirds in Practice
Most of us have heard of the rule of thirds. And probably for most readers of this blog it’s second nature by now. But for those somewhat new to photo/videography or if you just want to see how someone else uses/breaks the rules, I figured a serendipitous photoshoot recently would be a good example. What is the Rule of Thirds? It’s splitting an image into three parts vertically and horizontally. This can help you create a more pleasing image composition. And like all rules, it’s meant to be bent and broken. Let’s talk about how to use it. Sometimes you use the rule of thirds while you’re shooting. If you’re doing a portrait, you can pose your model, frame her in camera and take the shot. Personally, I tend to be more of a wildlife photographer. Birds and whales don’t usually pose for you… you’re just trying to take the shot as fast as f’ing possible while you have the chance! You can crop the photo later to make it fit the rule of thirds (or not). Recently I was sitting on the balcony of my house and a hawk decided to perch himself right in front of me on a neighbors house. So, grabbed the camera for an impromptu photoshoot: Those are the cropped ‘rule of third’ shots. Here are the original shots (which are cool in their own way by showing more of the environment): Let’s talk about why I cropped them the way I did. First off, look at the cropped images. Did you notice that I’m trying to tell a small story with how they’re cropped? (or, at least, framing things so there’s some context to the sequence of images) Let’s take a look at the first image. One of the things that makes the Rule of Third compelling is that asymmetrical compositions generally look better. But not always! Here we have the ‘hero’ shot of our hawk. I’m introducing him and, as such, he’s pretty much in the center of the frame. In the next picture, he turns his head to look at something. Now he’s off center and edging towards left and down. We’re creating space off to the right side of the image. Where is he looking? What is he looking at? I want the viewer to be as curious about that as the hawk is. So I want to add space in the image so you can follow his gaze. Now he’s preparing to take off! His wings are up and he’s getting ready to fly. I want to add even more space to the right and above him. So I crop the image so he’s split down the middle by the first third line. Because his wings are raised, he’s centered vertically, but he’s still weighted towards the lower third. Hopefully your eye is drawn to where he might be going. Lift Off! His wings come down and he levitates in preparation to fly. Again, I want the greenery in the shot, so he’s a little lower in the frame than is ideal, but it works. He’s about to take off so having a lot of space up and in the direction he’s going to be flying is all good. (I love this shot… birds of prey are just so amazing) However, usually you don’t want your subject quite so close to the edge. I think it’s a great shot, but you could definitely make the case there’s too much space in the rest of the image. If the trees were closer, I would’ve cropped it differently, but to get them in the image, I had to stretch it a bit towards the upper, right corner. With wildlife you don’t always get to pick your shot! And he’s off! And… so is this image. Why is this not a great composition? The hawk really should be centered more vertically. He’s a little low in the frame. To correct it, I’d at least move where the wing bends into the upper third. Bonus tip: the glaring issue with all these photos… well, you can see it hopefully. It’s something easily fixed with Photoshop’s Content Aware Fill. And if you can’t see the problem, perhaps it’s absence will give you a clue: So hopefully that’s a good intro on how to use the rule of thirds. It’s really about drawing the eye in the direction the subject is looking or heading towards. And, of course, it’s not a hard and fast ‘rule’. Just one way to think about composing your images.
Transcription Accuracy: Adobe Sensei vs Transcriptive A.I.
Speechmatics, one of the A.I. engines we support, recently released a new speech model which promised much higher accuracy. Transcriptive Rough Cutter now supports that if you choose the Speechmatics option. Also with Premiere able to generate transcripts with Adobe Sensei, we get a lot of questions about how it compares to Transcriptive Rough Cutter. So we figured it was a good time to do a test of the various A.I. speech engines! (Actually we do this pretty regularly, but only occasionally post the results when we feel there’s something newsworthy about them) You can read about the A.I. testing methodology in this post if you’re interested or want to run your own tests. But, in short, Word Error Rate is what we pay most attention to. It’s simply: NumberOfWordsMissed / NumberOfWordsInTranscript where NumberOfWordsMissed = the number of words in the corrected transcript that the A.I. failed to recognize. If instead of the word ‘Everything’ the A.I. produced ‘Even ifrits sing’, it still missed just one word. In the reverse situation, it would count as three missed words. We also track punctuation errors, but those can be somewhat subjective, so we put less weight on that. What’s the big deal between 88% and 93% Accuracy? Every 1% of additional accuracy means roughly 15% less incorrect words. A 30 minute video has, give or take, about 3000 words. So with Speechmatics you’d expect to have, on average, 210 missed words (7% error rate) and with Adobe Sensei you’d have 360 missed words (12% error rate). Every 10 words adds about 1:15 to the clean up time. So it’ll take about 18 minutes more to clean up that 30 minute transcript if you’re using Adobe Sensei. Every additonal 1% in accuracy means 3.5 minutes less of clean up time (for a 30 minute clip). So small improvements in accuracy can make a big difference if you (or your Assistant Editor) needs to clean up a long transcript. Of course, the above are averages. If you have a really bad recording with lots of words that are difficult to make out, it’ll take longer to clean up than a clip with great audio and you’re just fixing words that are clear to you but the A.I. got wrong. But the above numbers do give you some sense of what the accuracy value means back in the real world. The Test Results! All the A.I.s are great at handling well-recorded audio. If the talent is professionally mic’d and they speak well, you should get 95% or better accuracy. It’s when the audio quality drops off that Transcriptive and Speechmatics really shine (and why we include them in Transcriptive Rough Cutter). And I 100% encourage you to run your own tests with your own audio. Again, this post outlines exactly how we test and you can easily do it yourself. Speechmatics New is the clear winner, with a couple first place finishes, no last place finishes, and at 93.3% rate overall (you can find the spreadsheet with results and the audio files further down the post). One caveat… Speechmatics takes about 5x as long to process. So a 30 minute video will take about 3 minutes with Transcriptive A.I. and 15-20 minutes with Speechmatics. If you select Speechmatics in Transcriptive, you’re getting the new A.I. model. Adobe Sensei is the least accurate with two last place finishes and no first places, for a 88.3% accuracy overall. Google, which is another A.I. service we evaluate but currently don’t use, is all over the place. Overall, it’s 80.6%, but if you remove the worst and best examples, it’s a more pedestrian 90.3%. No idea why it failed so badly on the Bill clip, but it’s a trainwreck. The Bible clip is from a public domain reading of the bible, which I’m guessing was part of Google’s training corpus. You rarely see that kind of accuracy unless the A.I. was trained on it. Anyways, this inconsistency is why we don’t use it in Transcriptive. Here are the clips we used for this test: Here’s the spreadsheet of the results (SM = Speechmatics, Green means best performance, Orange means worst). Again, mostly we’re focused on the Word Accuracy. Punctuation is a secondary consideration:
How do We Test Speech-to-Text Services for Accuracy?
Transcriptive-A.I. doesn’t use a single A.I. services on the backend. We don’t have our own A.I., so like most companies that offer transcription we use one of the big companies (Google, Watson, Speechmatics, etc). We initially started off with Speechmatics as the ‘high quality’ option. And they’re still very good (as you’ll see shortly), but not always. However, since we had so many users that liked them, we still give you the option to use them if you want. However, we’ve now added Transcriptive-A.I. This uses whatever A.I. services we think is best. It might use Speechmatics, but it might also use one of a dozen other services we test. Since we encourage users to test Transcriptive-A.I. against any service out there, I’ll give you some insight on how we test the different services and choose which to use behind the scenes. Usually we take 5-10 audio clips of varying quality that are about one minute long. Some very well recorded, some really poorly recorded and some in between. The goal is to see which A.I. works best overall and which might work better is certain circumstances. When grading the results, I save out a plain text file with no timecode, speakers or whatever. I’m only concerned about word accuracy and, to a lesser degree, punctuation accuracy. Word accuracy is the most important thing. (IMO) For this purpose, Word 2010 has an awesome Compare function to see the difference between the Master transcript (human corrected) and the A.I. transcript. Newer versions of Word might be better for comparing legal documents, but Word 2010 is the best for comparing A.I. accuracy. Also, let’s talk about the rules for grading the results. You can define what an ‘error’ is however you want. But you have to be consistent about how you apply the definition. Applying them consistently matters more than the rules themselves. So here are the rules I use: 1) Every word in the Master transcript that is missed counts as one error. So ‘a reed where’ for ‘everywhere’ is just one error, but ‘everywhere’ for ‘every hair’ is two errors. 2) ah, uh, um are ignored. Some ASRs include them, some don’t. I’ll let ‘a’ go, but if an ‘uh’ should be ‘an’ it’s an error. 3) Commas are 1/2 error and full stops (period, ?) are also 1/2 error but there’s an argument for making them a full error. 4) If words are correct but the ASR tries to separate/merge them (e.g. ‘you’re’ to ‘you are’, ‘got to’ to ‘gotta’, ‘because’ to ’cause) it does not count as an error. That’s it! We then add up the errors, divide that by the number of words that are in the clip, and that’s the error rate!
Upgraded to FCP 10.6? Please Update Your Plugins.
Apple just launched Final Cut Pro 10.6 which has some cool new features, like Face Tracking. Unfortunately they also introduced a bug or two. One of which prevents our plugins from registering. So… we updated all our plugins to work around the issue. Please go here: https://digitalanarchy.com/demos/psd_mac.html And you can download the updated version of any plugin you own. You only need to do this if you’re doing a fresh install of the plugins. Updating FCP should not cause the problem. But if you’re re-installing the plugin, then you might need the updated version. The issue is that the Yellow Question Mark at the top of the Inspector, doesn’t open a dialog when it’s clicked. It should open up our registration dialog (or about box) as shown here: So if you’re clicking on the Question Mark to register and nothing happens… Please update your plugins! These are free updates if you own the most recent version of the plugin. If you own an older version and don’t want to upgrade, the licensing dialog DOES work in Motion. It’s only an FCP problem. So if you have Motion, you can apply the plugin there and register it.
