Video Production with Generative AI
Metadata
- Publisher
- SMPTE — Hollywood, CA
- Doc Type
- Conference Paper
- Content Type
- Original Research
- Volume
- 00, pp. 1–10
- Abstract
- The number of applications for Generative AI (GenAI) in video production is rapidly increasing. Many kinds of GenAI models are used, from audio generation to text generation models. However, video generation models (VGMs) are foundational for video production and are gaining importance as their power and expressiveness increases. In regard to expressiveness, VGMs are different from prior kinds of GenAI models in that VGM users must have domain-specific knowledge in order to optimally leverage VGMs to express their artistic vision. More specifically, it can be demonstrated that VGM users must know at least the basics of the visual language of cinematography in order to properly translate their artistic vision into quality VMG output. The tool by which users apply cinematographic language to VGMs is prompt engineering, which is used to craft VGM output into something useful and aesthetically viable. Using prompt engineering, it is possible to ask a VGM to instantiate the basic elements of cinematography such as: camera placement, camera movement, shot composition, shot size, focus/depth of field, and lighting. Additionally, GenAI can be used to create storyboards and animatics to help plan out these cinematographic elements. VGMs and other GenAI models also are useful in post-production workflows. This includes not only editing, but also other post-production tasks such as visual effects.
- Publication Date
- 2024-10-21
- DOI
10.5594/MOO/3041- ISBN
- Electronic:
978-1-61482-965-2 - Link
- https://doi.org/10.5594/MOO/3041
- Author(s)
- Brent Rabowsky
- Keyword(s)
- Generative AI, GenAI, Video Generation, Video Production
- Copyright
- © 2024 SMPTE
Bibliographic Reference(s)
- T. Maglio, “Sony Will Use AI to Cut Film Costs, Says CEO Tony Vinciquerra,” In die Wire, 30-May-2024. [Online]. Available: https://www.indiewire.com/news/breaking-news/sony-pictures-will-cut-film-costs-using-ai-1235010605/. [Accessed: 3-August-2024]. EXTERNAL
- N. Magoon,, “Tech in Content Production: Will AI Kill the Video Star?,” Bain & Company, 28-Sept-2023. [Online]. Available: https://www.bain.com/insights/tech-in-content-production-will-AI-kill-the-video-star/. [Accessed: 3-August-2024]. EXTERNAL
- B. Rabowsky, “Applications of Generative AI to Media,” SMPTE Mot. Imag. J., 132 (8): 53–57, Sept. 2023. EXTERNAL
- A. Still, “Bringing generative AI to video editing workflows in Adobe Premiere Pro,” Adobe Blog, 15-April-2024. [Online]. Available: https://blog.adobe.com/en/publish/2024/04/15/bringing-gen-ai-to-video-editing-workflows-adobe-premiere-pro. [Accessed: 4-August-2024]. EXTERNAL
- M. Seymour, “Actually Using SORA,” fxguide, 14-April-2024. [Online]. Available: https://www.fxguide.com/fxfeatured/actually-using-sora/. [Accessed: 4-August-2024]. EXTERNAL
- M. Seymour, “1st SORA music video,” fxguide, 8-May-2024. [Online]. Available: https:/Iwww.fxguide.com/fxfeaturedl1st-sora-music-video-how-sora-is-evolving-guessing-possible-pricing/. [Accessed: 4-August-2024]. EXTERNAL
- E. David, “Toys R Us unveils first commercial made with OpenAI's Sora,” VentureBeat, 25-June-2024. [Online]. Available: https://venturebeat.com/ai/toys-r-us-unveils-first-commercial-made-with-openais-sora/. [Accessed: 11-August-2024]. EXTERNAL
- Y. Zhang,, “ControIVideo: Training-free Controllable Text-to-Video Generation,” arXiv:2305.13077, May 2023 [Online]. Available: https://arxiv.org/abs/2305.13077. [Accessed: 14-Julv-20241. EXTERNAL
- A. Blattmann,, “Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets,” arXiv:2311.15127, November 2023 [Online]. Available: https://arxiv.org/abs/2311.15127. [Accessed: 10-August-2024]. EXTERNAL
- A. Schomer, “How Much Should AI Giants Pay Hollywood?,” Variety, 21-August-2024. [Online]. Available: https://variety.com/vip/ai-licensing-training-studios-hollywood-content-film-tv-12361129601. [Accessed: 22-August-2024]. EXTERNAL
- A. Weprin, “Lionsgate Inks Deal With AI Firm to Mine Its Massive Film and TV Library,” The Hollywood Reporter, 18-September-2024. [Online]. Available: https://www.hollywoodreporter.com/business/business-news/lionsgate-deal-ai-firm-runway-1236005554/. [Accessed: 24-September-2024]. EXTERNAL
- J. Alton, Painting with Light, Macmillan: New York, NY, 1949. EXTERNAL
- B. Brown, Cinematography: Theory and Practice, Focal Press: New York, NY, 2002. EXTERNAL
- Amazon.com, Inc., “Amazon Ads launches a new AI Video generator-here's your first look at the beta,” [Online]. Available: https://www.aboutamazon.com/news/innovation-at-amazon/amazon-ads-generative-ai-video-generator-advertisers. [Accessed: 24-September-2024]. EXTERNAL
- Runway AI, Inc., “Gen-3 Alpha Prompting Guide,” [Online]. Available: https://help.runwaymI.com/hc/en-us/articles/30586818553107-Gen-3-AIpha-Prompting-Guide. [Accessed: 12-August-2024]. EXTERNAL
- G. Chen,, “VideoLLM: Modeling Video Sequence with Large Language Models,” arXiv:2305.13292, May 2023 [Online]. Available: https://arxiv.org/abs/2305.13292. [Accessed: 17 -August-2024]. EXTERNAL
Source Data (JSON)
Full registry record with provenance metadata. Open directly: /api/doc/10.5594-MOO-3041.json
Reference Tree
Explore all references and references to this document, as a navigable tree.
Open Reference TreeReference this Doc
Plain text (ISO 690 compliant)
Preview:
Brent Rabowsky; Video Production with Generative AI, MTS 2024, Article 30 (pp. 1 to 10); SMPTE, 2024, ISBN: 978-1-61482-965-2. Available at https://doi.org/10.5594/MOO/3041
Snippet:
Brent Rabowsky; Video Production with Generative AI, MTS 2024, Article 30 (pp. 1 to 10); SMPTE, 2024, ISBN: 978-1-61482-965-2. Available at https://doi.org/10.5594/MOO/3041
HTML (ISO 690 compliant)
Preview:
Brent Rabowsky; Video Production with Generative AI, MTS 2024, Article 30 (pp. 1 to 10); SMPTE, 2024, ISBN: 978-1-61482-965-2. Available at https://doi.org/10.5594/MOO/3041
Snippet:
<span class="citation">Brent Rabowsky; <cite>Video Production with Generative AI</cite>, MTS 2024, Article 30 (pp. 1 to 10); SMPTE, 2024, ISBN: 978-1-61482-965-2. Available at <a href="https://doi.org/10.5594/MOO/3041" target="_blank" rel="noopener">https://doi.org/10.5594/MOO/3041</a></span>
SMPTE's HTML Pub
Preview:
Brent Rabowsky; Video Production with Generative AI, MTS 2024, Article 30 (pp. 1 to 10); SMPTE, 2024, ISBN: 978-1-61482-965-2
doi: 10.5594/MOO/3041
url: https://doi.org/10.5594/MOO/3041
doi: 10.5594/MOO/3041
url: https://doi.org/10.5594/MOO/3041
Snippet:
<li> Brent Rabowsky; <cite id="bib-10-5594-moo-3041">Video Production with Generative AI</cite>, MTS 2024, Article 30 (pp. 1 to 10); SMPTE, 2024, ISBN: 978-1-61482-965-2 <span class="doi">10.5594/MOO/3041</span> </li>