On Monday, Adobe announced Firefly Video Model, a new AI-powered text-to-video generation tool that can create novel videos from written prompts. It joins similar offerings from OpenAI, Runway, Google, and Meta in an increasingly crowded field. Unlike the competition, Adobe claims that Firefly Video Model is trained exclusively on licensed content, potentially sidestepping ethical and copyright issues that have plagued other generative AI tools.

Because of its licensed training data roots, Adobe calls Firefly Video Model “the first publicly available video model designed to be commercially safe.” However, the San Jose, California-based software firm hasn’t announced a general release date, and during a beta test period, it’s only granting access to people on a waiting list.

In the works since at least April 2023, the new model builds off of techniques Adobe developed for its Firefly image synthesis models. Like its text-to-image generator, which the company later integrated into Photoshop, Adobe hopes to aim Firefly Video Model at media professionals, such as video creators and editors. The company claims its model can produce footage that blends seamlessly with traditionally created video content.

https://arstechnica.com/ai/2024/10/adobe-unveils-ai-video-generator-trained-on-licensed-content/

On Tuesday, Adobe announced a new symbol designed to indicate when content has been generated or altered using AI tools, reports The Verge, as well as verifying the provenance of non-AI media. The symbol, created in collaboration with other industry players as part of the Coalition for Content Provenance and Authenticity (C2PA), aims to bring transparency to media creation and reduce the impact of misinformation or deepfakes online. Whether it will actually do so in practice is uncertain.

The Content Credentials symbol, which looks like a lowercase “CR” in a curved bubble with a right angle in the lower-right corner, reflects the presence of metadata stored in a PDF, photo, or video file that includes information about the content’s origin and the tools (both AI and conventional) used in its creation. The information is automatically added by supporting digital cameras and AI image generator Adobe Firefly, or it can be inserted by Photoshop and Premiere. It will also soon be supported by Bing Image Creator.

If credentialed media is presented in a compatible app or using a JavaScript wrapper on the web, users click the “CR” icon in the upper-right corner to view a drop-down menu containing image information. Or they can upload a file to a special website to read the metadata.

Adobe worked alongside organizations such as the BBC, Microsoft, Nikon, and Truepic to develop the Content Credentials system as part of the C2PA, a group aiming to establish technical standards for certifying the source and provenance of digital content. Adobe refers to the “CR” symbol as an “icon of transparency,” and the C2PA chose the initials “CR” in the symbol to stand for “credentials,” avoiding potential confusion with Creative Commons (CC) icons. Notably, the coalition owns the trademark for this new “CR” icon, which Adobe thinks will become as common as the copyright symbol in the future.

Adobe likens the content credentials to a “digital nutrition label,” or list of ingredients that make up the piece of media. “This list of ingredients will show verified information as key context so people can be sure of what they’re looking at,” the C2PA writes on its website. “This can include data about a piece of content, such as: the publisher or creator’s information, where and when it was created, what tools were used to make it, including whether or not generative AI was used, as well as any edits that were made along the way.”

Adobe isn’t the only company working on measures to provide ways to track the provenance of AI-generated content. Google has introduced SynthID, a content marker that similarly identifies AI-generated content within the metadata. Additionally, Digimarc has released a digital watermark that includes copyright information, aimed at tracking the use of data in AI training sets, according to The Verge.

https://arstechnica.com/?p=1975166

On Tuesday, Adobe announced major updates to AI image synthesis features across several products, including Photoshop, Illustrator, and Adobe Express. The updates include three new generative AI models—Firefly 2, Firefly Design Model, and Firefly Vector Model—which improve its previous offerings and add new capabilities. With the vector model, Adobe is notably launching its first text-to-vector AI image generator.

It’s been a busy year for generative AI, and Adobe has not been content to sit it out. In March, the graphic design software giant debuted its text-to-image synthesis model called Firefly, which it billed as an ethical alternative to Stable Diffusion and Midjourney due to being trained on Adobe Stock imagery only. Firefly can generate novel images based on text descriptions called prompts (i.e., “a scenic vista” or “a beefy-looking barbarian”). The company later brought the technology to Photoshop and web apps and promised to cover any legal bills that might arise from copyright claims against artwork generated with its tools.

Now, Adobe is extending its reach with a wave of new generative AI features. Adobe’s Firefly Image 2 model is an update to its original Firefly AI image generator, which powers Photoshop features like Generative Fill. Adobe claims this new version offers improved image quality, particularly in areas like foliage, skin texture, and facial features. In addition to these enhancements, the Firefly Image 2 model introduces AI-driven editing capabilities that can adjust various photo settings, like depth of field and motion blur. A new “Prompt Guidance” feature also aids users in refining the wording of their text descriptions and automatically completes prompts to boost efficiency.

Adobe unveiled a “Generative Match” feature, allowing users to match the style of generated content to specific images. It’s basically an “img2img” feature, similar to the technique often used in Stable Diffusion. This could appeal to companies wanting to consistently apply their brand style across various materials. The feature will remain in beta, and commercial use won’t be permitted during this trial period.

Adobe also introduced new Firefly models specifically for Illustrator that let users create editable vector images based on text prompts. Vector graphics differ from bitmapped graphics (found in files such as JPEGs) because the vector shapes are stored as mathematical formulas and can be scaled to any size without quality loss.

While Adobe claims that the Firefly Vector Model is “the world’s first generative AI model for vector graphics,” a quick Google search disproves this claim, showing earlier vector-based AI models that range from commercial offerings to older experimental research. However, it’s safe to say that this is the first vector-generating AI model from a company that is currently considered a major player in the image synthesis space. It’s also notable that the technology is baked into one of the world’s most popular vector graphics editors, Adobe Illustrator.

Meanwhile, in Adobe Express, Adobe’s Firefly Design model focuses on generating customizable templates for various media types, from print to online advertising. They call it “text to template,” in contrast to “text to image” with Firefly 2.

Adobe’s push to the mainstream

Since the launch of DALL-E 2 in April 2022, image synthesis models have improved in quality rapidly. Adobe’s additions come when competitors such as Stability AI, Midjourney, Microsoft, OpenAI, and Meta have continued to update their image synthesis tools. If you’ve followed our AI coverage, you’re likely familiar with the general controversy over AI-generated artwork due to unresolved issues over copyright and worries about replacing human artists. Adobe has not magically waved those issues away completely, but it hopes to keep moving image synthesis technology into the mainstream.

Adobe’s design leader, Scott Belsky, goes into detail in a blog post about new policies being developed to “prohibit abuse, discourage bad actors, and encourage transparency and attribution,” including training its AI models on licensed and public domain content and applying content moderation to the training data set.

The company has also introduced features like Content Credentials to mark AI-generated content using metadata, and it is advocating for legislative measures such as the FAIR Act, which would provide legal recourse for artists whose styles are intentionally and commercially impersonated through AI tools. Even if some of those initiatives don’t gain traction, they allow Adobe to say it’s working on the issue while it continues its AI rollout.

Firefly 2 (including Generative Match) is available for testing in Adobe’s web-based beta and is expected to be integrated into Creative Cloud apps later. Meanwhile, the Firefly Vector model is available via an Adobe Illustrator update in the Creative Cloud app, and Firefly Design Model is available as a beta feature in Adobe Express.

https://arstechnica.com/?p=1974894

Over the weekend, AI-powered makeovers of famous music album covers went viral on Twitter thanks to Adobe Photoshop’s Generative Fill, an image synthesis tool that debuted in a beta version of the image editor last week. Using Generative Fill, people have been expanding the size of famous works of art, revealing larger imaginary artworks beyond the borders of the original images.

This image-expanding feat, often called “outpainting” in AI circles (and introduced with OpenAI’s DALL-E 2 last year), is possible due to an image synthesis model called Adobe Firefly, which has been trained on millions of works of art from Adobe’s stock photo catalog. When given an existing image to work with, Firefly uses what it knows about other artworks to synthesize plausible continuations of the original artwork. And when guided with text prompts that describe a specific scenario, the synthesized results can go in wild places.

For example, an expansion of Michael Jackson’s famous Thriller album rendered the rest of Jackson’s body lying on a piano. That seems reasonable, based on the context. But depending on user guidance, Generative Fill can also create more fantastic interpretations: An expansion of Katy Perry’s Teenage Dream cover art (likely guided by a text suggestion from the user) revealed Perry lying on a gigantic fluffy pink cat.

If you’d like to make your own expanded artwork, you’ll need to subscribe to Creative Cloud and download the Adobe Photoshop beta within the Creative Cloud app. Next, open an image and expand the canvas size. Select the entire original image, then inverse the selection so the outer portion of the canvas is selected. If desired, type in a description, then click “Generative Fill.” Photoshop will create imagery to fill the space around the original image, and you can flip between various possibilities.

Album covers aren’t the only images that have been expanded recently to audience-amusing effect on social media. Aside from obvious expansions of famous paintings like the Mona Lisa, a tweet that expanded several famous meme images garnered almost 10,000 likes on Tuesday.

You can see more expanded album artwork in threads by Dobrokotov and Lorenzo Green on Twitter.

For various understandable reasons, including technical deficiencies in the results, sanctity of art objections, and ethical issues related to utilizing AI-generated artwork, not everyone has been amused or impressed by the results. Browsing quote tweets, it’s fair to say that response to the image expansion trend has been mixed between delighted and disgusted. Regardless, Generative Fill has been a visual toy for social media users that keeps on giving.

https://arstechnica.com/?p=1943319

On Tuesday, Adobe added a new tool to its Photoshop beta called “Generative Fill,” which uses cloud-based image synthesis to fill selected areas of an image with new AI-generated content based on a text description. Powered by Adobe Firefly, Generative Fill works similarly to a technique called “inpainting” used in DALL-E and Stable Diffusion releases since last year.

At the core of Generative Fill is Adobe Firefly, which is Adobe’s custom image-synthesis model. As a deep learning AI model, Firefly has been trained on millions of images in Adobe’s stock library to associate certain imagery with text descriptions of them. Now part of Photoshop, people can type in what they want to see (i.e., “a clown on a computer monitor”), and Firefly will synthesize several options for the user to choose from. Generative Fill uses a well-known AI technique called “inpainting” to create a context-aware generation that can seamlessly blend synthesized imagery into an existing image.

To use Generative Fill, users select an area of an existing image they want to modify. After selecting it, a “Contextual Task Bar” pops up that allows users to type in a description of what they want to see generated in the selected area. Photoshop sends this data to Adobe’s servers for processing, then returns results in the app. After generating, the user has the option to select between several options of generations or to create more options to browse through.

When used, the Generative Fill tool creates a new “Generative Layer,” allowing for non-destructive alterations of image content, such as additions, extensions, or removals, driven by these text prompts. It automatically adjusts to the perspective, lighting, and style of the selected image.

Generative Fill isn’t the only AI-powered feature added to the Photoshop beta. Firefly has also enabled Photoshop to remove parts of an image entirely, erase objects from a scene, or extend the dimensions of an image by generating new content that surrounds the existing image, an AI technique known as “outpainting.”

These features have been available in OpenAI’s DALL-E 2 image generator and editor since August of last year (and in various homebrew releases of Stable Diffusion since around the same time), so Adobe is just now catching up to adding them to its flagship image editor. Admittedly, it’s a fast turnaround for a company that might have a huge liability target painted on its back when it comes to issues of harmful or socially stigmatized content generation, utilizing artists’ images for training data, and powering propaganda or disinformation.

Along those lines, Adobe not only blocks certain copyrighted, violent, and sexual keywords from generating but also relies on terms of use that restrict people from generating “abusive, illegal, or confidential” content.

Also, with Generative Fill’s ability to easily warp the apparent media reality of a photo (admittedly, something Photoshop has been doing since its inception), Adobe is doubling down on its Content Authenticity Initiative, which uses Content Credentials to add metadata to generated files that help track their provenance.

Generative Fill in the Photoshop beta app is currently available to all Creative Cloud members with a Photoshop subscription or trial through the “Beta apps” section of the Creative Cloud app. It is not yet available for commercial use, not accessible to individuals under 18, not available in China, and currently supports English-only text prompts. Adobe plans to make Generative Fill available to all Photoshop customers by the end of the year so that anyone can make yard clowns with ease.

If you aren’t a Creative Cloud subscriber, you can also try Generative Fill for free on the Adobe Firefly website with an Adobe login through a web-based tool. Adobe recently removed the waitlist from its Firefly beta.

https://arstechnica.com/?p=1941261

On Monday, Adobe announced plans to add AI video synthesis tools to Premiere Pro and After Effects, reports Reuters. This follows an announcement last week that AI-powered text-based video editing will come to Premiere Pro in May.

Adobe’s generative video effects will come courtesy of Adobe Firefly. Firefly, an image synthesis mode we previously covered in March, will expand to include video and audio generation tools. Premiere Pro is Adobe’s flagship video editing application, a popular industry tool for assembling TV shows and Hollywood films.

Currently, performing certain special effects or editing tasks can often be labor-intensive. Adobe aims to streamline that process with generative AI assistance. In a demo reel posted on its blog, Adobe showed “Firefly for Video” generating a background soundtrack, generating sound effects, changing the visual season in a video, creating a visual effect on the word “molten,” locating and brightening a person’s face, and generating audio transcripts via text commands.

In its promotional blog, Adobe mentions features we might expect from its new AI tools, including some listed above (like generating sound effects and music), but also text-based color enhancements, text and logo animations, automatic storyboard creation, and creative “co-pilots” that can supposedly generate how-to’s to help users master editing skills.

If these kinds of text-based video editing tools seem familiar, it’s because Runway teased similar capabilities in September, although Runway has yet to deliver on those promises with the fidelity previewed in its teaser. Similarly, it’s probably wise to take Adobe’s marketing video with a grain of salt until the product is released.

(And only time will tell if Adobe Firefly will ever be able to generate a video of Will Smith eating spaghetti that can rival the masterpiece devised by ModelScope, an open source video synthesis model.)

In a separate but related announcement last week, Adobe announced it is integrating text-based video editing into Adobe Premiere Pro, which will allow editors to trim and re-arrange video based on automatically detected transcripts of speech pulled from video clips. Here’s how Adobe describes the feature:

Premiere Pro is the only professional editing software to incorporate Text-Based Editing, revolutionizing the way creators approach their craft by making video editing as simple as copying and pasting text. Powered by Adobe Sensei, Text-Based-Editing automatically analyzes and transcribes clips so editors, producers and assistants can simply copy and paste sentences into any order they want and instantly see them appear on the timeline. Transcripts can be searched in the transcript window to identify exact words and phrases.

Adobe initially previewed a similar text-based video editing capability as part of “Project Blink” in October, although now Adobe says these features come from its “Adobe Sensei” framework. Whether the difference is technical or merely branding is unclear. It’s worth noting that a company called Descript offers a similar feature for video and podcast editing.

Adobe says its Firefly for Video features will begin rolling out “later this year” while also allowing people to sign up to try Firefly, which is currently in closed beta testing. Meanwhile, Adobe promises that text-based video editing is coming to Adobe Premiere Pro in May.

https://arstechnica.com/?p=1932354

On Tuesday, Adobe unveiled Firefly, its new AI image synthesis generator. Unlike other AI art models such as Stable Diffusion and DALL-E, Adobe says its Firefly engine, which can generate new images from text descriptions, has been trained solely on legal and ethical sources, making its output clear for use by commercial artists. It will be integrated directly into Creative Cloud, but for now, it is only available as a beta.

Since the mainstream debut of image synthesis models last year, the field has been fraught with issues around ethics and copyright. For example, the AI art generator called Stable Diffusion gained its ability to generate images from text descriptions after researchers trained an AI model to analyze hundreds of millions of images scraped from the Internet. Many (probably most) of those images were copyrighted and obtained without the consent of their rights holders, which led to lawsuits and protests from artists.

To avoid those legal and ethical issues, Adobe created an AI art generator trained solely on Adobe Stock images, openly licensed content, and public domain content, ensuring the generated content is safe for commercial use. Adobe goes into more detail in its news release:

Adobe Firefly will be made up of multiple models, tailored to serve customers with a wide array of skillsets and technical backgrounds, working across a variety of different use cases. Adobe’s first model, trained on Adobe Stock images, openly licensed content and public domain content where copyright has expired, will focus on images and text effects and is designed to generate content safe for commercial use. Adobe Stock’s hundreds of millions of professional-grade, licensed images are among the highest quality in the market and help ensure Adobe Firefly won’t generate content based on other people’s or brands’ IP. Future Adobe Firefly models will leverage a variety of assets, technology and training data from Adobe and others. As other models are implemented, Adobe will continue to prioritize countering potential harmful bias.

In addition to its commitment to a more ethical form of AI generator, Adobe is doubling down on ethics with a “Do Not Train” tag for creators who do not want their content used in model training. According to Adobe, this tag will “remain associated with content wherever it is used, published, or stored.”

In terms of performance, Adobe Firefly looks similar to OpenAI’s DALL-E, although Ars has not been able to evaluate its performance for itself yet. Demos on the Firefly website show various capabilities, including “text to image” (creating unique images from a text description), “text effects” (applying styles or textures with a text prompt), and “recolor vectors” (creating unique variations of a work from a text description).

Whether artists will adopt Firefly into their workflows remains to be seen, but the new AI model seems, on its face, like a positive step forward for AI ethics advocates. As always, we’ll need to take Adobe’s claims with a grain of salt, and we’ll keep you updated as new details emerge.

https://arstechnica.com/?p=1925844