Home IOS Development The way to Use Secure Diffusion to Generate Pictures with Swift CLI

The way to Use Secure Diffusion to Generate Pictures with Swift CLI

The way to Use Secure Diffusion to Generate Pictures with Swift CLI


Developed by Stability AI in collaboration with educational researchers and non-profit organizations, Secure Diffusion is a deep studying mannequin that generates detailed pictures based mostly on textual content descriptions or what so referred to as textual content prompts. Moreover, it will also be utilized for duties resembling inpainting, outpainting, and producing image-to-image translations guided by textual content prompts. Since its public debut in August 2022, it has gained reputation amongst artists, builders, and hobbyists.

Secure Diffusion is just not the one text-to-image mannequin. You could have heard of Midjourney or DALL-E 2, that are additionally succesful for producing nice wanting AI pictures from a textual content immediate. Nevertheless, one notable side of Secure Diffusion is that its supply code and mannequin weights have been made publicly accessible. What this implies is that you would be able to set up Secure Diffusion in your Mac or PC and run it domestically. Comparably, the opposite text-to-image fashions can solely be accessed through some sorts of cloud companies.

Secure Diffusion with Core ML

In case you are an iOS developer wanting so as to add text-to-image options or different inventive options to your app, Secure Diffusion could also be simply what you want. Because of optimizations launched by Apple’s ML engineers for Core ML, Secure Diffusion can now be used on Apple Silicon units working macOS 13.1 or later and iOS 16.2 or later. With these updates, builders can simply incorporate Secure Diffusion into their apps, and most significantly, the mannequin is saved domestically on the machine, so customers don’t want an web connection to make use of the AI picture era characteristic.

The discharge of Core ML Secure Diffusion features a Python package deal that enables builders to transform Secure Diffusion fashions from PyTorch to Core ML utilizing diffusers and coremltools, in addition to a Swift package deal for deploying the fashions. To make use of the text-to-image characteristic, you should use the Swift package deal in your Xcode initiatives. Nevertheless, on this tutorial, I’ll concentrate on offering you a quick overview of Secure Diffusion and clarify the way to use it with Swift Command Line (CLI). We’ll have one other full tutorial on constructing apps with Secure Diffusion.

To observe this tutorial, it’s necessary to make sure that you’ve gotten Xcode 14.3 put in and that the command line instruments for Xcode are additionally put in. In the event you downloaded Xcode from the Mac App Retailer, the command line instrument ought to already be included.

Hugging Face – The Hub of Pre-trained ML Fashions

In the event you’re new to machine studying, you might not have heard of Hugging Face. Nevertheless it’s time to test it out as Hugging Face is a worthwhile useful resource for anybody excited by machine studying and it offers a wealth of pre-trained fashions and instruments that may be simply built-in into a wide range of functions.

Hugging Face is a machine studying platform based in 2016 to democratize NLP by providing fast entry to over 20,000 pre-trained fashions. It offers Knowledge Scientists, AI practitioners, and Engineers with pre-trained fashions that can be utilized for a wide range of duties, together with textual content processing in additional than 100 languages, speech recognition, object detection, picture classification, and regression and classification issues with tabular knowledge.


Head over Hugging Face and seek for “Secure Diffusion”. You’ll discover numerous variations of Secure Diffusion which might be accessible for obtain for free of charge together with Secure-Diffusion-v1-4, Secure-Diffusion-v1-5, Secure-Diffusion-2, and Secure-Diffusion-2-1.

Operating Secure Diffusion with Core ML

To run Secure Diffusion on Apple Silicon units with Core ML, you may convert the fashions by your self utilizing python_coreml_stable_diffusion, which is a Python package deal for changing PyTorch fashions to Core ML format. Alternatively, you should use the pre-built Core ML fashions, ready by Apple. For comfort, we’ll use the pre-built Core ML fashions (e.g. coreml-stable-diffusion-2-1-base) on this tutorial.


All these fashions are hosted on Git repositories. Since these fashions are usually massive in measurement, it’s mandatory to put in the git LFS (Massive File Storage) extension with a view to obtain the mannequin recordsdata.

First, it’s good to set up Homebrew, which is rather like a package deal supervisor, in the event you haven’t executed so. To put in Homebrew in your Mac, open Terminal and key within the following command:

Subsequent, kind the next command to put in Git LFS:


As soon as the set up completes, execute the next command to initialize Git LFS:

Lastly, run the next command to clone and obtain the Secure Diffusion repository. We use the most recent model of Secure Diffusion (i.e. v2.1) on this tutorial. For older variations, you could find them right here.

It’ll take a while to obtain the mannequin recordsdata as they’re fairly massive in measurement. When you full the obtain, it’s best to discover the mannequin recordsdata beneath each authentic and split_einsum folders.


Every Core ML Secure Diffusion mannequin comes with a number of variants which might be optimized for various hardwares. For Core ML Secure Diffusion fashions, there are two consideration variants accessible: authentic and split_einsum. The authentic consideration is just appropriate with CPU and GPU, whereas split_einsum is optimized for Apple’s Neural Engine units, that are current in trendy iPhones, iPads, and M1/M2 Macs.

Along with consideration variants, there are two subfolders accessible beneath every variant: compiled and packages. The packages subfolder is appropriate for Python inference, permitting for testing of transformed Core ML fashions earlier than trying to combine them into native apps. Alternatively, the compiled subfolder is what’s required for Swift apps. The compiled fashions cut up the massive UNet mannequin weights into a number of recordsdata, enabling compatibility with iOS and iPadOS units.

Utilizing Core ML Secure Diffusion to Generate Pictures

After finishing the mannequin file preparations, the subsequent step in incorporating Secure Diffusion into your app is to acquire the Core ML Secure Diffusion package deal, accessible at https://github.com/apple/ml-stable-diffusion. This complete package deal comprises a Swift package deal that may be simply built-in into your Xcode mission, enabling you to leverage the capabilities of Secure Diffusion in your app. Moreover, the package deal features a pattern software that showcases the ability of Secure Diffusion by producing pictures with Swift CLI.

To obtain the package deal, key within the following command in Terminal:

As soon as the package deal is prepared in your Mac, you should use the built-in app named StableDiffusionSample with the syntax under:

The --resource-path possibility is so that you can specify the trail of the mannequin variant. On this case, it’s the compiled folder of the split_einsum variant. Here’s a pattern command for producing an inside design picture:

Relying on the processing energy of your machine, it might take 10 seconds or a number of minutes for Secure Diffusion to generate the picture. When completed, the AI-generated might be discovered within the output folder.


You possibly can change the textual content immediate to generate different pictures. This web site provides an in-depth walkthrough if you wish to learn to craft good prompts to create some gorgeous AI-generated pictures.


Secure Diffusion is a robust deep studying mannequin that permits customers to generate detailed pictures based mostly on textual content prompts. With the discharge of Secure Diffusion with Core ML, iOS builders can now simply incorporate the text-to-image characteristic into their apps, and most significantly, the mannequin is saved domestically on the machine.

By following this tutorial, it’s best to know the way to get began with Secure Diffusion and use the instruments offered by Apple to run Secure Diffusion with Swift CLI. Within the subsequent tutorial, let’s see the way to combine the mannequin and construct an iOS app for picture era.



Please enter your comment!
Please enter your name here