Neural Style Transfer: Fusing Picasso with Your Selfies!

Three panels side by side: a photograph of a lone cypress tree at dusk, an abstract oil painting of swirling impasto brushwork, and the result, which keeps the tree and the composition but renders them in the painting's brushstrokes and palette.

Written October 2023 and rewritten in 2026. The original explained style transfer in metaphors and linked a Colab. This version has the actual objective function in it, and the run above (600 steps, 59 seconds on a laptop) is from the code below.

Neural style transfer takes a photograph and a painting and produces the photograph as though it had been painted. It is from 2015, it has been thoroughly superseded for practical purposes, and it is still the single best thing to implement if you want to understand what the layers of a convolutional network actually contain.

The reason is that it inverts the usual relationship. Nothing is trained. The network is frozen from start to finish. The thing being optimised is the image.

The one weird idea

Normally: fix the data, compute a loss, adjust the weights. Here: fix the weights, compute a loss, adjust the pixels.

img = content.clone().requires_grad_(True)     # the *image* is the parameter
opt = torch.optim.Adam([img], lr=0.02)         # optimiser over pixels

for step in range(STEPS):
    feats = net(img)                           # net is frozen throughout
    loss  = CONTENT_W * content_loss(feats)  \
          + STYLE_W   * style_loss(feats)    \
          + TV_W      * total_variation(img)
    opt.zero_grad(); loss.backward(); opt.step()
    img.clamp_(0, 1)

That is the entire algorithm. Everything else is the definition of the two losses, and those definitions are where the interesting content is.

Content: match the activations

Content loss is squared error between the generated image’s activations at one deep layer and the content image’s activations at the same layer.

ℒ_{content} = \frac{1}{2} \sum_{i j} {(F_{i j}^{l} (X) - F_{i j}^{l} (P))}^{2}

X is the image being optimised, P the content image, F the activations at one deep layer.

CONTENT_LAYERS = {21: "conv4_2"}       # one layer, deep in the network

c_loss = F.mse_loss(feats[21], content_targets[21])

Depth is the whole design decision. Early layers respond to edges and colours, so matching them pins the image to the original almost pixel-for-pixel and leaves no room to restyle. Deep layers respond to arrangements of parts, so matching them says “a tall dark object here, horizon there” and is indifferent to how it is rendered.

That indifference is the space the style gets to fill.

Style: match the correlations, throw away the positions

The elegant part. Style is represented by the Gram matrix: every feature channel’s correlation with every other channel, averaged over all spatial positions.

G_{i j}^{l} = \frac{1}{C H W} \sum_{k} F_{i k}^{l} F_{j k}^{l}

i and j index feature channels; k runs over every spatial position. The sum over k is what erases location.

def gram(f):
    b, c, h, w = f.shape
    m = f.view(c, h * w)
    return (m @ m.t()) / (c * h * w)

Averaging over h * w is not a normalisation detail. It deletes where anything happened and keeps only what tends to occur together. “Curved strokes co-occur with thick impasto and cobalt” survives; “there is a swirl in the upper left” does not.

That is a workable definition of texture, and it explains the behaviour you see in the result: the brushwork transfers everywhere and the composition does not move.

Style loss is computed at five layers rather than one, which captures texture at several scales: fine grain from the early layers, larger gestural structure from the deep ones.

STYLE_LAYERS = {0: "conv1_1", 5: "conv2_1", 10: "conv3_1",
                19: "conv4_1", 28: "conv5_1"}

s_loss = sum(F.mse_loss(gram(feats[i]), style_targets[i])
             for i in STYLE_LAYERS)

The two losses are enemies

Style loss falls steeply from over 100 to about 5 by step 500. Content loss rises from zero to about 6. Both spike sharply near step 550 before partly recovering.

I initialise the image as the content photograph rather than as noise, which makes the tension unusually easy to see. Content loss starts at exactly zero (the image is the content image) and goes up. Style loss starts at 805 and comes down.

There is no setting at which both are minimised. Every step that adds brushwork costs fidelity to the photograph, and the weights (STYLE_W = 1e6 against CONTENT_W = 1) are you declaring where on that trade-off you want to sit. Those wildly different magnitudes are not a hack: Gram matrices of normalised features are small numbers, and the weight is mostly correcting for scale.

Note the spike around step 550, in both panels at once. That is Adam at lr=0.02 becoming unstable near convergence, not a property of the algorithm. It is also why the classical implementations use L-BFGS, and why my final numbers (style 9.7, content 5.8) are slightly worse than they were at step 500 (5.1 and 4.7). I left the run as it came out rather than cherry-picking the best step.

What became of it

This algorithm is slow: it optimises from scratch per image, which is the 59 seconds above for one 512×512 result. Two things replaced it.

Feed-forward style transfer (Johnson et al., 2016) trains a network per style so that inference is a single forward pass. This is what shipped in phone apps.
Diffusion models made the whole framing obsolete for end users. You no longer supply a style image; you describe the style, and the model has learned the association from captions. Better results, and none of the mechanism is visible any more.

Which is exactly why it is still worth implementing. In the diffusion version, “style” is a direction in a learned space you cannot inspect. Here it is a Gram matrix, and you can look at it, break it, and watch what happens to the picture. There are not many places left where a fundamental idea in this field is that legible.

Notes on this run

Both input images were generated for this post rather than taken from a collection, so nothing here reproduces a specific artist’s work. The style image is an original abstract painting of brushwork and colour; no artist’s signature style is being copied, and none is named in the output.
512×512, 600 Adam steps, VGG19 features, 59 seconds on an M4 Pro through MPS. Resolution costs quadratically: the same run at 1024×1024 is roughly four times the work.
The total-variation term (TV_W = 1e-4) suppresses high-frequency speckle. Set it to zero and the result gets noticeably noisier; set it too high and the image turns to smooth mush.

References

A Neural Algorithm of Artistic Style. Gatys, Ecker and Bethge, 2015. The original, and unusually readable.
Perceptual Losses for Real-Time Style Transfer and Super-Resolution. Johnson et al., 2016. One forward pass instead of an optimisation.