CREATIVE ALGORITHMS

How AI and software are already making us more creative

robot writer

The end of the creative world is coming and it will be automatically ghost written by a neural network. Writers automated out of existence by the collected works of Octavia Butler and Hilary Mantel, songwriters out-classed by the library of The Beatles and Robert Johnson songs, furniture designers left to work and rework the already perfectly excellent designs of the Shakers!

Seem a little far-fetched? The tech is increasingly there, but will a technology powered by what has already been done really supplant the creative instinct to make something new? Is it really a replacement for the messy confusing inexplicable process of creation? Or is this just another version of something that has been happening for ages?

Can’t an algorithm make us more creative? Can we develop a literacy or an algo-dexterousness that allows us to harness these software tools in our creative process as a kind of amplifier of our own oddball creative feedback loop?

Surely, when used lazily (as they are, and will only be more-so in the future), they will produce boring stuff, or sometimes a high-verisimilitude content that is convincing enough, weird, and sometimes even deeply compelling. But I think we have already been creatively using AI tools for ages and when we accept that these tools can magnify creativity, I think we can shift into another gear.

For years, I would ask designers how they might be comfortable using AI-enabled design tools, and for the most part, they would express distaste, a sense that it was cheating, or that it couldn’t do anything but copy something else that had already been done. But there are a million tools that designers already use that are algorithmic, if not artificially intelligent, designed to let the software do some of the mundane work for you. Do you wanna draw each individual line of a gradient? Or perhaps you want to do it the previous way, of dropping a series of drops of colored ink on a platen of a letterpress machine or silk screen to create the effect every time you want to spread that fade. Nah, just use the gradient tool in illustrator that lets you not only choose any color, but where you want that color to start fading, and you can adjust it on the fly. That doesn’t limit your creativity, it expands it.

Neural Networks, Algorithms, AI Models, Machine Learning, Computer Vision; these are all really just ways of training computers to work similarly to our own physiology. We don’t totally know how our own brains work but they have given us some good ideas for how to start thinking about different ways of building calculators. In reality, in most of the cases listed above we have diverged from literally trying to create a facsimile of our own wet mental models but have used our own brains as a jumping off point.

Fundamentally all of these things are a kind of pattern matching system trained on a set of fixed examples, and at scale the system starts to see the familiar repeating elements or ways of doing things that it can say–on average–this is how to solve this problem. Whether it’s drawing a picture, writing a sentence, creating audio or sounds, it works to find what feels “right”, or can even work to find the opposite of “right”.

The fear a lot of folks have of these software-enabled thinking-tools, is rooted in seeing them as replacements for our own brains; an oracle that can come up with an answer to any question. But I don't think that is the right way to approach these tools with a set it and forget it mentality. These tools can be a great way to explode and reassemble the limitations that you are used to.

Yes, creativity is often unleashed by limitations, and sometimes working in the old ways gives you interesting challenges to bounce off of; trying to do good typography with rubylith and a silkscreen will train you to really sweat the details, and using a Japanese hand plane will make you do woodworking in a totally different way than if you just use a table saw and a power planer. But a woodworker is no more limited by the pull stroke of his Kanna plane than a writer that uses software tools.


I tend to break software enabled creativity into a few categories:

  1. Grunt Work - reducing the effort of things that are well understood but labor intensive

  2. Breakthrough Moments - completing a thought, expanding on the idea you have but just can’t reach

  3. Oblique Strategy - finding new ways to approach a problem beyond what occurs to you naturally

  4. Validative Perspective - identifying what will be considered “right” to a different set of people than yourself

The example of the gradient tool in illustrator really fits into the grunt work category here as it is a time consuming process that has an expected outcome that you really wish you could just say, “do it this way and don’t ask me any questions”. Perspective drawing tools, pitch shift tools in audio, arpeggiators, letterspacing, transparency, and excel formulas are all examples of grunt work tools that we already use.

There are so many ways we can expand these grunt work tools that don’t get in the way of our creativity, like CAD programs that automatically place the electrical receptacles in a building and identify conflicts, grid enabled design tools that understand common familiar grid systems, and layouts, video editing tools that know what works on Instagram and preps your assets appropriately.

Robin Sloan’s RNN-Writer tool that provides inline suggestions as you type based on different training libraries.

Robin Sloan turned me onto the potential of the breakthrough moments category some years back with his RNN-Writer tool. A word-processor that was trained on a large particular library of data, it helps you write a story in the style of the library it has been trained on. To be honest, his tool can be kind of used in all 4 of the above ways, but for me the first thing that clicked was the initially straightforward idea of expanding your rough idea into something more finished. Start typing and it completes your idea in the style it was trained on. This might be a whole sentence, or it might just be a better word within the sentence than you might have chosen. These breakthrough strategies can act as writers block tools, they can act as a kind of speed magnifier helping you get an idea out more quickly, they can show possible paths you might take as you type to encourage and spur you on. There are tons of examples of this already in the real world none more wide spread than the live spelling check and autocompletion that we use in word processing and texting. You might find it annoying sometimes, but try turning it off and seeing how much you’ve come to rely on it.

Examples of logos not done in the 21st century
Examples of logos not done in the 21st century

I remember a few years back, a tweet going around that the logos of Airbnb, Medium, Flipboard and others were similar if not identical to some previous work from the 70s. Whatever, there is not much new in the world and lots of ideas happen multiple times, but it occurred to me that a designer is only as good as their own library of references. Imagine being able to leverage all of the designs done in a style quickly while you are designing.

Usually the way this process works is that you tell a designer what feel you are going for, and you both pick a collection of examples that you like or that “feel right" in a moodboard, and then the designer goes about trying a lot of things that feel related to the examples, but are not the examples.

But a breakthrough approach would be to train a tool on that moodboard set and then let the tool go out and find other similar examples to further train itself. But rather than then having the model just create a facsimile of those examples in some new way, this model can be built into a drawing tool that that you actually pilot, showing possible paths as you draw. As you create a curve, if the “feeling model” you have trained knows that tight radii are out of the feel, it suggests a more “right feeling” radius to your curve. If the color palate is bright and you’re working with too muted a palate, it can suggest complementary colors. Of course having a human in this process means that breaking the rules is as much a part of the process as software recommendations, but having that context to know when to deviate is the magic.

Sketch RNN
The Sketch RNN demo that augments your drawing with an augmented intelligence.

There are tons of these drawing tools that are kinda jokey fun things, like draw an outline of a cat and it will create a photorealistic terror based on your drawing, or this one that is quite close to the above-mentioned radius detecting example where you start drawing a cat or a pineapple or whatever, and as you go it recommends other features based on what you have drawn so far.

For a while I’ve done a sort of manual library training with what I have called algorithm blending, which I use as another form of breakthrough strategy. I make a lot of playlists on Spotify, largely in the mode of what record collectors would call crate-digging, seeking out oddball choices and trying to find obscure stuff. Spotify is well known for their algorithmic playlist tools like Discover Weekly and their recommendation tools that suggest more songs for a given playlist based on what others are doing. These recommendations are based on what you add to playlists and what you favorite, so I am very careful of what I add to playlists lest I confuse the algorithm into making suggestions based on too wide a set of options. Sometimes, when I find another curator and playlist I like, I will carefully choose songs from that list to add to my lists. It's like cross-training my network by associating with others, and therefore algo-blending; like mixing wines together to create something better.

The oblique strategy is about finding new ways to solve problems that might not have occurred to you naturally. Sometimes these are tools that help you just do the opposite of what you might have done by default, or to find things that are radically different but still “feel right” by the rules of the system. The most famous version of this is Brian Eno’s “Oblique Strategies” deck which is a series of cards that ask intentionally provocative questions to come at a problem from a different angle. This isn’t software exactly but it just as easily could be, injecting a functionally absurd angle into a standard decision making model.

In design consultancy, I often think of this as another area of library-richness, if you can bring your knowledge of something seemingly unrelated to a problem and make a connection, you might find a new way to solve a challenge. Can awareness of script writing tools inform your presentation technique? Can a knowledge of automotive design affect your song writing practice? Who knows, but the breadth of your library and the intentional cross referencing of things allows you to see things differently.

Imagine the autocomplete drawing tool or sentence tool mentioned above, but that does the opposite, that makes it hard to do the obvious thing, that pushes you away from the familiar shapes and sounds. Or a drawing tool for mobile apps, trained on a model of Swiss poster design. Something that helps you reframe the question or change your starting position so that you head down a new or different path that you might naturally chose.

their.tube
Their.tube shows what different peoples YouTube feeds look like other than your own. This shows a climate change deniers recommendations.

It is really hard to create something for others, to get outside of your own head. Validative Perspective helps us get past our own biases and see what others might appreciate. If we are the product of our own library and our own set of familiar creative pathways, then these tools act as a kind of psychedelic experience showing us the world through another set of eyes. I’ve spent a lot of time recently with the TheirTube app that makes the filter bubble around each person's YouTube recs visible and allows you to see other types of people's YouTube experiences: the Climate Change Denier, The Prepper, The Frutitarian, and others. This is a pretty passive example but you can imagine using these kinds of tools to validate what you are working against a particular set of people or even a particular person.

Crystal email helps you tailor emails and communications to particular people based on what has worked on them in the past. It mines linkedIn, other emails, and network insights to help you craft something that is “perfectly” targeted to that individual. I’m sure it has its challenges, but you can see how this only gets more refined over time.

I’ve worked with tools that degrade your vision to simulate visual acuity for an interface, that have shown what your design will look like on tiny screens, even prosthetics that show you what it feels like to have degraded muscular ability for lifting and climbing stairs, so why not an algorithmic version of this that shows you what people might think of the sound, or the words that resonate best, or identify when they are most likely to be receptive to your thing.

Of course this is where a lot of creative people cringe the most, this hyper targeting of everything, just showing people what they think they want or what they have wanted in the past. But the key isn’t to use this exclusively, or in the absence of your own taste, but as context to creating things. Anyone that tells you that they can create something for people with pure intuition and no understanding of that person or their interests is lying to you. Every one has context whether they know it or not. You don't have to give people what they want, you might push people past their comfort zone, but the art is knowing where the edge is to push to, and that is the real skill here, informed intransigence.


We tend to think of these algorithms as a fundamentally new and technological thing, but really they’ve been around for as long as we have. An algorithm is really just a template, a pattern for how to approach a problem. Every tool has a particular set of uses, with unique quirks and qualities. For digital tools, these quirks are often defined by their traditional physical analogues and those qualities and limitations are what makes us think they feel right or not. Most of what we think of as good typography comes from traditional restrictions of lead type and printing presses, these tools we no longer use continue to shape our perception of what feels right because it is what we are used to.

Understanding what feels right, and knowing how to play with where the edge of right is, is the soul of the creative craft. Whether you are making something using traditional or modern tools, the skill is not just about wielding the tool but what you say and make with it.

One of the things I most vehemently reject in regards to algo-fear is the idea that these tools will make you less creative. We all use these tools already, we have used some version of them for our entire careers, and our predecessors have used versions of these tools. As a creator you are only as good as your reference library, whether you like it or not. So find ways to expand and capitalize on what you know and don’t know to be a better creator. Don’t use it as a crutch, and don’t be a mindless robot-maintenance person, use it as leverage, let it be a bicycle, amplifying your skill and creativity, and find ways to shift your creative process into the next gear.