The most interesting (to me) aspect of Alexander's post is that 5/11,000 people scored 49/50 (98%).
That could happen randomly if all people had 82% skill, which was clearly not the case.
It could also happen if a dozen people had 98% skill, except then someone would almost certainly would have gotten a perfect score by chance.
The most likely answer is that 100-200 people had >90% skill, and everyone else had lesser skill, or simply rushed so they could see the answers
I suspect the challenge would have been easier without human-operated computer-generated graphic art mixed in; most of the pictures had tells, but you had to be familiar with the genre.
The Riverside Cafe (extra chair legs) and Anime Girl in Black (excessively detailed armpits) were two of the most obvious AI examples. In the case of the Gauguin, the road leads to the village. In the AI version, there is a narrow footpath to nowhere and a random smattering of isolated houses.
Art isn't just about rendering something, it is about leading the eye on an emotional journey by drawing the eye toward significant details or symbols, however abstract.
Thomas Hoving once had a statement to the effect that it's hard to identify a forgery done in your own era, but much easier to identify one done decades ago. The principle could well apply to some of the comparisons in this post.
One problem is that even if AI art could surpass the last 10% of painting masterpieces, it would be an uphill battle for the works to be recognized and appreciated.
The good thing about music is that unlike painting it is still a vital form today. So we can see how AI will transform it. So far, we’re just at the beginning. But, audio is a much harder domain for AI given the difficulty of getting training data.
Regarding distinguishing vs generating good art. In AI there are discriminative models (which for example take an image and output how good the image is) and generative models (which create good images). Generally, generative modeling is harder!
On this note: it would not be hard to train an AI model to distinguish AI and human art to high accuracy given some training data. On the other hand, I suspect if humans were given a “training dataset” for Scott Alexander’s problems, they would have done a ton better too.
It's time to pull out the difficult but highly rewarding first novel of William Gaddis, "The Recognitions" and give it another read through. Gaddis interestingly addresses some of Scott's points in a novel that was written 70 years ago when AI was not a thing at all other than in some science fiction novels.
I would also posit a second idea, that comparing images on a computer screen is flawed. AI, at present, only gives us an image that is devoid of chiascuro and brush strokes. Perhaps I might be missing something not having read Scott Alexander's post.
I'm not trying to contradict anything you've said, but the historical origins of AI are really neat. Three years before "The Recognitions" was published (never heard of it before now), Claude Shannon designed a simple AI named Theseus: https://www.youtube.com/watch?v=_9_AEVQ_p74
"As an analogy, if in 2025 someone writes a book on evolution that is more accurate than The Origin of the Species, no one would argue that this makes the author a better scientist than Charles Darwin."
Are the Olympic record setting runners from the 1920s better than the runners than the amateurs running today who are faster than them?
I don't think they are better, but it sort of depends how you define "better". In my view, people like Wallace and Darwin were greater scientists than the typical modern biologist, because they were more innovative. But I don't think they are better than modern scientists in the sense of knowing more about biology. There's a difference between greatness in the sense of being more innovative, and greatness in the sense of knowing more. Our society puts a premium on innovation, when judging scientists, artists, etc.
I'm saying that society puts a premium on innovation when judging greatness. If some engineer at GE produces a light bulb that is better than Edison's bulb, and sells more of them than Edison did, it doesn't change the fact that the public views Edison as a hero for inventing (co-inventing?) the light bulb, and the GE engineer as someone who merely fine-tuned a previous invention.
Popularity doesn't equate to reputation. Taylor Swift is more popular than Mozart, but Mozart's reputation is higher.
McDonalds is more popular than a Michelin-starred French restaurant, but the latter has a better reputation.
But we should be able to analytically disentangle artistic praiseworthiness from artistic quality. You should enjoy the use of the better light bulb more than the vintage Edison light bulb. The paintings that are indistinguishable from Rembrants are just as good as Rembrants.
> What are the odds that they’d randomly pick impressionism from among all the other equally great styles, if they were not basing their taste on which style is the prettiest
This confuses me. What could one possibly base taste on *other* than what one perceives as the most attractive? Surely the whole point of art is to be pretty?
You talk at a couple of points as though the "goodness" of a work of art was some objective quality, independant of whether people in general like it. This makes no sense to me either- art is good if people like it, not otherwise.
No, I do not believe in the idea of objective quality. I am offering my opinion. Perhaps I could cite an example, to make my point. Consider a painting like Titian's The Flaying of Marsyas:
I don't think this painting is pretty, and I'd guess that most people would prefer a kitschy painting by Thomas Kinkade. But who would deny that this is one of the great works of Western art? After 450 years, it is still viewed as a masterpiece. Where will Kinkade be after 450 years?
"but I’m somewhat better than an AI (for now) at verifying whether one Rembrandt painting is better than another. I presume that is telling us something important about the type of intelligence that each of us possess, but I’m not quite sure what."
I think at least part of the answer to what intelligence we possess is that we have learned the myriad and conflicting rules of what today constitutes "great art" and many of these rules have nothing do do with the actual images themselves. Hence training an AI by exposing it to primarily the images is a very inefficient way of deducing the rules. Plus, the rules are in flux. Just consider that Van Gogh sold only one painting died in poverty.
Thanks for the post Scott. I have learned a lot from your meditations on films and paintings. I'm hardly an aficionado, but even I could tell the difference between the two 'Impressionist' works fairly easily. To me, the difference is about composition. The genuine one - which you say is a Gauguin, but from my perspective could be any great artist whose work hangs in the galleries I've visited - is telling a story: There's a person, presumably a man, walking towards a cart that has someone else next to it. Maybe they're travelling together, maybe one wants to buy goods from the other or find out some information. The scene on the left is like the type of boring novice photo that I would have taken on my youthful travels, but that even kids these days with their incredible phone cameras have learnt looks simple and dull.
I think being a fan of classic film gives you an advantage in seeing the differences. I saw Seven Samurai a short while ago and the main element that struck me was not the action scenes, or the acting, and certainly not the story, but the composition. Even scene looked like one where the director had a clear idea about showing balance and beauty to the audience. If you look at the stills for the film today, it strikes you immediately. Many modern films and TV shows with their constant use of drones and moving cameras that focus on the actors rather than 'the scene' in which the actors perform drive me insane. To me, they make whatever the show is look like a soap opera.
Good comment about The Seven Samurai. In general, classic Japanese films have very strong visuals.
"To me, they make whatever the show is look like a soap opera."
This nicely describes what I dislike about lots of American TV, even "quality TV". That visual style works best for comedy, and it seems like comedy is the strongest form of American TV.
I must object that AI is not now producing paintings, but simulated digital images of paintings. Not the same thing at all.
Take Agnes Martin, among others. They say one can't take a compelling photo of her paintings. If you produce a digital image from training on such photos what can you possibly end up with?
And Mondrian. Mondrian was an insistently improvisational painter, despite the relatively strict rails he put on his improvisation. His impact on the graphics arts is enormous, but that whole enterprise is very far removed from his intentions as a painter. My sense is that a serious Mondrian fan would think that AI would have serious difficulties producing a quality Mondrian.
But this isn't really fair to the AIs either. Maybe if we trained AI on the neural data from humans looking at the painting instead of scans of the paintings maybe it would solve these problems.
Maybe a better game is to compare original human digital art to AI digital art, except who cares? Maybe photography is fair game, or maybe it is an even deeper problem.
It's interesting to me how truly "great" many of the most famous painters (and artists) are, especially the early ones. If anyone could have painted (or made prints) like Durer, they would have been fabulously wealthy and successful. Of course, no one at the time came close. Arguably the same is true about later Rembrandt, to your point about his students. That top 10% is something of ineffable value we are far from reproducing by computer.
There's a crossover between painting and sculpture that has only been somewhat developed new technique wise. I can picture some of these techniques but, other than embossing, don't know or can't recall their names. Lots of room for continued development here, though.
As seen with face blending, people think symmetrical averages are attractive, but the faces considered most attractive deviate from these symmetrical averages. There's a lot of room to deviate and not be considered as good as an average, and probably less room to deviate and be considered better than an average. It's a huge computational task just trying to hit this average; exploring the non-average space is a much larger undertaking, if you're an AI and not a human.
The fact that it's hard to put into words what makes a face especially beautiful might relate to why it seems difficult for an A I to recognize beauty.
Getting exposed to art via art museums in the US (especially outside of New York and Chicago) is probably a genuine detriment to being able to appreciate great art. Not in an "Americans sure do have bad taste, am I right?" way, but because the best stuff mostly remained in Europe and American museums tend to get the lesser works by the greats. Visiting museums in California as a kid, I knew which artists I was supposed to appreciate but found it hard to get excited by their lesser works. Meanwhile, an hour in the Mauritshuis makes it pretty clear why the Dutch Masters are such a big deal today.
I agree. I will say that there are some good museums outside New York and Chicago, including Washington, Philadelphia, Boston, and even (to a lesser extent) places like Cleveland, Buffalo and Detroit.
But yes, 90% of the great western art is in Europe.
As for Rothko - Perhaps you know this Rajat but, a fake Rothko was at the center of an art forgery scandal where the renowned Knoedler gallery in Ney York City brokered the sale numerous fake works. I learned about this by watching the documentary "Made You Look"
Even though the test is closed I did a quick run through without scoring it and realized I was pretty uncertain about many of my answers. The exceptions were anything that was anime or sci-fi related because at this point I'm fairly certain that the majority of images in that style on the internet are AI generated. That AI is landing some of its hardest blows in the realm of mediocre sci-fi is only too appropriate. We can expect a lot of movies and TV to be generated by prompt-operators in the equivalent of sweatshops that are absolute flops with viewers. I hope Scott lives long enough to review some.
I'm genuinely curious about the impacts of AI art on the broader economy. I have yet to use it seriously in my profession (architecture) but I've always been a slow adopter of better software. When a client responds to one of my design proposals with an AI generated alternative I'll know it's time to burn my money in a wastebasket and listen to Terror through the wall.
The most interesting (to me) aspect of Alexander's post is that 5/11,000 people scored 49/50 (98%).
That could happen randomly if all people had 82% skill, which was clearly not the case.
It could also happen if a dozen people had 98% skill, except then someone would almost certainly would have gotten a perfect score by chance.
The most likely answer is that 100-200 people had >90% skill, and everyone else had lesser skill, or simply rushed so they could see the answers
I suspect the challenge would have been easier without human-operated computer-generated graphic art mixed in; most of the pictures had tells, but you had to be familiar with the genre.
The Riverside Cafe (extra chair legs) and Anime Girl in Black (excessively detailed armpits) were two of the most obvious AI examples. In the case of the Gauguin, the road leads to the village. In the AI version, there is a narrow footpath to nowhere and a random smattering of isolated houses.
Art isn't just about rendering something, it is about leading the eye on an emotional journey by drawing the eye toward significant details or symbols, however abstract.
In some cases, I was able to recognize the artist. I didn't take the test, but that might have been true of some of the high scorers.
Also of course some people might have cheated to get an especially high score.
Thomas Hoving once had a statement to the effect that it's hard to identify a forgery done in your own era, but much easier to identify one done decades ago. The principle could well apply to some of the comparisons in this post.
Good point. I made similar observation in a post I did 15 years ago:
https://www.themoneyillusion.com/the-identification-problem-vermeer-forgeries-and-ad-shocks/
One problem is that even if AI art could surpass the last 10% of painting masterpieces, it would be an uphill battle for the works to be recognized and appreciated.
The good thing about music is that unlike painting it is still a vital form today. So we can see how AI will transform it. So far, we’re just at the beginning. But, audio is a much harder domain for AI given the difficulty of getting training data.
Regarding distinguishing vs generating good art. In AI there are discriminative models (which for example take an image and output how good the image is) and generative models (which create good images). Generally, generative modeling is harder!
On this note: it would not be hard to train an AI model to distinguish AI and human art to high accuracy given some training data. On the other hand, I suspect if humans were given a “training dataset” for Scott Alexander’s problems, they would have done a ton better too.
It's time to pull out the difficult but highly rewarding first novel of William Gaddis, "The Recognitions" and give it another read through. Gaddis interestingly addresses some of Scott's points in a novel that was written 70 years ago when AI was not a thing at all other than in some science fiction novels.
I would also posit a second idea, that comparing images on a computer screen is flawed. AI, at present, only gives us an image that is devoid of chiascuro and brush strokes. Perhaps I might be missing something not having read Scott Alexander's post.
Yes, the baroque paintings I included would look far better in person, especially the Ruben's portrait.
I'm not trying to contradict anything you've said, but the historical origins of AI are really neat. Three years before "The Recognitions" was published (never heard of it before now), Claude Shannon designed a simple AI named Theseus: https://www.youtube.com/watch?v=_9_AEVQ_p74
"As an analogy, if in 2025 someone writes a book on evolution that is more accurate than The Origin of the Species, no one would argue that this makes the author a better scientist than Charles Darwin."
Are the Olympic record setting runners from the 1920s better than the runners than the amateurs running today who are faster than them?
No.
Are the pioneering biologists from the 1850s better at science-ing than the amateurs science-ing today who science better than them?
I don't think they are better, but it sort of depends how you define "better". In my view, people like Wallace and Darwin were greater scientists than the typical modern biologist, because they were more innovative. But I don't think they are better than modern scientists in the sense of knowing more about biology. There's a difference between greatness in the sense of being more innovative, and greatness in the sense of knowing more. Our society puts a premium on innovation, when judging scientists, artists, etc.
If "society" puts a premium on innovation over level of quality, then why are video games vastly more popular than the fine arts?
I'm saying that society puts a premium on innovation when judging greatness. If some engineer at GE produces a light bulb that is better than Edison's bulb, and sells more of them than Edison did, it doesn't change the fact that the public views Edison as a hero for inventing (co-inventing?) the light bulb, and the GE engineer as someone who merely fine-tuned a previous invention.
Popularity doesn't equate to reputation. Taylor Swift is more popular than Mozart, but Mozart's reputation is higher.
McDonalds is more popular than a Michelin-starred French restaurant, but the latter has a better reputation.
But we should be able to analytically disentangle artistic praiseworthiness from artistic quality. You should enjoy the use of the better light bulb more than the vintage Edison light bulb. The paintings that are indistinguishable from Rembrants are just as good as Rembrants.
> What are the odds that they’d randomly pick impressionism from among all the other equally great styles, if they were not basing their taste on which style is the prettiest
This confuses me. What could one possibly base taste on *other* than what one perceives as the most attractive? Surely the whole point of art is to be pretty?
You talk at a couple of points as though the "goodness" of a work of art was some objective quality, independant of whether people in general like it. This makes no sense to me either- art is good if people like it, not otherwise.
No, I do not believe in the idea of objective quality. I am offering my opinion. Perhaps I could cite an example, to make my point. Consider a painting like Titian's The Flaying of Marsyas:
https://en.wikipedia.org/wiki/Flaying_of_Marsyas_(Titian)#/media/File:Titian_-_The_Flaying_of_Marsyas.jpg
I don't think this painting is pretty, and I'd guess that most people would prefer a kitschy painting by Thomas Kinkade. But who would deny that this is one of the great works of Western art? After 450 years, it is still viewed as a masterpiece. Where will Kinkade be after 450 years?
https://www.google.com/search?sca_esv=cafa545287eff4ac&sxsrf=ADLYWIIPeHjHnf-K9K4wszzamngVooJoIg:1733335147083&q=kinkade+art&udm=2&fbs=AEQNm0Aa4sjWe7Rqy32pFwRj0UkWd8nbOJfsBGGB5IQQO6L3J03RPjGV0MznOJ6Likin94qgEKgZsTg4PX84JG3F-btD6HZ96SO26gpHdhe184a4g8SIySIaQxlkbf_9LYdgWhpzG2HznnjoXADuLb0NM_e6Xen2HcRK3rGETQyb9srtJ4jI89f-5cZ59x9f6pQub1F3cWqFOZXe7vUnVGcGRvTWEMXo4g&sa=X&ved=2ahUKEwiCyd_N2I6KAxWlDEQIHcJSDZ4QtKgLegQIEhAB&biw=2048&bih=958&dpr=1.25
"but I’m somewhat better than an AI (for now) at verifying whether one Rembrandt painting is better than another. I presume that is telling us something important about the type of intelligence that each of us possess, but I’m not quite sure what."
I think at least part of the answer to what intelligence we possess is that we have learned the myriad and conflicting rules of what today constitutes "great art" and many of these rules have nothing do do with the actual images themselves. Hence training an AI by exposing it to primarily the images is a very inefficient way of deducing the rules. Plus, the rules are in flux. Just consider that Van Gogh sold only one painting died in poverty.
I don't believe that great art is determined by "rules". Quality is recognized, sometimes after a long delay (as with Van Gogh and Vermeer.)
Hi Scott
Bessent appears to be a formidable choice, no?
He's fine, but not the best day to be praising Trump's picks.
Art history and the art market [the two are massively intertwined] are only tangentially to do with the "quality" of the work, they are a relic cult. See e.g. http://www.artnet.com/magazineus/features/bown/art-value5-25-10.asp
Thanks for the post Scott. I have learned a lot from your meditations on films and paintings. I'm hardly an aficionado, but even I could tell the difference between the two 'Impressionist' works fairly easily. To me, the difference is about composition. The genuine one - which you say is a Gauguin, but from my perspective could be any great artist whose work hangs in the galleries I've visited - is telling a story: There's a person, presumably a man, walking towards a cart that has someone else next to it. Maybe they're travelling together, maybe one wants to buy goods from the other or find out some information. The scene on the left is like the type of boring novice photo that I would have taken on my youthful travels, but that even kids these days with their incredible phone cameras have learnt looks simple and dull.
I think being a fan of classic film gives you an advantage in seeing the differences. I saw Seven Samurai a short while ago and the main element that struck me was not the action scenes, or the acting, and certainly not the story, but the composition. Even scene looked like one where the director had a clear idea about showing balance and beauty to the audience. If you look at the stills for the film today, it strikes you immediately. Many modern films and TV shows with their constant use of drones and moving cameras that focus on the actors rather than 'the scene' in which the actors perform drive me insane. To me, they make whatever the show is look like a soap opera.
Good comment about The Seven Samurai. In general, classic Japanese films have very strong visuals.
"To me, they make whatever the show is look like a soap opera."
This nicely describes what I dislike about lots of American TV, even "quality TV". That visual style works best for comedy, and it seems like comedy is the strongest form of American TV.
I must object that AI is not now producing paintings, but simulated digital images of paintings. Not the same thing at all.
Take Agnes Martin, among others. They say one can't take a compelling photo of her paintings. If you produce a digital image from training on such photos what can you possibly end up with?
And Mondrian. Mondrian was an insistently improvisational painter, despite the relatively strict rails he put on his improvisation. His impact on the graphics arts is enormous, but that whole enterprise is very far removed from his intentions as a painter. My sense is that a serious Mondrian fan would think that AI would have serious difficulties producing a quality Mondrian.
But this isn't really fair to the AIs either. Maybe if we trained AI on the neural data from humans looking at the painting instead of scans of the paintings maybe it would solve these problems.
Maybe a better game is to compare original human digital art to AI digital art, except who cares? Maybe photography is fair game, or maybe it is an even deeper problem.
Yes, those are all good points. My interest here was primarily the Turing test aspect of the comparison, but many other objections are also valid.
It's interesting to me how truly "great" many of the most famous painters (and artists) are, especially the early ones. If anyone could have painted (or made prints) like Durer, they would have been fabulously wealthy and successful. Of course, no one at the time came close. Arguably the same is true about later Rembrandt, to your point about his students. That top 10% is something of ineffable value we are far from reproducing by computer.
In an earlier post I discussed why so many of the greatest painters worked during the 1500s and 1600s.
https://scottsumner.substack.com/p/italian-mannerism-david-lynch-and
Yeah this was great.
There's a crossover between painting and sculpture that has only been somewhat developed new technique wise. I can picture some of these techniques but, other than embossing, don't know or can't recall their names. Lots of room for continued development here, though.
As seen with face blending, people think symmetrical averages are attractive, but the faces considered most attractive deviate from these symmetrical averages. There's a lot of room to deviate and not be considered as good as an average, and probably less room to deviate and be considered better than an average. It's a huge computational task just trying to hit this average; exploring the non-average space is a much larger undertaking, if you're an AI and not a human.
The fact that it's hard to put into words what makes a face especially beautiful might relate to why it seems difficult for an A I to recognize beauty.
Getting exposed to art via art museums in the US (especially outside of New York and Chicago) is probably a genuine detriment to being able to appreciate great art. Not in an "Americans sure do have bad taste, am I right?" way, but because the best stuff mostly remained in Europe and American museums tend to get the lesser works by the greats. Visiting museums in California as a kid, I knew which artists I was supposed to appreciate but found it hard to get excited by their lesser works. Meanwhile, an hour in the Mauritshuis makes it pretty clear why the Dutch Masters are such a big deal today.
I agree. I will say that there are some good museums outside New York and Chicago, including Washington, Philadelphia, Boston, and even (to a lesser extent) places like Cleveland, Buffalo and Detroit.
But yes, 90% of the great western art is in Europe.
I would add that I feel like I've seen a lot of attempts to copy Mondrian. I think it's very hard to do. Maybe Rothko would be easier?
Mondrian might be hard to fake but not impossible:
https://jpisbouts.medium.com/why-the-art-world-continues-to-be-vexed-by-forgeries-2ffe6f8d2635
As for Rothko - Perhaps you know this Rajat but, a fake Rothko was at the center of an art forgery scandal where the renowned Knoedler gallery in Ney York City brokered the sale numerous fake works. I learned about this by watching the documentary "Made You Look"
Even though the test is closed I did a quick run through without scoring it and realized I was pretty uncertain about many of my answers. The exceptions were anything that was anime or sci-fi related because at this point I'm fairly certain that the majority of images in that style on the internet are AI generated. That AI is landing some of its hardest blows in the realm of mediocre sci-fi is only too appropriate. We can expect a lot of movies and TV to be generated by prompt-operators in the equivalent of sweatshops that are absolute flops with viewers. I hope Scott lives long enough to review some.
I'm genuinely curious about the impacts of AI art on the broader economy. I have yet to use it seriously in my profession (architecture) but I've always been a slow adopter of better software. When a client responds to one of my design proposals with an AI generated alternative I'll know it's time to burn my money in a wastebasket and listen to Terror through the wall.
To me, TV already seems AI generated. I recently watched the first two episodes of "Disclaimer", and I can't imagine that an AI would do much worse.