Exercises in Style is a 1947 work of experimental fiction that finds 99 ways to tell a vapid anecdate about a loud man on a train with a hat. The stories were written by Raymond Queneau, a founder of The Institute for Potential Literature, known as Oulipo (Oulipo is a portmanteau of the full name in French). I found out about Exercises in Style reading one of its contemporary spinoffs, Exercises in Mathematical Style, a book that solves the same equation with 100 different approaches. Queneau’s book is mind-expanding. What more direct illustration of literary potential can you imagine?
I’m a professor, and I teach large classes, so I encounter student misconduct, and I rely on automation, and I handle my misconduct detection with automation. This doesn’t just save me a lot of work, it maintains a uniform bar for students. My tools have excellent accuracy. But the problem is that my students simply don’t believe that I can tell that they’re using AI. “It’s too good to detect,” they think. So they go ahead anyway. But I can tell. I try to prove it. I’ve published my methodology, and my catch statistics. I’ve shared my background as a cognitive scientist, my ideas about AI, and my publication at NeurIPS, the premier AI/ML conference. Because I don’t want them to get caught. But they don’t believe me.
I’m grading student writing right now. It’s given me occasion to read a lot of AI written answers to the same questions. With that kind of consistency, you start learning more about your author than its subject. And you can’t help but notice things. It’s been giving me great insight into the nature of these tools.
In my assignments, I give students a very open-ended hypothetical scenario and ask for two possible explanations for it, based in course concepts. Some students get it, some don’t, and it’s a good assessment. Around the nucleus of a few strong, reliable answers that consistently come up is a periphery of randomness and creativity. It actually used to be fun to grade. Student answers, even when they’re “the same”, are always stated from different vocabulary and conceptual basis. So of like students building the same model car, but one with legos, one with cardboard, one with matchsticks. You could really tell when students started to substitute their work for AI. You don’t have to be an expert on AI because you’re an expert on your assignments. In reading dozens of AI answers to the same prompt, I don’t know what’s more remarkable: how much the prose differs from submission to submission, or how much the logic coheres. In my assignments, the versions AI composes will be completely different but 80% of answers will offer the same two explanations in the same order with the same buildup.
It would be a stretch to say that each AI answer has a different style. Certainly they have a different texture, though conspicuously bland against student work. And the bones are the same: not just the logical flow, but the basic conceptual units of the argument, like every model car being made out of legos. The possible answers stand together like a neighborhood of pre-fab homes, seemingly different but just playing out the combinatorics of color, trim, and whether it’s porch-then-garage or garage-then-porch.
Driving through that kind of neighborhood, you learn a lot about the architect, how they see the problem. Watching AI excel at my homework, in hundreds of different flavors of the same way, it’s both satisfying and disturbing to a cognitive scientist whose professors were convinced none of this would happen. I remember sitting in Stuart Russell’s office hours as a student in the second AI winter, early 2000’s, asking him if the flexibility of neural networks would ever get us through the rigidity of formal approaches. I didn’t buy his skepticism, but I did accept that machines would never figure out “fundamentally human” things like GO, or driving, or faces. I’ve been so consistently wrong about AI. It’s a delight and it makes catching cheating fun. As much as I wish it would stop. I can’t keep reporting 20%-50% of my class. But my students, they just won’t believe me.
