Cheating Before ChatGPT

As prevalent as cheating was before ChatGPT, we were at least decently good at catching it. Copied an essay from your friend? The plagiarism detector probably found it. Copy/pasted off Chegg or Wikipedia? Ditto.

Educators also had straightforward, albeit time-consuming, strategies for mitigating these cheating risks. They could randomize variables in math questions, come up with lots of different questions to assess the same skills, and even find students admitting their academic crimes on Chegg. It was a grind, but nothing that a couple dedicated TAs couldn’t handle.

Of course, some students could circumvent these common cheater traps by hiring other people online to do their work for them, but that’s not accessible for the vast majority of students, at least not on a regular basis.

The bottom line is that cheating before ChatGPT was harder, more expensive and time-consuming, and the chances of getting caught were reasonably high.

ChatGPT Makes Cheating Easy

As you have probably noticed personally by now, ChatGPT lowered the bar for ‘good’ cheating — by at least an order of magnitude. Cheating that used to take hours of searching online, cobbling together resources, and hiring ghostwriters online could now be done in a matter of seconds — for free. And all the strategies that educators had used to catch and reduce cheating basically stopped working overnight. Detractors of traditional education rejoice!

This essay has little to do with the ethical or pedagogical concerns surrounding the use of ChatGPT, but a lot of educators today believe that using ChatGPT to do your coursework for you is against the rules. That’s a reasonable stance, but the only issue is how do those concerned educators enforce the rules? That’s what brings us to what I believe is one of the most noteworthy outright scams in education, edtech specifically, in recent years. And as most frauds in edtech, it will likely go on for a few years, until enough students have been hurt by it, that we’ll try to forget that we ever went down this path.

The Foundation of a Scam

One of the most important ingredients in a successful scam is a vulnerable audience. Old people with money. Poor migrants. You know the drill… In this case, the vulnerable audience is educators who feel powerless against the rampant AI-enabled cheating that is going on in their classrooms. They also happen to have budget.

Now, there’s absolutely nothing wrong with selling to people who have really painful problems that they can’t solve themselves — the entrepreneur in me is practically salivating at the thought of such a hungry customer base! The only issue is that the people1 selling subscriptions that claim to solve this problem are full of shit. By the end of this essay, you’ll understand exactly why that’s the case.

Oh, the irony of selling software-as-a-snake-oil to solve academic dishonesty… Let’s take a technical detour to understand why I’m confident that all of these ‘solutions’ are garbage. Stay with me here as we talk about some technical challenges behind AI-written content detection.

The Challenges of Detecting AI-written Text

If the notion of being able to throw some prose at an AI model and getting a definitive answer on whether it was written, in part or whole, by another AI sounds like bunk, then you’re already ahead of the curve. Broadly, this is why ChatGPT detection is likely long-term impossible:

  1. Language models like ChatGPT are trained on massive datasets of real human writing. Their output mimics natural writing styles and logical reasoning. There are no obvious “tells” that give it away as AI-generated. 2
  2. The AI can be prompted to write in different styles or even mimic a specific author. This makes the writing even more indistinguishable from a human’s. Students are prompting ChatGPT to write essays “in the style of a high school student” or “write this coding assignment as a computer science major would.” We are also on the cusp of having tools that will make it just as easy to fine tune an LLM’s style to your own, as it is to emulate your own voice through an AI. 3
  3. Students can easily edit the AI’s output to fix any oddities and make it read perfectly natural. After generating a draft essay, they can go through and change some vocabulary, fix typos, add specific examples, and polish the flow. This removes any detectable unevenness in the writing. Powerful paraphrasing tools abound nowadays.
  4. For technical writing, like in a computer science course, the AI combines logical reasoning with factual knowledge, making it very difficult to detect statistical anomalies. For example, ChatGPT can generate high-quality code that uses proper syntax, common libraries, and reasonable variable names. It’s practically impossible to determine if a 5-page Python program came directly from an AI.
  5. With open source language models proliferating, watermarking or fingerprinting AI-generated text is never going to work. Even if commercial models adopted them, the open source models like Llama are already available. They trivially allow students to run customized LLMs on consumer hardware that would never contain any identifiable metadata.
  6. Many academic assignments are pretty boilerplate themselves, so it’s almost expected, especially when it comes to technical or non-fiction content, that students’ solutions will not sound particularly unique or novel. Some call this ‘average’ writing. On the contrary, this test itself is largely useless anyway, because by providing novel context in your prompt, AI can generate very novel-sounding writing.

How do the Detectors (Attempt to) Work?

I won’t spend too much time going deep on how the detectors work, but at a high-level they use two key metrics: perplexity and burstiness.

The first, perplexity, is simply a measure of how different is the content from what exists in the model’s training corpus. This is why common phrases get flagged as AI-generated, because even if they are provably human-authored (e.g., the Constitution of the USA https://arstechnica.com/information-technology/2023/07/why-ai-detectors-think-the-us-constitution-was-written-by-ai/), they will have very low perplexity. We would also expect the vast majority of school work to have very low perplexity, because it’s plastered all over the internet.

The second, burstiness, or lack thereof, is more of a facet of ChatGPT’s default writing style. While humans tend to write with bursts of thought, LLMs have an almost too-perfect writing style that lacks the variability that human authors typically display. This can be easily edited into or out of prose. It’s also not unexpected for shorter and fact-dense prose to have low burstiness.

If this sounds kind of silly, it’s not just you. The fact is that these are just poor heuristics for spotting obviously AI generated writing of the flavor that today’s AI models happen to spit out by default.

Herein lies the crux of my argument for long-term impossibility: there is absolutely nothing in a chunk prose that can tell you whether it was written by a GPU or someone’s fingers on a keyboard.

You might wonder, couldn’t we use some sort of forensic linguistic analysis to determine the author? Well, sort of, but not really. AI can assume the writing style of basically anyone, so the moment you find some sort of pattern, it’s likely that the models or prompts can be updated to evade detection. And since real humans’ writing style does vary naturally, it wouldn’t need to be perfect to be passable. Unlike a crime scene, there’s no DNA underneath the characters on the screen that will tell you who left them there.

Research Points Towards the Impossibility of Reliable Detection

So far, you’ve understood why catching AI-produced content is so difficult and the heuristics that the detectors use to flag such content, but we haven’t talked about the actual accuracy of these detectors yet. Let’s take a look at the data:

There’s nothing that proves that accurate detection of AI-written prose is impossible5, but there’s more than enough to be sure that it’s at least really, really hard. Furthermore, there hasn’t been a single credible example of an accurate detector (even for the older LLMs, not to mention the state-of-the-art models like GPT-4).

This should make you default-skeptical of any commercial vendor, especially one that doesn’t have a world-class AI research reputation, that advertises metrics that sound too good to be true. Unfortunately, most educators are not familiar with these facts.

Peak Edtech Grift

We know that reliable AI prose detection is really hard, perhaps impossible, and that the undisputed leader in LLM research, OpenAI, has failed to create a model that it believes is safe. But what if I told you a well-known plagiarism detection company, Turnitin, managed to take an open-source Huggingface model, fine-tune it on a private dataset, and build the dream: a reliable AI-written text classifier?

Well, Turnitin 6 claims that they have done just that, with a false positive rate of below 1% 7. Are your bullshit detectors firing off yet? They should be. Let’s take a closer look.

On Turnitin’s AI classifier’s marketing page, the first thing you’ll notice is an obvious lack of technical rigor. There is no credible research being cited (or produced), no technical discussion of their model, and no way to run your own work through it to test it out. Might letting students try it out make it too easy for the student (or even another AI) to edit their work to evade detection?

Their “learn more” page, blog posts, and FAQs do exceptionally little to provide concrete technical basis for their claims. Whether this is simply security through obscurity or a deliberate attempt to downplay they immaturity of their technology, neither is a good sign.

That said, I’m not the only one who’s understandably skeptical and concerned. A Washington Post article in early April found the tool to perform marginally better than the abysmal GPTZero and OpenAI’s classifiers. It also found that mixed human and AI content was substantially less likely to be properly flagged, while paraphrased text (shout out to Quillbot!), was not able to be detected at all!

Since the Washington Post piece, Turnitin made some changes to their marketing (and detection model):

  • Removed the “98% accuracy” claim (but kept the <1% false positive claim) 8
    • In some places this <1% claim is qualified “for documents with over 20% of AI writing”
    • There’s also a new note saying that in order to support the <1% false positive rate, the detector “might miss 15% of AI written text in a document”
    • No guidance on what that 15% of missed AI-generated text might be correlated with or caused by
  • Added warnings in the UI whenever a document had less than 20% AI writing detected
    • What’s the point of even flagging submissions with such low likelihood?
  • Added a note in their FAQ that their tool “should not be used as the sole basis for action or a definitive grading measure by instructors.”

While Turnitin clearly doesn’t want to apologize and properly rein in their tool (as nearly every other vendor has done to date), they’ve quietly taken these steps to cover their asses, without sacrificing the commercial potential of their new product. It’s almost as if they’re not as confident as their marketing would lead you to believe…

And here’s the thing, Turnitin is facing an impossible problem. As generative AI models, and the students who use them to cheat, get better, Turnitin will either have to deal with missing way more than 15% of the AI-generated text, or Turnitin will be forced to increase their false positive rate.

Turnitin is Playing a Lose-lose-lose Game of Edtech Grift

In the case that Turnitin takes measures to catch a greater percentage of AI writing, they will inevitably increase their false positive rate. This will directly result in the prosecution of innocent students, ruining their future academic and professional career outcomes. At some point, it’s likely that this converges towards basically all content being flagged as AI-generated and educators will have to throw in the towel (canceling their Turnitin subscriptions).

On the other hand, if Turnitin focuses on maintaining a low false positive rate, they will fail to catch the vast majority of AI-enabled academic dishonesty. Fewer students will be wrongfully accused of using AI to cheat, but the minority of ‘honest’ students will find themselves further disadvantaged in the GPA maximization game. The students who are willing and able to prompt engineer their way to success will have been emboldened and will use AI to complete an ever-increasing chunk of their schoolwork.

Based on where things are currently headed, the latter scenario appears more likely. It’s also a quieter and slower death — to both the traditional model of education and Turnitin’s subscription base.

At this point it’s clear that Turnitin has found itself, disadvantaged, in arms race that it can’t win. Whichever way things go, the only certain result here is lose-lose-lose: teachers will lose, students will lose, and Turnitin themselves will ultimately lose too. The only winners in the end will be the developers of LLMs and educators who embrace these tools in a way that is compatible with the necessary future of education.

Where do we go from here?

Ah, the trillion-dollar question… Well, we can’t even begin to answer this without throwing away these useless tools, and focusing inward.

Seriously, first things first, schools need to stop (a) making false accusations of dishonesty the student’s problem and (b) forcing instructors to use these tools. If you don’t believe that schools would act with such blatant disregard for the facts, check out Boston’s University GAIA policy here.

The unfortunate reality is that most of the things we learn in school are very boring, yet impossible to live without. Consider how you’d contribute to society without being able to comprehend basic algebra, read and write, or use a computer. If anything, our AI-enabled future will favor the most intellectually capable among us. The notion that “AI can do all these things, and therefore they are useless,” is just nonsense. And that’s coming from someone who didn’t enjoy most of their traditional education…

A thorough answer requires another essay, but I’m fairly certain about a few things:

  • A strategic combination of high-tech and low-tech education is an essential part of the solution. Smartphones9 have absolutely no place in the classroom, and writing with pen and paper10, however dreadful it may be, is inescapable. Students must be capable of being productive without an AI assistant, and become proficient (to an even greater level) when paired with an AI assistant. Students must buy into this truth, otherwise they will fight it relentlessly, to their own detriment.
  • However controversial as of late, deadlines and consequences aren’t a bad thing(!), especially for less mature (often male) students. For some reason, even before ChatGPT and COVID, educators decided that there was no point in failing students, assigning homework, giving zeros for late work, or even having deadlines to begin with. A lot of cheating comes from overwhelmed students who have found themselves in a situation where they believe that cheating is the only realistic solution, but giving students absolutely zero consequences until the end of the term is crazy. It just moves the problem further into the future, where it is orders of magnitude more painful, and has the odd side effect of seeding an unproductive sense of entitlement among students.
  • Educators should embrace AI as a means for covering substantially more material and enabling students to personalize their educational experience. A large part of education isn’t optional — you have to learn addition and subtraction. However, AI-driven tutoring solutions could make learning the basics much more interesting for each student, while also guiding them through discovering their own personal interests. It’s important to not underestimate the value of having an expert on just about anything in your pocket.
  • Proctoring isn’t a bad thing. It got a bad rap during COVID, but the reality is that sometimes making sure that people aren’t gaining an unfair advantage over others is important. Expectations should be set ahead of time, and no proctor-free accommodations should be made for any reason.

With these key ingredients in place, the existential need for AI-written text detection tools fades away. If students are being assessed in class appropriately, on a regular basis, their low-effort AI-powered homework will be impossible to miss. A brief in-person conversation with the student is enough to prove that their work is not original. This also holds true for the vast majority of forms of cheating — at least the material ones where a truly incapable student leverages an unfair advantage to unnaturally improve their performance. It’s unclear why students shouldn’t be able to augment their writing with AI tools, the same way that they use spell check, for assignments that are authored digitally.

In some more advanced educational contexts, students might be expected to perform with access to AI assistance and the bar will be increased for everyone. It’s fair cheating, if you will. No need for the detection of AI content, because sufficiently complex assignments are not trivially solvable by LLMs to begin with. In fact, this is the only way to truly ‘teach’ students how to effectively use these tools to their benefit11.

  1. I say people, because companies are just groups of people. Saying companies has a dehumanizing effect for no reason. ↩︎

  2. I know what you’re thinking, but I’ve seen ChatGPT-written emails, and they just don’t sound right! Well, you’re not wrong. By default, output from an LLM doesn’t take on the style of the human ‘author’ (the person who prompted it) and it often has a boilerplate-sounding style. The problem is that this is because when you ‘average’ out all of the training data, this is basically a representation of how humans write. And humans do often write like that as well. The gist of what I’m saying here is that there is nothing in the text output itself that can tell you it was written by an AI. ↩︎

  3. Learn more about voice cloning here ↩︎

  4. The handful of cases where there was a Turnitin-based false accusation that I am personally aware of happened to be brought against non-native speakers. A lot of university students are non-native English writers, so take it for what it’s worth. ↩︎

  5. There are some good arguments that would indicate that it might be impossible. At least against the backdrop of rapidly-evolving LLMs and prompt hacking. ↩︎

  6. You might be wondering why I’m targeting Turnitin specifically. A few reasons, (1) they are the only well-known vendor that hasn’t walked back their claims, (2) they appear to have the most (paid) reach with educators, and (3) they are completely unapologetic when it comes to the implications of their technology. This is not some small startup run by fresh grads with good intentions to address academic dishonesty. Turnitin is a large business, supposedly run by competent adults, seeking to profit off the relative unsophistication and pain that educators are experiencing post-ChatGPT as quickly as possible. That’s wrong, and I feel like it’s important to warn people about it. ↩︎

  7. Earlier this year, Turnitin was advertising “98% accuracy” (whatever the hell that means) for their AI writing detector. They have since removed that wording entirely and replaced it with a guarantee of a “false positive rate below 1% for a document.” ↩︎

  8. We all know that people don’t read the fine print and pick up on all the requisite nuance. Teachers are being sold on this notion that the Turnitin model should be treated as true beyond a reasonable doubt. This is further complicated by the fact that a student who genuinely didn’t cheat using ChatGPT has no way to really prove that they didn’t. It’s disgusting to see a scenario where unwitting teachers are sending students, guilty until proven innocent, to academic integrity boards on the basis of a tool that is no better than the proverbial stochastic parrot itself! Every case that has been personally brought to my attention involved professors and academic integrity boards that fundamentally misunderstood the reliability of the Turnitin technology. ↩︎

  9. This obviously has nothing to do with the phones themselves, but rather the endless opportunity for distraction that they provide. Most schools, especially public schools, for a variety of mostly parent-induced and silly regulatory reasons, do not take students’ phones away while they are in class. This is such a shame, because class should be a sanitary environment, as free from unnecessary distractions as possible. ↩︎

  10. If anything, the AI technologies for scanning in notes after class should make writing on paper a non-issue. ↩︎

  11. Contrived examples are not really useful. Give students genuinely hard tasks — write a program with a fully-fledged user interface to annotate PDFs, for example — and watch them use an AI copilot to work more efficiently in a way that was not previously possible. Educators can even force students to “show their work” by uploading their chat logs. ↩︎