How Well Do These Websites Offering Ai-detection Actually Work?
As someone who uses AI every day for work, I was curious, how well do these websites offering ai-detection actually work? There have been stories circulating in the news about students cheating on tests with chatGPT, as well as rumors of sneaky workers doing their jobs secretly with Jasper AI. So it stands to reason that if people are using AI to cheat on their responsibility, there must be some Artificial Intelligence software that can detect fake writing? I had to know is there a way to detect if text is written by a machine? Old habits die hard, so I googled 'ai-detector'.
Websites That Boldly Claim to Be Able to Detect AI Generated Text
I got a full page of websites that boldly claim to be able to detect ai generated text. The first link on the page of results was writer.com I was already familiar with this company; they offer a wide range of services for blog writers like myself. To test this first machine, I gave the 'detector' one paragraph. I used the advertising text I had just lifted off another website's AI detection page, then I fed it into GPT4 to summarize into a short paragraph. I didn't edit the text in any way. As a control of sorts I then instructed GPT4 to generate one more paragraph with the 'continue' command and this 'control' was the first one to go into the ai-detection software.
The result of the first test was great, the ai-detection algorithm correctly detected the paragraph with a 99% chance it was written by AI, so far so good. I then ran the first paragraph from the same series.
I was surprised to see this paragraph was given a 97% chance of being written by a human author. This caused me to become a little alarmed, so I moved on to the next detector on the list. Hopefully copylinks.com would be more effective at spotting the robot writing.
I copied the advertising text directly off the copylink detector webpage and dropped it into GPT4, and instructed GPT to summarize the webpage. It gave me this text here:
The AI Content Detector is an enterprise solution for detecting AI-generated content with 99% accuracy, LMS and API integration, and support for multiple languages. It uses AI technology to identify AI-created content across various models, including ChatGPT4, GPT3, and Jasper. The platform provides a precise probability score, fast and accurate results, and in-depth analysis highlighting human-written and AI-written content. It supports languages such as English, Spanish, French, Portuguese, German, Italian, Russian, Polish, Romanian, Dutch, Swedish, Czech, and Norwegian, with more languages being developed.
These Results Should Give People Who Use Ai-detectors Pause
I dropped the computer generated words into the text box and I hit enter - 99% human was the response, so I tried the 'control' text next. 59.8% human it said of the second GPT generated paragraph. Now in defence of copylinks.com this was a very small data sample. I would have to do many more of these tests to get a better feel for the true accuracy level of the service they offer. That said, these results should give people pause when they use these services for any serious reasons, because the AI-detection websites DO give false positives.
Now I was becoming more concerned about the viability of artificial intelligence detection as a business model. Is this AI-detecting technology even ready for public use yet? With this question fresh in my head I moved down the list to the next site crossplag.com.
On the crossplag.com website they explained:
How does the AI Content Detector work?
...Using a vast dataset of content created by AI and human-written content, the tool is trained to learn the patterns and characteristics of each form of writing and can easily detect it...
They make the claim that they could easily detect the 'characteristics' of AI writing, so I plopped in the same paragraph that befuddled the two previous detectors and smashed ENTER. Now i'm a bit sad to tell all of you university professors who use AI 'detection' for grading papers - IT FAILED TOTALLY!
I had watched GPT4 spit the text out, line by line
On both paragraphs their detector software said they were almost certainly written by a human hand, but I had watched GPT4 spit the text out, line by line so I know that's not accurate.
If you are relying on these detection services to catch cheaters, you might want to re-think your rationel because Ai-detection is not quite ready for prime-time in my humble opinion. You may just accuse an innocent person of cheating at this point. Buyer Beware.
If you have any questions or suggestions for further research in this area, then use the contact form plz, thanks for stopping by.