Let me grab my old man hat, and put my banjo down for a moment and talk to you about the state of AI replacing software engineers. Personally, I don't think it's fair to say you shouldn't study engineering because the future of AI includes software engineers as well as everyone you've ever met who's said "I have an idea for an app" at every party where you've awkwardly explain what you do for work.
Now, we've entered the part of our timeline where every week a new AI company is claiming to have created the next big thing that will revolutionize the tech industry. I have to admit, it's been a wild ride the past year. Recently, it's a startup called Cognition Labs, proudly unveiling their creation: Devin, the world's first "fully autonomous AI software engineer."
According to their studious announcement, Devin is a tireless coding machine capable of planning and executing complex engineering tasks, learning unfamiliar technologies on the fly, building and deploying apps from scratch, finding and fixing bugs, and even training its own AI models. Essentially, Devin is a one-stop shop for all your software engineering needs, ready to replace your entire dev team with a single AI entity.
Check out their intro video:
Now, if you're a developer and panicking about being replaced by our robot overlords, let's take a step back and examine these claims with a healthy dose of skepticism (and optimism!).
First off, the examples provided by Cognition Labs are, shall we say, a tad underwhelming. Sure, Devin can run a pre-existing computer vision model and compile a report, but that's hardly groundbreaking stuff -- yet. And yes, it can supposedly solve bugs in open-source repositories, but the provided example involves a relatively simple logarithm calculation fix. Not exactly the kind of earth-shattering achievement that would warrant such grandiose claims.
But perhaps the most eyebrow-raising assertion is Devin's alleged performance on the SWE-bench coding benchmark. Cognition Labs proudly proclaims that Devin "correctly resolves 13.86% of the issues end-to-end, far exceeding the previous state-of-the-art of 1.96%." Now you might say "Wow, a whole 13.86%! Someone call the Nobel committee, because we've clearly reached the pinnacle of AI achievement" and to that I would say "clearly you don't understand how fast this revolution is happening before us."
In all seriousness, while a 13.86% success rate may represent a technical improvement over previous models, it hardly qualifies as a breakthrough that warrants dubbing Devin the "world's first fully autonomous AI software engineer." Unless, of course, you're comfortable entrusting your multi-million dollar codebase to an AI that gets things wrong 86.14% of the time. The gap between the current capabilities and the future where AI can reliably manage your technical IP with the help of engineers using AI are fast approaching.
Now, to be fair, Cognition Labs does acknowledge that Devin is still in "early access" as they ramp up capacity. But that begs the question: Why make such a grandiose announcement before the technology is truly ready for prime time? Could it be a clever marketing ploy to attract investors and top talent to their well-funded startup? Perish the thought (hire me!).
At the end of the day, only time will tell if Devin lives up to the hype or ends up being yet another overhyped AI product that fails to deliver on its lofty promises. I'm looking forward to seeing how Cognition Labs navigates this market with such strong competition. Until then, we mere human software engineers can rest easy, secure in the knowledge that our jobs are safe – at least until the robots learn how to correctly resolve more than 13.86% of coding issues.