Suramya's Blog : Welcome to my crazy life…

AI (or rather LLM’s) use is becoming more & more common and one of the issues with it (amongst many) is that when you ask it to summarize/process/or whatever something you have to upload that content to servers that are outside your control, then to make things worse a lot of these services use the data uploaded to further train their models. It is not illegal because they have put it in the user agreement that you allow them to use the data in anyway they want. Once the model is trained on data you have uploaded then it is possible for others to extract this data with a specific prompt. This happened to Samsung back in 2023 where an engineer uploaded internal source code to ChatGPT and another user was able to download it.

So the stakes are quite high when scientists are asked to peer-review papers that can contain significant breakthrough’s. Secondly a good peer review evaluates a paper and a lot of times identify issues that the author needs to address before publishing. Using AI for this can significantly degrade the quality of the review and this was identified as an issue by multiple publications.

The 40th Annual Conference on Neural Information Processing Systems (NeurIPS) has strict rules banning peer reviewers from uploading papers they are reviewing to AI chatbots but they are allowed to use AI chatbots for background research. This is pretty normal nowadays but they went a step further to enforce this policy and catch illicit AI use by deliberately concealing instructions for large language models (LLMs) in papers sent out for peer review. These instructions instruct LLM’s to use specific phrases in the review report allowing the organizers to identify the reviewers who breached policy.

The instructions tell an LLM to use telltale phrases—such as “This work addresses the central challenge” and “The claims of the paper”—in a peer-review report. Some researchers have already been caught trying to sneak secret messages into their papers in a bid to game AI tools into giving them favorable referee reports. Many publishers ban the use of AI in peer review.

Most people are in favor of this action but there are some who have been pretty vocal about not liking this action. They claim that “Designing a trap that presumes bad faith corrodes the relationship the whole system depends on,” and “You do not build a healthy reviewing culture by treating your reviewers as suspects.” I have a feeling that folks in this group are the ones who were previously using LLM’s in their review and are not happy about being caught. Is this a perfect method to catch folks who are cheating? Of-course not, but it is better than what we had earlier.

Some of the justified criticism against this method is coming from reviewers who found the embedded prompts in the paper and assumed that they were put in by the paper authors (there have been multiple cases where authors put in hidden prompts in their papers to give them favorable reviews).

Source: Scientists decry conference’s use of hidden prompts to snare AI peer reviews

– Suramya

I have talked about Vibe coding in a lot of my posts about AI and I just realized that some of the readers of my Blog Posts might not actually know what it means. ACM (Association for Computing Machinery) recently shared a Tech Brief on Vibe Coding (AI-Assisted Software Development, or Vibe Coding: Benefits and risks of AI-driven Software Development) that gives a good high level overview along with the benefits and risks associated with the practice so I am sharing it here.

You can download/view the PDF version of the document at: Vibe Coding: Benefits and risks of AI-driven Software Development.

AI-Assisted Software Development, or Vibe Coding: Benefits and risks of AI-driven Software Development
by simson Garfinkel, mohan sankaran, rohan sharma, Shrinivass Arunachalam Balasubramanian, Arpan Pandey, and Aruun Kumar

AI-Assisted Software Development, often referred to as “Vibe Coding,” is the practice of using Generative Artificial Intelligence to create or modify software systems in which humans describe what they want to build or modify, and an AI coding assistant writes and debugs computer code. Several popular vibe coding systems are built on top of Agentic AI systems, an “approach of making AI systems capable of setting or refining plans and executing tasks with minimal or
no human oversight”

Vibe Coding Benefits
Vibe coding enables people with little or no coding experience to create highly functional applications [2]. It can also assist experienced programmers by generating code that leverages complex application programming interfaces (APIs), a hallmark of modern software development.

Because vibe coding lets developers spend less time writing code, they can focus on higher-level concerns like design, user experience, and other creative problem-solving. Vibe coding might thus shift developer effort from time-consuming implementation toward higher-level design and intent specification.

Many developers report feeling more productive when using AI to generate code [3], especially with mundane programming tasks that do not require significant creativity [4], although these reports are subjective and may not be borne out by empirical measurements over time.

Vibe Coding Risks
Software engineering’s established practices produce systems that are generally secure, reliable, and maintainable. Vibe coding circumvents these practices. While it can produce code that meets immediate requirements for style, conventions, and targeted (“unit”) tests, it does not produce well-designed software systems. Because many of these systems have been trained on data that includes cybersecurity vulnerabilities, there is a risk that they will replicate these in the code that they generate [5, 6].

A core principle of modern software development is that a program’s functions and behavior need to be specified in advance. “A program that has not been specified cannot be incorrect, it can only be surprising” [7]. AI-generated code typically lacks specifications. Even when specifications are provided, many of today’s vibe coding platforms lack mechanisms to enforce them. As a result, AI-generated code drifts away from stated requirements, including core functionality.

Few vibe coding platforms systematically test their AI-generated code to ensure it runs correctly and consistently [8]. Although it is possible to give these systems acceptance tests for the code they generate—or even have them generate their own tests—AI systems have been observed to modify, disable, or simply remove such tests rather than correcting
their code [9, 10].

Vibe coding platforms often produce over-engineered solutions with redundant code and subtle errors that create maintenance nightmares, known as “technical debt” [11]. Entry-level programmers do this as well, but they are typically supervised by senior programmers when code is critical. Entry-level programmers often seek to improve their skills and
are penalized if they try to subvert internal controls. AI-generated code, in contrast, is frequently unaudited, and there is no way to penalize a misbehaving AI. This can result in code that is, paradoxically, maintainable only by AI: the sheer volume and complexity of AI-generated code make manual code review impractical, increasing the likelihood that
undetected errors slip into production.

Recently, many vibe coding platforms have added “agentic” features that go beyond software development, allowing the platform to run programs on the software developer’s behalf, often without the human first reviewing and approving the program’s execution. This can make users more productive, since the platform can operate more quickly without
human intervention. However, it also lulls the user into granting the platform increased authority to run new executables without explicit review.

The agentic platforms can typically execute these programs not only on users’ computers but also on any computer reachable over their network. This leaves the users and their networks at risk if the AI executes commands users did not intend. For example, deleting critical information, sending confidential information outside the enterprise security
perimeter, downloading and executing software from the Internet, or reconfiguring computers so they become susceptible to intrusion. Vibe coding platforms can also be vulnerable to “prompt injection attacks” when third parties embed malicious commands in software that are interpreted as instructions from the programmer [12].

Vibe coders may generate significantly more CO2 emissions than traditional programmers. This is often debated, as vibe coding produces code faster than humans do, and in small-language models, the total energy difference between AI and prolonged code development could be comparable. But because vibe coding often overproduces code, it still
requires human intervention to refine and optimize. Energy consumption with “standard, widely-used models is far more environmentally strenuous” [13].

Vibe coding may also have long-term negative effects on skill development in the programming profession. An internal study from a major AI provider found that students and early-career programmers using vibe coding showed decreased mastery of sophisticated programming concepts and skills [14]. In educational settings, students with advanced pro-
gramming skills were more likely to succeed in building a program with AI assistance, whereas students with less coding experience were less likely to do so, indicating that instruction in fundamental programming concepts remains necessary.

Vibe coding may thus contribute to a hypothesized “experience gap,” in which AI automates many early-career skills that are both drudgery for more experienced programmers and a necessary step in building mastery. Such skills include simplifying redundant code, porting code to new environments, and the routine addition of simple features, which
typically require a programmer to first understand the codebase. Some studies have shown significant cognitive erosion resulting from AI tools, although they did not specifically consider vibe coding [15, 16]. Nevertheless, by eliminating opportunities for junior programmers to become senior while simultaneously deskilling those later in their careers,
increased AI use in software development may paradoxically contribute to a shortage of more experienced workers.

Conclusion

It is unclear what vibe coding means for the future of programming or the economic outlook for the programming profession. While the job market for programmers appears to be cooling [17], some studies find that junior developers see the biggest impact of vibe coding, which makes it less likely they will themselves be replaced with AI agents [18].
Vibe coding can make expert developers more productive and allow novice developers to create and deploy working apps, but current platforms do not enforce modern software engineering practices. The core issues are systemic: these platforms do not create formal specifications and frequently ignore them when provided; they do not systematically test
their outputs and may remove/modify failing tests rather than address the underlying problems; and they generate code that becomes maintainable only by AI, not by human developers. The same mechanism responsible for these failures — the lack of a rigorously enforced semantic model that allows AI systems to validate their outputs — is also responsible for AI hallucinations more broadly. Because of these fundamental limitations, vibe coding requires that users and organizations compensate with improved technical checks and governance mechanisms to avoid predictable failure modes.

Existing techniques for improving code quality can be applied to both human- and AI-generated code. This includes the use of mathematical verification and other formal methods and techniques [19], as well as new work on developing specially tuned AI models adept at finding security vulnerabilities [20]. Such techniques will be needed to make vibe
coding a cost-effective and secure alternative to traditional software development.

Hopefully you found this as useful as I did to understand Vibe-Coding, what it means and how it impacts software development.

– Suramya

Was talking to a friend working in a startup with an AI focused product and asked him how is AI helping them. He answered that it allows them to make releases faster. You should have seen the look on his face when I asked “so what? Are the releases bug free? Do they solve the business requirement without errors?” It blew his mind when I asked this and he told me they can now release the fix faster.

The above behavior is typical when you talk to AI proponents. The main selling point for them is that you can release faster. My counterpoint is that are the faster releases solving business problems faster? Or allowing you to push out fixes for stuff that doesn’t work/broke in production because you didn’t check it correctly? If it is the former then fantastic. That is what I need AI to help me do, nut if it is the latter then it is of no use to me or the business. People forget that IT is not there in a company to try out the latest tools or use the latest technologies. It is there to solve business problems and deliver solutions that help business proceed. If this means using a 30 years old technology because ‘it just works’ then that is what you do. Whatever we do that doesn’t give fast, reliable and efficient releases is of no use.

Taking the example of being able to release faster. It is awesome if I can release features faster to production, but if the release introduces bugs or breaks functionality it is worse than a slow release because till the fix is deployed their work is stuck or they are getting wrong information which means that the work needs to be redone post the fix being deployed. How is that a win for the business? Sure, in some cases it is a genuine win because you released a feature faster but in a majority of vibe-coded instances it is something that kind-of-sort-of works and you have to go back and release a fix because something broke. This is apparent in the stability and uptime of every single application/site that has boasted of using vibe-coding be it Microsoft with its multiple bug-fix releases, Twitter going down almost daily, Amazon services going down because of AI release deleting production data and many other such examples.

Another issue that people don’t really think about is maintainability of code. People tend to thing that code can easily be replaced with newer code when we need to, but the people who think like that never had to work with 30 years old legacy code that can’t be replaced because it is running critical systems and it is too expensive to replace. Every bank I have worked in has ongoing multi-year project to replace mainframes with newer systems. Think about that, mainframes are older than I am still run critical banking systems worldwide. Similarly we have other critical systems that run old code that has to be managed and with AI generated code that is difficult to achieve if you have not reviewed/updated/understood the code on an ongoing basis. It does get things to a working state (most of the time) but it also in a lot of cases create code that is very hard to maintain. For example, the below screenshot was posted on the vibecoding reddit a little while ago and this is similar experiences faced by others in the industry when they do pure vibe-coding.

r/vibecoding ( 19h ago )
vibe coded for 6 months. my codebase is a disaster.

the app works. users are happy. revenue is coming in.( that’s
actually the only good part)

but i just tried to onboard a dev to help me and he opened
the repo and went quiet for like 2 minutes. then said “what is
this.”

6 months of cursor and lovable and bolt. every feature
worked when i shipped it. but nobody was thinking about
structure. the Al just kept adding. new file here, duplicate
function there, 3 different ways to handle the same thing
across the codebase.

tried to refactor it myself last week. gave up after 2 hours.
the thing is so tangled that touching one part breaks
something completely unrelated.

the generation was fast. the cleanup is a nightmare.

is there even a way out of this or do i just rewrite everything from scratch?

Finally, if AI/LLM’s were so good and perfect in generating code you wouldn’t need an industry wide media campaign to get people to use it, folks would use it on their own without companies having to track the usage and incentivize it. I have been coding for 28+ years now and have seen multiple advances/changes in how we code over the years. For example when IDE’s started supporting auto-complete for boiler-plate stuff people immediately started using it. When git came out folks started using it and immediately found it useful so no push was needed to get people to adopt the new tool. The same folks then pushed their work IT teams to start supporting git in the enterprise. If Microsoft/Amazon and other companies have to mandate their teams to use AI then it looks like the rank and file are not finding the tools to be that useful.

Personally I love it for Proof of Concept or quick and dirty prototyping/trying out new things. But before any code that is AI generated goes into production you need to ensure it is reviewed by a human who knows coding.

– Suramya

Over the past few days an Anonymous post on Reddit (Archive.org link since the original has been deleted) that alleged significant fraud at an unnamed food delivery app. The post made some serious allegations and the entire thing just exploded everywhere with a lot of discussions on how this kind of behavior is true. The reason everyone thought it was true was because Gig based companies have been caught doing similar things in the past.

Now here’s the twist that no one expected, apparently the whole thing was a hoax. Yes, you read that correctly. Casey Newton at Platformer has posted an entire writeup on this Platformer.news: Debunking the AI food delivery hoax that fooled Reddit that is a fascinating read. You should check out the whole writeup for the details on how Casey figured out it was a hoax. The part which was really scary is towards the end of the article where he talks about how AI/LLM is making fact checking harder.

“On the other hand, LLMs are weapons of mass fabrication,” said Alexios Mantzarlis, co-author of the Indicator, a newsletter about digital deception. “Fabulists can now bog down reporters with evidence credible enough that it warrants review at a scale not possible before. The time you spent engaging with this made up story is time you did not spend on real leads. I have no idea of the motive of the poster — my assumption is it was just a prank — but distracting and bogging down media with bogus leads is also a tactic of Russian influence operations (see Operation Overload).”

For most of my career up until this point, the document shared with me by the whistleblower would have seemed highly credible in large part because it would have taken so long to put together. Who would take the time to put together a detailed, 18-page technical document about market dynamics just to troll a reporter? Who would go to the trouble of creating a fake badge?

Today, though, the report can be generated within minutes, and the badge within seconds. And while no good reporter would ever have published a story based on a single document and an unknown source, plenty would take the time to investigate the document’s contents and see whether human sources would back it up.

I’d love to tell you that, having had this experience, I’ll be less likely to fall for a similar ruse in the future. The truth is that, given how quickly AI systems are improving, I’m becoming more worried. The “infocalypse” that scholars like Aviv Ovadya were warning about in 2017 looks increasingly more plausible. That future was worrisome enough when it was a looming cloud on the horizon. It feels differently now that real people are messaging it to me over Signal.

We are going to see it more and more of this going forward. The only way to counter is to double or triple check everything you read online, especially if it is baiting you into outrage. I try to do the same thing when I write about stuff but there are times when I have been fooled as well and have usually posted a comment on the post (or a correction in it) explaining it. Basically if it seems too good to be true, it probably is.

Source: @inthehands@hachyderm.io

– Suramya