Your Vibe Coding Project is Infested with Vulnerabilities!
Building software with AI has gone from science fiction to everyday reality. Your AI-coded project may work perfectly… until hackers find the flaws. In this post, we will cover the way from AI-generated code full of findings and vulnerabilities to a trusted product by walking through an experiment we conducted to test Scribe’s AI-agentic analysis and auto-remediation workflows on a real-life project created with the Bolt.new vibe coding platform.
Generating the Project
The challenge was simple:
Build a web application that monitors hardware GPIO lines in real time and displays their status on a dynamically updated web page.
With a single prompt, Bolt.new generated the entire project – a Node.js project complete with a WebSocket server, a React.js front-end, and even a README.md file for GitHub.
The generation process took just three minutes, and we were able to download the code to a local Linux environment (Ubuntu 22.04 on WSL), install, and run it.
It worked right away, which surprised us. Here’s the running app screenshot:
First Impressions
Once deployed, we dug into the project’s structure. The app had 20 source-code files, plus 15 config files. The complete install, including dependencies, came to over 10,000 files. That raised an important question… How secure and reliable is this AI-generated code?
Phase 1: Traditional Instrumentation
Our first step was to run Scribe’s classic instrumentation, collecting security attestations from the build process. This gave us:
– Detailed SBOMs (Software Bills of Materials)
– Vulnerability reports (direct and transitive dependencies)
– SAST (static analysis) findings
– Secret scanning results
Vulnerability Report
- 4 critical vulnerabilities
- 6 high-severity vulnerabilities
SAST Findings
- 3 high-severity findings
- 12 medium-severity findings
Phase 2: Auto-Remediation with Scribe AI
Next, we uploaded the project to GitHub and used Remus, Scribe’s AI auto-remediation agentic workflow. Remus, in fact, a network of four AI agents, is designed to operate autonomously and collaboratively. Each agent has specialized capabilities, allowing for a distributed approach to complex problem-solving. This architecture enables the tool to handle a wide range of tasks, from data analysis and pattern recognition to decision-making and predictive modeling.
Running Remus led to a pull request with targeted fixes for the previously identified vulnerabilities and SAST issues.
Here’s an example of a commit automatically suggested by Remus to fix a vulnerability:
After merging the PR, we rebuilt the project, validated that it was working as before, and rescanned the project for issues.
Results After Remus Remediation
The transformation was dramatic!
– Vulnerability Scan: only 1 medium-severity issue remained
– SAST Findings: all fixed except for one medium-severity item
Scribe’s AI agentic remediation workflow effectively resolved critical and high-severity issues, leaving only minor problems for manual intervention, without hampering the product’s functionality.
Key Takeaways
This experiment showed that:
- While AI-generated code can appear fully functional initially, its out-of-the-box functionality comes with significant security risks and trust concerns. Projects utilizing such code face a high risk from threat actors.
- Security instrumentation in the pipeline is essential. Scribe’s GitHub plugin revealed multiple high-impact vulnerabilities in the generated project.
- Scribe AI-powered automatic remediation agentic workflow is powerful: it reduced the project’s risk profile dramatically with automated fixes while leaving the project fully functional.
Final Thoughts
The combination of AI-generated projects and AI-driven remediation suggests a future where software is both created and secured faster than ever before. While human oversight remains crucial, tools like ScribeHub with its AI agentic AppSec workflows prove that automated code repair isn’t just possible – it’s practical.
This content is brought to you by Scribe Security, a leading end-to-end software supply chain security solution provider – delivering state-of-the-art security to code artifacts and code development and delivery processes throughout the software supply chains. Learn more.
Related posts
