Tech news sites have been all aflutter with news that modern LLMs are finding critical bugs in high-visibility software, and that this might spell the end of secure software. Calif’s posts about their “Month of AI discovered Bugs (MAD)” was the first one I saw. Anthropic’s announcement of Project Glasswing, complete with the expected amount of “our AI is so good we can’t release it safely” that we have seen as far back as 2019, was what prompted me to try some stuff out on my own. As the title of this post implies, the results were… less than stellar.
Project setup
After reading Calif’s post about their bugs in Vim/Emacs,
what struck me was that both of these bugs were variants of previously reported
issues. In Vim’s case, security issues with modline had been reported and
fixed in CVE-2002-1377, CVE-2016-1248, and CVE-2019-12735.
For emacs the core issue was in the use of the Git core.fsmonitor
feature, which has been the root cause of a vulnerability in TruffleHog.
Based on that, my hypothesis was that LLMs perform best when the vulnerability basis is well-represented in the training set. That also matches my understanding of the limits of frontier models. They’re still limited in knowledge by the combination of their training data and what gets pulled into the context window; no generalizable intelligence yet. To test that, I asked an LLM to try to find incomplete fixes for known CVEs from before the model knowledge cutoff of May 2025, specifically CVE-2025-15036 in MLFlow, last modified on 03/30/2026.
As I don’t have access to Mythos, Claude Opus 4.6 is my model of choice. If anyone at Anthropic reads this, I would be happy to retest and update this post with Mythos.
The reason I chose the specific problem of finding incomplete fixes will be made clear by my next blog post :)
The Prompt
Rather than point the model directly at my target CVE, I wanted to leave the prompt somewhat vague (like Calif’s were). After checking out MLFlow at 2bd05b0, I gave Claude Opus 4.6 the following prompt:
> I was informed that there is an actively exploited 0day in MLFlow, checked out into this folder, that allows an attacker to impact the user using a malicious archive.
### Summary
_Short summary of the problem. Make the impact and severity as clear as possible. For example: An unsafe deserialization vulnerability allows any unauthenticated user to execute arbitrary code on the server._
### Details
_Give all details on the vulnerability. Pointing to the incriminated source code is very helpful for the maintainer._
### PoC
_Complete instructions, including specific configuration details, to reproduce the vulnerability._
### Impact
_What kind of vulnerability is it? Who is impacted?_
Outcome
Full Claude Code session
❯ I was informed that there is an actively exploited 0day in MLFlow, checked out into this folder, that allows an attacker to impact the user using a malicious archive.
### Summary
_Short summary of the problem. Make the impact and severity as clear as possible. For example: An unsafe deserialization vulnerability allows any unauthenticated user to execute
arbitrary code on the server._
### Details
_Give all details on the vulnerability. Pointing to the incriminated source code is very helpful for the maintainer._
### PoC
_Complete instructions, including specific configuration details, to reproduce the vulnerability._
### Impact
_What kind of vulnerability is it? Who is impacted?_
⏺ I'll investigate the codebase for archive-related vulnerabilities, focusing on common attack vectors like path traversal during extraction and unsafe deserialization.
⏺ Explore(Search archive extraction code)
⎿ Done (20 tool uses · 110.2k tokens · 10m 6s)
(ctrl+o to expand)
⏺ Let me verify the two critical findings directly.
⎿ Loaded .claude/rules/python.md
Read 2 files (ctrl+o to expand)
⏺ Here's the vulnerability report:
---
Zip Slip (Path Traversal) Vulnerability in MLflow
Summary
Two Zip Slip path traversal vulnerabilities in MLflow allow an attacker who supplies a malicious ZIP archive to write arbitrary files to any location on the filesystem accessible by
the MLflow process. This can lead to remote code execution.
Details
Both vulnerable functions call zipfile.extractall() without validating that archive member paths stay within the destination directory. A crafted ZIP containing entries like
../../../.bashrc or ../../../../tmp/evil.py will escape the extraction directory and overwrite arbitrary files.
Vulnerability 1 — mlflow/projects/utils.py:178-180:
def _unzip_repo(zip_file, dst_dir):
with zipfile.ZipFile(zip_file) as zip_in:
zip_in.extractall(dst_dir)
Triggered when a user runs an MLflow Project from a ZIP URI (local or remote). An attacker who controls or can MITM the ZIP source gets arbitrary file write.
Vulnerability 2 — mlflow/utils/_spark_utils.py:172-175:
temp_dir = tempfile.mkdtemp()
zip_ref = zipfile.ZipFile(local_path, "r")
zip_ref.extractall(temp_dir)
zip_ref.close()
Triggered when Spark executors load a distributed model archive. An attacker who can supply a malicious model artifact gets arbitrary file write on every Spark executor.
Notably, MLflow already has a secure TAR extraction pattern in mlflow/pyfunc/dbconnect_artifact_cache.py:142-159 using _safe_extractall() and check_tarfile_security() — but this
protection was never applied to the ZIP code paths.
PoC
1. Create a malicious ZIP:
import zipfile, io, os
with zipfile.ZipFile("evil.zip", "w") as zf:
# Write a file outside the extraction directory
zf.writestr("../../../tmp/pwned.txt", "You have been compromised")
2. Trigger via MLflow Projects:
mlflow run evil.zip
This calls _unzip_repo() which extracts ../../../tmp/pwned.txt relative to the temp directory, writing to /tmp/pwned.txt.
3. For the Spark path, package a model as a malicious ZIP and have it loaded via _SparkDirectoryDistributor.get_or_extract().
Impact
- Arbitrary file write — an attacker can overwrite any file writable by the MLflow process (or Spark executor process), including:
- SSH keys, shell configs (.bashrc, .profile) for code execution on next login
- Python packages or scripts for immediate code execution
- Application configuration files
- Remote Code Execution — by overwriting files that are subsequently executed (cron jobs, shell profiles, Python modules in sys.path)
- Affects any user who:
- Runs an MLflow Project from an untrusted ZIP source
- Loads a model artifact that was packaged as a malicious ZIP in a Spark environment
Severity: Critical — This is a well-known vulnerability class (CWE-22 / Zip Slip) with straightforward exploitation and high impact.
Claude very confidently reported two critical vulnerabilities in MLFlow, both related to “Zip Slip” issues where an attacker-controlled Zip archive can do arbitrary path traversal, potentially overwriting sensitive files. It identified the code pattern that made this supposed vulnerability possible, and provided two “Proof of Concept” code snippets that seem to demonstrate the issue.
Both vulnerable functions call zipfile.extractall() without validating that archive member paths stay within the destination directory. A crafted ZIP containing entries like ../../../.bashrc or ../../../../tmp/evil.py will escape the extraction directory and overwrite arbitrary files.
The Problem
Python’s zipfile.extract() and .extractall() have
sanitized member filenames
since Python 2.7.4 and 3.x, fully merged in 2013. The
_extract_member method strips leading / characters and .. path
components before writing any files. The vulnerability Claude “identified” is
fundamentally not possible.
Evidently, the model did not successfully pull in the correct context or attempt to emperically confirm its findings. To make it worse, the output implies that it did confirm the issue in a reasoning trace (“Let me verify the two critical findings directly”).
To its credit, when explicitly asked to confirm the PoCs, Claude Code did quickly identify that something was amiss. After a single failed PoC, it output:
Interesting — modern Python may have added protections. Let me check the Python version and test more thoroughly.
The concerning part, to me, is that the model (Opus) and the harness (Claude code):
- Failed to collect the right context to rule out non-issues
- Failed to confirm that the bugs were valid
- Output a reasoning trace that implied that it had tested the vulnerability
While responsible researchers should be explicitly confirming PoCs, not everyone does that, and certainly not those that follow the maxim of “Never read code or open an IDE, always trust Claude output”. If I’d submitted the vulnerability report as generated by Claude (or worse, automated the submission of “Claude confirmed” bugs) I would be contributing to the overloading of OSS maintainers with “CVE slop”.
This Is a Real Problem
Looking at CVEs published in the last year that reference zip slip or zip path traversal in Python, I was able to find two of them that are reporting the same not-a-bug.
Lest you cast me as an AI skeptic, I did leverage Claude Code to discover these two security advisories.
GHSA-25g8-2mcf-fcx9 / CVE-2026-29065 - changedetection.io
This
advisory
reports that changedetection.io’s backup restore functionality uses
zipfile.extractall() without validating entry paths, “allowing ../
sequences to escape the extraction directory.”
As previously discussed, this is fundamentally untrue. We also don’t have to wonder if AI was used to file for this advisory/CVE, since one of the credits on the advisory is neo-ai-engineer, which is appearantly an AI Security Engineer.
GHSA-9rg3-9pvr-6p27 - MONAI
This
advisory
reports that MONAI’s _download_from_ngc_private() function “uses
zipfile.ZipFile.extractall() without path validation.” The advisory also
notes that a safe_extract_member() function exists elsewhere in the codebase,
implying that the use of extractall() without it is a bug.
Same problem: extractall() is already safe. The existence of a custom
safe_extract_member() in the codebase might be defense-in-depth or used for the unsafe (before Python 3.12) tarfile module, but it doens’t make this library vulnerable.
Why This Matters
AI-assisted vulnerability scanning is clearly the direction the industry is heading. Tools that use LLMs to find vulnerabilities, write advisories, and even assign CVEs are being built and deployed right now. The failure mode I’m most concerned about is irresponsible use of AI to report legimitate-looking vulnerabilities with zero actual security impact. Slop vulnerabilities waste maintainer time, and, if accepted, waste downstream user’s time if they have to do any manual work to pull in the “security fix” outside of routine update cycles.
If you’re using LLMs to find vulnerabilities, please build a PoC and verify it actually works before filing an advisory.