Flight Log / mendableai/firecrawl #1880
Fix robots.txt HTML filtering to check content structure — a Devin (Cognition) bot flight
The Devin bot (Cognition) opened this pull request in mendableai/firecrawl on Jul 29, 2025; merged clean in 16 min, merged by @micahstairs: +1 −1, 1 file, 3 reviews.
Flown by
- Lines
- +1 −1
- Files
- 1
- Commits
- 1
- Reviews
- 3
- To merge
- 16 min
- Task
- fix
Line and file counts are summed per commit from the AIDev dataset; GitHub's net diff can be smaller.
Devin fixed a bug in mendableai/firecrawl where valid robots.txt files served as text/html were filtered out, so checkRobotsOnScrape=true failed silently for those sites. The change was one file, one line added, one removed, in one commit. Three reviews were recorded, with zero change requests and zero review comments. The PR merged clean after 0.27 hours.
From the pull request description
# Fix robots.txt HTML filtering to check content structure ## Summary This PR fixes a bug where valid robots.txt files served with incorrect `text/html` content-type headers were being filtered out and ignored, causing `checkRobotsOnScrape=true` to silently fail for such sites. **Root Cause**: Th…
Excerpt (first 300 characters) — the full description is on GitHub.
View the pull request on GitHub →
This is your flight? Claim it — sign in with GitHub. Ownership is proven by the sign-in: the account that opened the PR is the only one that can claim it.
Flight timeline
- —committedDevin AI
- —reviewed
- —reviewed
- —reviewed
- Jul 29, 2025mergedmicahstairs
- Jul 29, 2025closedmicahstairs
Read next
Cite this flight: AgentRiders Flight Log, mendableai/firecrawl#1880, record updated Nov 4, 2026, https://agentriders.com/logs/mendableai/firecrawl/1880/
Data from public GitHub activity; this page is not affiliated with GitHub or the repository.