Flight Log / mendableai/firecrawl #1895
feat(crawler): replace robotstxt library with texting_robots for ENG-3016 — a Devin (Cognition) bot flight
The Devin bot (Cognition) opened this pull request in mendableai/firecrawl on Jul 30, 2025; open and stale: +291 −19, 4 files, 1 review.
Flown by
- Lines
- +291 −19
- Files
- 4
- Commits
- 1
- Reviews
- 1
- To merge
- —
- Task
- feat
Line and file counts are summed per commit from the AIDev dataset; GitHub's net diff can be smaller.
From the pull request description
# feat(crawler): replace robotstxt library with texting_robots for ENG-3016 ## Summary This PR replaces the `robotstxt` library with `texting_robots` in the Firecrawl Rust crawler to fix crashes caused by malformed or non-UTF8 robots.txt content. The original `robotstxt = "0.3.0"` library had reli…
Excerpt (first 300 characters) — the full description is on GitHub.
View the pull request on GitHub →
This is your flight? Claim it — sign in with GitHub. Ownership is proven by the sign-in: the account that opened the PR is the only one that can claim it.
Flight timeline
- —committedDevin AI
- —reviewed
- Jul 31, 2025mergedmogery
- Jul 31, 2025closedmogery
Read next
Cite this flight: AgentRiders Flight Log, mendableai/firecrawl#1895, record updated Nov 3, 2026, https://agentriders.com/logs/mendableai/firecrawl/1895/
Data from public GitHub activity; this page is not affiliated with GitHub or the repository.