Problem
tldextract is a core dependency used in one place — _utils/urls.py (_domain_under_public_suffix()), only for the opt-in same-domain enqueue strategy. The default strategy everywhere is same-hostname, so for most users it never runs. Two issues:
-
It's the only reason the requests stack is in every install. Nothing in src/crawlee imports requests directly. Removing tldextract drops 7 packages / ~1.8 MB:
tldextract, requests, requests-file, filelock, urllib3, certifi, charset-normalizer
(idna stays — needed by yarl)
-
It hits the network on first use. TLDExtract(cache_dir=tempfile.mkdtemp()) + tldextract defaults → the first same-domain lookup tries to fetch the PSL over the network before falling back to the bundled snapshot (and writes a disk cache, which breaks on read-only filesystems).
What we actually need
Given a host, get its registrable domain (eTLD+1) from the Public Suffix List. For same-domain we only compare two hosts for equality.
Options (with runtime dep counts)
| # |
Approach |
New deps |
Effect |
| A |
Vendor a PSL snapshot + ~40-line matcher (like Crawlee JS / tldts) |
0 |
removes all 7 pkgs, no network; we refresh snapshot via CI |
| B1 |
Swap for publicsuffixlist |
1 (0 transitive) |
net −6 pkgs; pure Python, bundles PSL, auto-updated daily, actively maintained |
| B2 |
Swap for tld |
1 (0 transitive) |
net −6 pkgs; pure Python, bundles PSL, slower cadence |
| B3 |
Swap for publicsuffix2 |
1 (0 transitive) |
net −6 pkgs; unmaintained since 2019 — avoid |
| C |
Lazy-import + move tldextract to an extra |
0 default / 7 for same-domain |
keeps network fetch, only hides the problem |
| D |
Naive "last two labels", no PSL |
0 |
removes all 7 pkgs but breaks .co.uk, github.io, … — not acceptable |
Recommendation: A (vendored snapshot) — zero deps, no network, full control. If we'd rather not maintain a snapshot, B1 (publicsuffixlist) is the best fallback.
same-domain behavior must stay correct for multi-part suffixes; add test cases in tests/unit/_utils/test_urls.py.
Problem
tldextractis a core dependency used in one place —_utils/urls.py(_domain_under_public_suffix()), only for the opt-insame-domainenqueue strategy. The default strategy everywhere issame-hostname, so for most users it never runs. Two issues:It's the only reason the
requestsstack is in every install. Nothing insrc/crawleeimportsrequestsdirectly. Removingtldextractdrops 7 packages / ~1.8 MB:It hits the network on first use.
TLDExtract(cache_dir=tempfile.mkdtemp())+ tldextract defaults → the firstsame-domainlookup tries to fetch the PSL over the network before falling back to the bundled snapshot (and writes a disk cache, which breaks on read-only filesystems).What we actually need
Given a host, get its registrable domain (eTLD+1) from the Public Suffix List. For
same-domainwe only compare two hosts for equality.Options (with runtime dep counts)
tldts)publicsuffixlisttldpublicsuffix2tldextractto an extrasame-domain.co.uk,github.io, … — not acceptableRecommendation: A (vendored snapshot) — zero deps, no network, full control. If we'd rather not maintain a snapshot, B1 (
publicsuffixlist) is the best fallback.same-domainbehavior must stay correct for multi-part suffixes; add test cases intests/unit/_utils/test_urls.py.