Skip to content

Remove tldextract from core dependencies to drop the requests stack #2056

Description

@vdusek

Problem

tldextract is a core dependency used in one place_utils/urls.py (_domain_under_public_suffix()), only for the opt-in same-domain enqueue strategy. The default strategy everywhere is same-hostname, so for most users it never runs. Two issues:

  1. It's the only reason the requests stack is in every install. Nothing in src/crawlee imports requests directly. Removing tldextract drops 7 packages / ~1.8 MB:

    tldextract, requests, requests-file, filelock, urllib3, certifi, charset-normalizer
    (idna stays — needed by yarl)
    
  2. It hits the network on first use. TLDExtract(cache_dir=tempfile.mkdtemp()) + tldextract defaults → the first same-domain lookup tries to fetch the PSL over the network before falling back to the bundled snapshot (and writes a disk cache, which breaks on read-only filesystems).

What we actually need

Given a host, get its registrable domain (eTLD+1) from the Public Suffix List. For same-domain we only compare two hosts for equality.

Options (with runtime dep counts)

# Approach New deps Effect
A Vendor a PSL snapshot + ~40-line matcher (like Crawlee JS / tldts) 0 removes all 7 pkgs, no network; we refresh snapshot via CI
B1 Swap for publicsuffixlist 1 (0 transitive) net −6 pkgs; pure Python, bundles PSL, auto-updated daily, actively maintained
B2 Swap for tld 1 (0 transitive) net −6 pkgs; pure Python, bundles PSL, slower cadence
B3 Swap for publicsuffix2 1 (0 transitive) net −6 pkgs; unmaintained since 2019 — avoid
C Lazy-import + move tldextract to an extra 0 default / 7 for same-domain keeps network fetch, only hides the problem
D Naive "last two labels", no PSL 0 removes all 7 pkgs but breaks .co.uk, github.io, … — not acceptable

Recommendation: A (vendored snapshot) — zero deps, no network, full control. If we'd rather not maintain a snapshot, B1 (publicsuffixlist) is the best fallback.

same-domain behavior must stay correct for multi-part suffixes; add test cases in tests/unit/_utils/test_urls.py.

Metadata

Metadata

Assignees

No one assigned

    Labels

    debtCode quality improvement or decrease of technical debt.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions