Skip to content

feat(volc_mysql): add VolcMySQL backend with HNSW vector index support - #804

Merged
XuanYang-cn merged 2 commits into
zilliztech:mainfrom
FishMage:volc_mysql
Jul 6, 2026
Merged

feat(volc_mysql): add VolcMySQL backend with HNSW vector index support#804
XuanYang-cn merged 2 commits into
zilliztech:mainfrom
FishMage:volc_mysql

Conversation

@FishMage

Copy link
Copy Markdown
Contributor

Add a vector database backend for VolcMySQL (Volcano Engine MySQL with native VECTOR type and HNSW vector index), connecting via mysql-connector-python over the MySQL wire protocol.

Components:

  • volc_mysql.py: VectorDB implementation with VECTOR(dim) table, bulk LOAD DATA LOCAL INFILE loading, and CREATE VECTOR INDEX via SECONDARY_ENGINE_ATTRIBUTE (algorithm/distance/m/ef_construction and optional quant_algorithm/quant_type/refine_type for SQ/PQ/RaBitQ)
  • config.py: DBConfig (host/port/user/password) and VolcMySQLHNSWConfig (m/ef_search/ef_construction + quantization params)
  • cli.py: Click command VolcMySQLHNSW

Registration:

  • Add VolcMySQL to the DB enum in backend/clients/init.py with lazy imports for init_cls, config_cls, and case_config_cls
  • Register VolcMySQLHNSW CLI command in cli/vectordbbench.py
  • Add volc_mysql optional dependency in pyproject.toml

Binary VECTOR path with auto-probe fallback:

  • Vectors are sent as raw little-endian float32 bytes (UNHEX on load, the _binary introducer on query), avoiding to_vector() text parsing and Python str() formatting; _binary literals stay constant-foldable so the HNSW index scan is preserved
  • init() probes once per connection (session-local TEMPORARY TABLE) whether the server accepts the binary path and transparently falls back to the to_vector() text path -- for both insert and query -- when it does not. VDB_BINARY_VEC overrides the probe (1=force binary, 0=force text)

Tests: tests/test_volc_mysql_encoder.py covers binary and text TSV encoding.

@FishMage

Copy link
Copy Markdown
Contributor Author

/assign @XuanYang-cn

@XuanYang-cn

Copy link
Copy Markdown
Collaborator

@FishMage Please fix the ut to pass the github actions

@FishMage
FishMage force-pushed the volc_mysql branch 2 times, most recently from 53723e4 to 3856312 Compare July 3, 2026 03:00
@XuanYang-cn

Copy link
Copy Markdown
Collaborator

@FishMage THe code check still fails, PTAL

Add a VectorDB backend for VolcMySQL (native VECTOR type and HNSW index)
over the MySQL wire protocol via mysql-connector-python:

- client, config, and Click CLI command; registered in the DB enum and CLI
- per-thread connections (thread_safe=False, one connection per worker)
- NonFilter and NumGE filter support via prepared SQL templates
- binary float32 vector path with per-connection auto-probe fallback to
  to_vector()
- unit tests for TSV encoding, config, filters, and init teardown

@XuanYang-cn XuanYang-cn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: FishMage, XuanYang-cn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@XuanYang-cn
XuanYang-cn merged commit 3bc1be4 into zilliztech:main Jul 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants