feat: faster replication

[yogesh1801]

fixes #1036

[yogesh1801]

I am still working on the design for it, just a peak if the direction seems well @levkk

[levkk]

Cool! Would be good to see some benchmark numbers, just to get a sense of how much faster this is.

[yogesh1801]

Do you have any such setup so that i can benchmark it?

[codecov]

Codecov Report

❌ Patch coverage is 78.35821% with 58 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...backend/replication/logical/subscriber/pipeline.rs	78.37%	48 Missing ⚠️
...c/backend/replication/logical/subscriber/stream.rs	80.00%	9 Missing ⚠️
pgdog/src/backend/replication/logical/error.rs	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[levkk]

Do you have any such setup so that i can benchmark it?

This might work:

benches/bench.sh

Specifically, see resharding folder.

[yogesh1801]

PGDOG_BIN=/tmp/pgdog-new bash benches/resharding/replication/run.sh --baseline main

[yogesh1801]

For BENCH_SCALE = 3000, DEST_LATENCY_MS=10 we have more performance boosts

[yogesh1801]

RUNS = 2
WARMUP = 1
BENCH_SCALE = 5000
DEST_LATENCY_MS = 10
USE_TOXI = 1

@levkk results seems good but also too good at a point

[levkk]

Async should be a big improvement, so not that surprising. This is very cool. Let me know when you're ready for a review, I'll tag Kiryl.

[yogesh1801]

@levkk ready for review

[meskill]

Do you think we should add more unit tests here? I think we should have some for testing the whole flow - prepare -> execute -> await. And also some tests for possible errors