feat: add lazy multipart form streaming

[fpawel]

Summary

This PR adds a runtime-level prototype for sequential lazy processing of
multipart/form-data.

File payloads are exposed directly from the HTTP request body without buffering
the complete multipart form and without background goroutines.

This is the runtime foundation discussed in:

• Proposal: support opt-in streaming for multipart file uploads go-swagger/go-swagger#3407
• Refs Proposal: support lazy streaming multipart file binding #506

API and lifecycle

• MultipartFormStream.NextFile() advances sequentially to the next file part.
• Ordinary form fields are added to request.PostForm and request.Form as
  they are encountered.
• Only one file reader may be active at a time.
• StreamedFile.Close() drains only the current file part.
• MultipartFormStream.Drain() consumes all remaining parts, collects trailing
  fields, and closes the request body.
• MultipartFormStream.Close() stops multipart processing and closes the
  request body without explicitly draining the remaining multipart parts.
• No background goroutines are created.
• MultipartFormStream.Fields() returns a snapshot of multipart fields
  discovered so far.
• MultipartFormStream.Files() returns a snapshot of file metadata discovered
  so far, in wire order.
• The discovered state does not include parts that have not yet been reached.
• Generated binding only constructs and passes the stream; traversal,
  multiplicity and application-specific validation belong to the handler.

The existing BindForm behavior is unchanged.

Covered cases

• file payload exposed before the complete body arrives;
• fields before, between, and after files;
• multiple file parts;
• partial reads and automatic draining before advancing;
• explicit drain and immediate abort;
• request body and filename/file-count limits;
• malformed multipart input;
• reads attempted after request context cancellation;
• read and close error propagation.

Checks

go test ./...
golangci-lint run --new-from-rev upstream/master
git diff --check

[fredbi]

Could only see a few nits, essentially about redundant parts with the existing form options.
Other than that I now fully get that we definitely need a new type with a new interface.

So it looks great. I like the care taken to closing and draining.
one question I couldn't figure out by just reading the code: how would your reader fare if wrapped in a LimitHandler that interrupts the consumption of the body?
I suppose it would just bubble the error, but can't be sure.

A good playground to check how it would operate may be to modify this example there: https://github.com/go-swagger/examples/tree/master/file-server/restapi/operations. That's a minimal example of a file upload server.
So I assume your new Reader would be bound there: https://github.com/go-swagger/examples/blob/869b7a290ed6a55fe1b09e6cdb962cee206b9510/file-server/restapi/operations/uploads/upload_file_parameters.go#L44
and https://github.com/go-swagger/examples/blob/869b7a290ed6a55fe1b09e6cdb962cee206b9510/file-server/restapi/operations/uploads/upload_file_parameters.go#L55

and that the custom handler manipulating the new reader could be demonstrated there https://github.com/go-swagger/examples/blob/869b7a290ed6a55fe1b09e6cdb962cee206b9510/file-server/restapi/configure_file_upload.go#L59

so far looks good. great work

Fred

[fredbi]

// NOTE(fred): could embed config

[fpawel]

Done. MultipartFormStream now embeds multipartFormStreamConfig instead of duplicating its fields.

[codecov]

Codecov Report

❌ Patch coverage is 83.58974% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.37%. Comparing base (d801284) to head (3590828).
⚠️ Report is 2 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
multipart_stream.go	83.33%	18 Missing and 14 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #507      +/-   ##
==========================================
- Coverage   83.43%   83.37%   -0.07%     
==========================================
  Files          64       65       +1     
  Lines        4521     4715     +194     
==========================================
+ Hits         3772     3931     +159     
- Misses        581      601      +20     
- Partials      168      183      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[fpawel]

Could only see a few nits, essentially about redundant parts with the existing form options. Other than that I now fully get that we definitely need a new type with a new interface.

So it looks great. I like the care taken to closing and draining. one question I couldn't figure out by just reading the code: how would your reader fare if wrapped in a LimitHandler that interrupts the consumption of the body? I suppose it would just bubble the error, but can't be sure.

A good playground to check how it would operate may be to modify this example there: https://github.com/go-swagger/examples/tree/master/file-server/restapi/operations. That's a minimal example of a file upload server. So I assume your new Reader would be bound there: https://github.com/go-swagger/examples/blob/869b7a290ed6a55fe1b09e6cdb962cee206b9510/file-server/restapi/operations/uploads/upload_file_parameters.go#L44 and https://github.com/go-swagger/examples/blob/869b7a290ed6a55fe1b09e6cdb962cee206b9510/file-server/restapi/operations/uploads/upload_file_parameters.go#L55

and that the custom handler manipulating the new reader could be demonstrated there https://github.com/go-swagger/examples/blob/869b7a290ed6a55fe1b09e6cdb962cee206b9510/file-server/restapi/configure_file_upload.go#L59

so far looks good. great work

Fred

@fredbi Thanks, Fred. I’ve pushed a follow-up addressing the review comments.

I also added handler-level tests with http.MaxBytesHandler, with the stream’s own body limit disabled. The resulting *http.MaxBytesError is propagated both when reading the file payload and when StreamedFile.Close() drains the unread remainder.

The file-server example makes sense as an integration playground.

After the runtime API review is settled, I’ll prepare a separate prototype against that example to validate the generated binding, handler ownership, and MaxBytesHandler behavior.

Pawel.

[fpawel]

I’ve prepared an integration playground based on the file-server example:

go-swagger/examples#71

It includes a black-box httptest.Server/io.Pipe test showing that the handler
starts consuming the streamed file before the client completes the multipart
body. The binder changes are manual and represent the intended future codegen.

[fredbi]

All good. Currently the debate has moved to how a handler can use the streamer exposed here and cover all hybrid/mixed cases without banging their head too much. So far excellent implementation.
Just waiting for the example usage to settle before the final validation.

Note: adapting the “untyped API” to benefit from this improvement may be deferred to a future improvement. We’ll focus on the “generated API” first.

[fpawel]

I’ve pushed the handler-owned version we discussed.

Generated binding now only constructs and passes MultipartFormStream. It no longer stores a file, advances the stream, or performs required/multiplicity validation.

The handler owns traversal and application-specific validation. The runtime Fields() and Files() accessors expose snapshots of the multipart state discovered so far.

The black-box test still verifies that the handler starts consuming the file before the client completes the multipart request body, and now also exercises the discovered state.