github-featured / thu-maic--openmaic

发布

0 发布 73 Git 标签

RSS 订阅源
  • @openmaic/storage@0.2.4 6825363de1
    比较

    feat(storage): server asset registry over a pluggable byte layer (#1007) (#1077)

    frostbyte_neo 发布于 2026-08-10 04:30:06 +00:00

    • docs(storage): specify the asset registry HTTP contract (#1007)

    Adds the contract document and the server-side store, byte-layer, and
    handler types for the asset layer, the only layer with no server backend.
    Runtime behaviour is unchanged: the new modules are deliberately
    unreferenced until the backends land.

    Reclamation moves offline, and that decision carries the design. remove
    deletes a registry entry and nothing else -- it never reads, counts, or
    deletes bytes. A separate collector takes byte rows that have had no
    references for a grace period. Three things follow. The hardest race in
    the design disappears, because no request can delete bytes another
    request is adopting, so no request path needs a lock on the byte row and
    the write path never has to reconcile taking one with never branching on
    whether the bytes already existed. A side channel closes, because a
    remove that only deletes a row costs the same whether or not another
    principal holds the same bytes. And nothing is given up: under global
    deduplication remove could never promise the bytes were destroyed, so
    deleting them synchronously bought far less than it appeared to. The cost
    is that storage grows until the collector runs, which a deployment must
    configure rather than inherit.

    Offline reclamation is also what makes the byte layer pluggable, so this
    keeps #1007's pluggable-backend model rather than narrowing it. With
    collection out of every request path, the requirement on a byte layer is
    an ordering rule rather than an atomicity one: bytes are written before
    the row that references them, deleted after the last row referencing
    them, and the reference count is serialized on the blob row that every
    implementation keeps regardless of where bytes sit. Two implementations
    are planned -- bytes in a column of the transactional store, and an
    object store keyed by content hash -- which is why the interface exists;
    an interface with one implementation is not a seam, and this package has
    removed such a seam before. The browser backend keeps no such seam,
    because it reclaims inline and its bytes therefore cannot leave its
    transaction; four comments that stated this as a storage difference now
    state it as a reclamation one.

    Metadata crosses the wire as a multipart part. It is open-ended and
    callers already fill it with generated-narration text and image prompts,
    so a query parameter would put unbounded user content into the request
    target and into every intermediary's logs. That also motivates a rule
    the other layers do not need: metadata must never appear in a URL, a
    response header, a log line, or an error message. Metadata gets a value
    domain, and the document is explicit that this is a capability
    reduction -- the browser backend's structured clone carries Date, Map,
    Set, and cycles faithfully, so a deployment moving to a server backend
    must audit its metadata rather than expect a quietly lossy path to
    tighten.

    Resolution yields a client-minted object URL, superseding the resolution
    that a server deployment resolves to a self-hosted proxy path. A media
    element sends only ambient credentials, so a proxy path works for
    cookie-session deployments and silently fails for header-authenticated
    ones; closing that gap means minting URL tokens, a second credential
    form inside a storage library. Fetching bytes over the client's own
    authenticated request needs none and removes a class of hash-disclosure
    surface. The client obligations that make it preserve the browser
    backend's semantics are specified, and split into the two the shared
    suite already pins and the four an HTTP backend must pin itself. The
    cost, no range requests and no progressive playback, is stated.

    Channels that could carry a principal or a content hash are closed
    structurally rather than by enumeration: the route table admits no
    segment that could carry either, and no route takes a query string at
    all. Headers are closed by not reading them -- the contract assigns no
    meaning to any header beyond content type and length, and a handler must
    not derive identity from one outside its authenticate hook. Rejecting
    header names by pattern was tried and removed: no pattern broad enough
    to catch X-Remote-User spares User-Agent, and the forwarded-identity
    headers such a rule would reject are exactly what an authenticating
    reverse proxy injects for that hook to read.

    Ids that cannot be a path segment are resolved by the client as misses
    rather than raised as errors, a deliberate divergence from the KV
    contract. An unknown id is a miss by this layer's id-domain rule and the
    shared suite pins the empty string as one, so a client that threw would
    fail that suite and reintroduce the shape oracle the rule prevents.

    Also specified: response values may not derive from the shared byte row
    (Last-Modified off that row is an existence oracle), a collision-
    resistant digest is required because writes are unconditional and an
    object layer keys on that digest, byte responses carry the revision and
    refuse Range, HEAD errors carry the code in a header since they have no
    body, quota exhaustion has a status and a code, and relabelling a
    non-renderable type is not refusal -- resolve still returns a URL.

    • feat(storage): add pluggable server asset store (#1007)

    • fix(storage): write asset bytes only after claiming the blob row (#1007)

    The write path wrote bytes twice: once before the registry transaction
    and once inside it, after the upsert that claims the blob row. Only the
    second one carries any safety, and the first is what a large-media
    deployment pays for on every upload.

    The ordering that matters is claim, write, reference. Claiming the blob
    row takes its lock, so a collector already holding that lock finishes
    before the write proceeds; writing bytes before the claim instead lets
    the collector delete them while the claim waits, leaving a fresh entry
    that points at nothing. Writing them before the entry keeps every
    surviving entry backed by bytes that were actually stored. Dropping the
    first write preserves both ends of that order and removes a crash window
    that produced orphans for no reason.

    For a byte layer inside the transactional store this also means a failed
    write now leaves nothing at all, rather than an orphan the collector had
    to sweep later.

    The four tests this changed were each pinned to the removed write:

    • the statement-sequence test hardcoded the old four-statement shape; it
      now pins claim, write, entry, and still asserts the two existence paths
      are identical, which is the property that matters;
    • the quota test asserted two writes per accepted put;
    • the crash-window test asserted an orphan that a transactional byte
      layer no longer produces. It is now two tests: a transactional layer
      leaves nothing, and a non-transactional one strands an object with no
      blob row -- which the collector deliberately cannot see, since
      recovering it is deployment housekeeping rather than reference
      counting. It also no longer depends on state left behind by earlier
      tests in the file;
    • the collector-race test drove its interleaving off the removed write.
      It was choreography rather than concurrency in any case: PGlite is
      single-connection, and the test serialized transactions, so no row lock
      was ever contended. It is replaced by the two invariants that actually
      make the race safe -- the byte write is unconditional, and bytes the
      collector has already removed are re-stored by an adopting put. The
      contended-lock case belongs to the real-PostgreSQL suite, where it can
      be tested rather than mimed.

    Reverting the unconditional write fails two of them, so they pin it.

    The contract said a failed write leaves no partially written bytes while
    its byte-layer section permits exactly that orphan; the two are
    reconciled, and the ordering rule now states the claim step and the
    unconditional write it depends on, along with the cost that follows --
    the byte write happens with the registry transaction open, so an object
    store holds a row lock across a network upload.

    • test(storage): fix the real-PostgreSQL asset tests against the new write order

    Both were written against the removed pre-transaction byte write, and
    both failed once it went away -- one of them by hanging, which also
    timed out the next test's TRUNCATE.

    The lock-contention test waited for the adopting put to start writing
    bytes before releasing the collector. Bytes are now written after the
    upsert that claims the blob row, and the collector holds that lock, so
    the write could never start: a circular wait. It now waits for a backend
    to actually appear in pg_stat_activity blocked on a lock, which is the
    condition it meant to wait for, observed rather than signalled. Its final
    assertion carries real weight now -- had the bytes been written before
    the claim, the collector would have deleted them and the adopting entry
    would resolve to nothing.

    The orphan test asserted that a failed registry transaction strands
    bytes. With this byte layer inside the registry's transaction it strands
    nothing, so it now asserts that instead. Object storage does strand an
    object, and that case is covered where it belongs.

    All of this ran against PostgreSQL 16: 781 tests pass with 3 skipped,
    and those 3 pass against a real S3-compatible server. Nothing in the
    suite is now unverified for lack of infrastructure.

    • feat(storage): add asset HTTP backend (#1007)

    • docs(storage): separate trusting a header from reading one (#1007)

    The header rule said a server reads exactly Content-Type and
    Content-Length, while the size rules require reading Content-Encoding in
    order to reject it -- the two cannot both be literal, and the
    implementation had to pick one.

    What the rule means is narrower: no request header may be trusted to say
    who is asking, outside the authenticate hook. Reading a header to decide
    how to frame or refuse a request is a different thing, and the transport
    headers are read and acted on.

    • chore(storage): bump to 0.2.3 for the asset server backend (#1007)

    Additive: new asset registry, byte layers, collector, HTTP handler and
    client, and their entry points. Nothing existing is removed or changed
    incompatibly, so this is a patch under the pre-1.0 rule.

    • fix(storage): close four defects found reviewing the asset backend (#1007)

    Concurrent writes could exceed a principal's logical quota. The check
    ran on the pool before the write transaction, so two concurrent puts both
    read the old total and both passed; enough concurrency amplified it
    arbitrarily. It now runs inside the write transaction behind a
    transaction-scoped advisory lock on the principal, taken only when a quota
    is configured -- a branch on deployment configuration, never on data.
    Pinned by a real-PostgreSQL test: four concurrent six-byte writes against
    a ten-byte quota, of which exactly one may be accepted. Removing the lock
    accepts three.

    A part name smuggled inside a quoted filename was read as a real part
    name.     Content-Disposition was scanned with a regex that does not
    understand quoted strings, so filename="x; name=meta; y" parsed as a
    name parameter here while an RFC-aware intermediary sees an unnamed
    part. That is exactly the parser differential the contract requires be
    rejected. Replaced with a tokenizer that honours quoted strings and
    escapes and requires exactly one real name.

    Any S3 404 was reported as an absent object. NoSuchBucket, a
    misdirected endpoint, and a revoked access point all answer 404 while the
    bytes still exist, so a storage outage surfaced as 404 ASSET_NOT_FOUND
    and a caller clearing its reference on that would turn it into real data
    loss -- the same failure the contract spells out for 401. Only
    key-absent codes map to a miss now; everything else propagates to
    500 INTERNAL_ERROR.

    Exceeding maxParts answered 400 rather than 413. It is one of
    the declared multipart resource limits and now answers like the other
    three.

    Verified against PostgreSQL 16 and a real S3-compatible server:
    840 tests, none skipped.

    • fix(storage): repair two defects the previous fix round introduced (#1007)

    Both were created by the fixes themselves rather than surviving them,
    which is the failure mode a second review round exists to catch.

    An over-quota replace answered 500. Moving the quota check inside the
    write transaction put it under a catch that re-threw only
    AssetNotFoundError, so AssetQuotaExceededError was collapsed into a
    generic registry failure and the handler mapped it to INTERNAL_ERROR --
    losing the status and code the contract gives that condition. Both typed
    errors now survive the catch.

    The hand-written disposition tokenizer accepted malformed input and
    trusted it.     The quoted-value loop never checked that it found a closing
    quote, so name="meta ran to the end of the header and returned meta;
    name="meta"junk returned meta as well; and an extended name* form
    competing with a plain one was silently ignored. Each recreates the
    parser differential the tokenizer was written to remove. A quoted value
    now requires its closing quote, rejects a dangling escape, and permits
    only whitespace and a separator after it, and an extended name form is
    refused rather than resolved.

    Also widened the quota lock key from hashtext to hashtextextended.
    The 32-bit form collides -- two unrelated principals sharing a key would
    block each other for the whole transaction, which spans a byte write that
    may be a network upload.

    Regression tests cover all three: an over-quota replace raises the quota
    error and leaves the original bytes, and each of the three malformed
    dispositions is refused.

    Verified against PostgreSQL 16 and a real S3-compatible server:
    842 tests, none skipped.

    • fix(storage): narrow the multipart disposition surface and the error boundary (#1007)

    Three review rounds have each found a defect in the part-disposition
    parser: a regex that misread quoted strings, then a tokenizer that
    accepted an unterminated quote, then RFC 2231 continuation forms
    (name*0*=UTF-8''bytes) and malformed empty parameter slots. Patching it
    a fourth time would have been the wrong move.

    This contract needs exactly one parameter, so it now accepts exactly one:
    a part disposition is form-data plus a single name whose value is
    meta or bytes, and everything else is a validation failure. That
    removes the continuation forms, the encoded forms, the empty slots, and
    filename -- the parameter every one of these attacks travelled in --
    without having to model RFC 2231 at all.

    That rule turned out to bind our own client too: it emitted a fixed
    filename="asset". Nothing reads it, and the contract already forbids
    deriving a response disposition filename from caller data, so the client
    no longer sends one.

    Separately, instanceof was being used as provenance. The byte layer is
    pluggable, so a byte store raising a same-named AssetQuotaExceededError
    or AssetNotFoundError inside the transaction was re-thrown as a logical
    registry outcome, carrying the byte layer's own message to a direct
    caller. The registry's checks now raise module-private sentinels, and
    only those are converted -- at the boundary, into fresh public errors
    with this package's fixed messages. Anything else, whatever its name,
    collapses to the generic registry failure.

    Also pinned the advisory-lock key width. Reverting hashtextextended to
    the 32-bit hashtext left every test green, so the emitted lock
    statement is now asserted. A SQL-text assertion is deliberate here: the
    behavioural difference is a collision between two particular keys under a
    database-internal hash, which is not a stable thing to assert against.

    Verified against PostgreSQL 16 and a real S3-compatible server:
    845 tests, none skipped.

    • fix(storage): define multipart disposition grammar (#1007)

    • fix(storage): delegate multipart parsing to Fetch (#1007)

    • fix(storage): require both multipart parts to be files (#1007)

    Delegating the framing to the platform parser cost two rules for a
    metadata part sent without a filename: such a part comes back as a
    string with its headers discarded, so its application/json type could
    no longer be checked, and its bytes had already been replacement-decoded
    -- {"x":"<0xFF>"}" was stored as {"x":"\uFFFD"}` instead of refused.

    The bytes part already had to be a file for binary safety. Metadata is
    now symmetric: the client sends a fixed filename on both, the server
    requires both to be files, and with the part preserved the media type is
    checked and the bytes are decoded fatally before parsing. Neither
    filename is read anywhere.

    The limits are also described honestly now. maxParts, maxMetaBytes
    and maxAssetBytes are validated after the parser has materialized
    every part, so they bound what is accepted rather than what is parsed,
    and maxRequestBytes is what caps memory. The contract said maxParts
    bounded parser work; it does not. Restoring that would mean adopting a
    streaming parser with its own grammar, which is the disagreement with
    intermediaries that delegating to the platform exists to remove -- so
    the trade is stated rather than reversed.

    Also corrected the media-type wording: the two branches are metadata
    present versus absent, and retention of an untyped replacement's prior
    type is a browser-backend behaviour the HTTP path cannot reproduce,
    because a conforming parser supplies a default type for a file part
    whose header omits one.

    Verified against PostgreSQL 16 and a real S3-compatible server:
    853 tests, none skipped.

    • docs(storage): the asset server backend ships in this branch (#1007)

    The contract still described the server backend as not yet shipped, which
    this branch is what changes. The conformance server remains test-only.

    • fix(storage): make asset HEAD bytes-free and pin reads (#1007)

    • fix(storage): harden asset write boundaries (#1007)

    • chore(storage): bump to 0.2.4 for the asset server backend (#1007)

    main merged in a 0.2.3 of its own, so this branch's bump no longer
    increased the version. Additive relative to the merged base -- new asset
    registry, byte layers, collector, HTTP handler and client, with nothing
    removed or narrowed -- so a patch bump under the pre-1.0 rule.

    Co-authored-by: 杨慎 117187635+cosarah@users.noreply.github.com

    下载附件