github-featured / thu-maic--openmaic

发布

RSS 订阅源

@openmaic/storage@0.2.5 969574bee4

比较
feat(storage): wire the asset backend into the app (#1007) (#1089)

frostbyte_neo 发布于 2026-08-11 05:01:16 +00:00
- fix(persistence): stop corrupting binary response bodies in the route adapter
The embedded persistence route bridges a Node RequestListener to the
Fetch API by hand. Its response object is cast with as unknown as ServerResponse, so the compiler checks none of that surface, and two
parts of it were wrong in ways only a byte-carrying handler would hit.

end decoded any Uint8Array chunk with Buffer.from(chunk).toString(),
which is UTF-8. ServerResponse.end accepts a Uint8Array, and those
bytes are not necessarily valid UTF-8, so every unpaired byte became
U+FFFD -- silent corruption with no error anywhere. The adapter now
buffers chunks as bytes and builds the response from them.

write was absent entirely, making any chunked handler a runtime
TypeError rather than a compile error. It is part of the surface this
object claims to implement, so it is now implemented.

Neither reaches a caller today: the document and runtime handlers only
ever end with JSON strings, which took the string branch. Both are traps
laid for the first handler that carries bytes.

Both cases are pinned by tests that fail before this change -- the
binary one on a byte comparison, the chunked one with a 500 from the
TypeError. The existing adapter round-trip test, whose comment already
called this the most bug-prone code in the route, covered only the
string path.
- feat(storage): wire the asset backend into the app (#1007)
The server asset registry shipped with #1007 but was reachable from
nothing: the persistence route never passed an assetStore, and no file
under app/, lib/ or components/ imported the asset server or client
modules. This connects it, additively -- no DSL change, no version bump,
and no behavioural change when server persistence is off.

The route now builds a PgAssetStore on the same pool and transaction as
the document store, with PgAssetByteStore by default and
S3AssetByteStore when ASSET_S3_BUCKET opts a deployment into it. The
AWS SDK stays an optional peer, reached only through a dynamic import on
that branch. The offline collector is deliberately not scheduled here,
and the route says so where a reader would look for it, because leaving
it unscheduled grows storage without bound and scheduling it is the
deployment's decision.

On the browser side the asset pool gains a single-shot configuration seam
modelled on the document store's, so that with server persistence enabled
the pool is an HttpAssetStore carrying the same auth headers as the
other stores.

Three browser-specific mechanisms needed decisions rather than
translation:

clearAssetPool deletes an IndexedDB database today. In server mode
there is nothing local to delete, and a "clear cache" action must not
remove server-side assets -- that would destroy user data from a button
promising to free space. It now revokes local object URLs and closes the
client, and deletes nothing remote.

The cross-tab replacement broadcast is unchanged. Its job is to make
another tab drop a warm object URL when the bytes behind an id change,
and that need is identical in server mode.

The exclusivity proof that decides whether an in-place replace is safe
reasons over this browser only: local documents, unflushed state, and a
cross-tab presence probe. In server mode those cover a strict subset of
the holders, since another device can reference the same id and no probe
can see it -- and asking the server who else references an id would be an
existence oracle over other principals. The proof therefore returns not
exclusive unconditionally in server mode, and regeneration takes the
existing fork path. That is the fail-closed direction and a graceful
degradation rather than a break.

The development authenticator had to grow the asset principal's required
key; without it every asset request would have been correctly denied.
It derives from the same learner partition, and a principal without one
still gets no asset access. The route now states plainly that this key
comes from a client-supplied header, so the cross-principal isolation the
asset contract describes is not in force under this authenticator -- that
warning previously lived only in the auth module's own header, where
nobody mounting the route would see it.
- fix(storage): close three defects in the asset app wiring (#1007)
The S3 branch could not load, and isolated nothing. The route
dynamically imported the storage package's S3 byte store, but that module
statically imported the AWS SDK, so merely resolving it pulled the SDK
into resolution and bundling. The route then separately performed an
ignored native import of the SDK from the app's own scope, which fails
with ERR_MODULE_NOT_FOUND because the SDK lives under the storage
workspace package -- so a deployment that set the bucket got a broken
store rather than an S3 one. There is now one lazy loader in one
resolution scope, the storage module no longer imports the SDK at module
load, and the bucket is validated at initialization rather than treated
as valid because it is non-empty. A test mocks the SDK to throw on
resolution and asserts that importing the byte-store module never
resolves it.

A cross-tab replacement could leave a mounted lease stale. The
broadcast handler re-resolves, and HttpAssetStore.resolve coalesces
concurrent calls for one id onto a single in-flight request -- correct on
its own, and required by the contract. Together they meant a resolve
triggered by the broadcast could merge into a GET that started before the
peer's replacement, return the old revision, and leave the consumer on
the old URL with nothing to dislodge it. The client gains an invalidation
that retires its cached snapshot and advances the per-id generation
without revoking any URL already issued; release would have been wrong
here, since the contract keeps an issued URL valid until its holder
releases it. Removing the invalidation call fails the new race test.

Clearing a server-backed pool left the closed client installed. In
server mode clearAssetPool closed the client and returned before
clearing the module singleton, so every later getAssetPool() handed
back a closed store until a page reload happened to intervene. Only the
IndexedDB deletion is skipped in server mode now; the singleton is always
cleared so the next call rebuilds through the configured factory.

Also adds the tests whose absence let these through: a route round-trip
carrying invalid UTF-8, the SDK-resolution assertion above, the broadcast
race, and reopening the pool after a clear. The earlier tests were not
wrong so much as aimed past the defect -- the route test used JSON text,
and the S3 test injected a loader rather than exercising the imports.
- chore(storage): bump package version to 0.3.0
- fix(persistence): make the route adapter faithful where it carries bytes
The embedded persistence route bridges a Node RequestListener to the
Fetch API by hand, and its response object is cast
as unknown as ServerResponse, so the compiler checks none of that
surface. Five divergences are closed here; the first two are the ones
that corrupt data, the last three were raised in review.

end decoded bytes as text. It ran Buffer.from(chunk).toString()
on a Uint8Array chunk, which is UTF-8, so every byte outside that
encoding became U+FFFD -- silent corruption with no error anywhere. The
adapter now buffers chunks as bytes.

write was absent, making any chunked handler a runtime TypeError
rather than a compile error.

write callbacks ran inline. Node invokes them after the chunk is
handed off. A handler that writes its next chunk from each callback
therefore recursed synchronously and could overflow the stack, and a
throwing callback threw in the wrong execution phase. They are now
deferred.

The encoding argument was ignored. write(s, 'latin1') emitted
0xc3 0xa9 where Node emits 0xe9, corrupting the body and potentially
invalidating a Content-Length the handler had set. Both write and
end now parse their full overload set and pass the encoding through.
Node's own behaviour for an invalid encoding was established by probing
ServerResponse on Node 22 rather than assumed: it throws
ERR_UNKNOWN_ENCODING synchronously, so the adapter validates with
Buffer.isEncoding and lets Buffer.from raise the native error.

Bodyless statuses were not suppressed. Node discards writes after
writeHead(204) or writeHead(304); this passed the buffered bytes to
the Fetch Response constructor, which throws for those statuses, so the
request became a 500 instead of the intended 204 or 304. Buffered data is
now discarded for 204, 205 and 304, and for HEAD.

None of the five reaches a caller today -- the document and runtime
handlers only ever end with JSON strings. They are traps laid for the
first handler that carries bytes, which is why they surfaced while
wiring the asset server backend, whose byte routes hit the first two
immediately.

Each is pinned by a test that fails when its fix is reverted.
- fix(storage): keep { client, bucket } working for the S3 byte store
S3AssetByteStoreOptions.commands was required and the store's methods
called it directly, so the published ./asset/s3-bytes export stopped
accepting the { client, bucket } construction it had always taken.
Anyone not going through loadS3AssetByteStore either failed to compile
or hit an undefined command factory at runtime.

Make commands optional. Omitted, the store resolves the AWS SDK's own
command constructors through a dynamic import on its first write /
read / delete and caches them, so { client, bucket } works again.
Importing the module and constructing a store still never reach the
optional peer dependency: a deployment that does not select S3 never
resolves it. When the SDK cannot be resolved the call rejects naming
@aws-sdk/client-s3 rather than crashing on an undefined property, and
the resolution is left uncached so installing the dependency fixes an
already-constructed store.

The lazily bound store is run against the byte-store contract, and the
new isolation tests pin both that construction does not resolve the SDK
and that an unresolvable SDK is reported by name.

With the break gone this branch's change to the package is additive, so
the version becomes a patch above main (0.2.4 -> 0.2.5) rather than the
minor it was carrying.

Refs #1007

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
- feat(persistence): run the asset collector in the shipped deployment
PgAssetStore.remove, and a replace that changes content, only stamp
asset_blobs.unreferenced_at. AssetCollector.collect is the only path
that deletes anything, and the persistence route deliberately did not
schedule it, on the grounds that scheduling is the deployment's decision.

The deployment this repository ships is docker-compose.yml: the app and
PostgreSQL, and nothing else that could make that decision. "The
deployment decides" therefore meant "it never runs", and ordinary asset
churn retained PostgreSQL bytes or S3 objects forever.

Schedule it from instrumentation.ts, which Next runs once per server
process — a route module can be instantiated more than once and has no
shutdown hook, so a schedule started from one is really started per
instantiation. Next 15 and later pick the file up with no config, so
next.config.ts is unchanged.

Collection is on by default with a 15-minute interval and the package's
one-hour grace period, so the Compose stack is correct with no operator
action. ASSET_COLLECTION_INTERVAL_MS, ASSET_COLLECTION_GRACE_MS, and
ASSET_COLLECTION_ENABLED change or disable it, each documented where it
is read. Without DATABASE_URL nothing is scheduled at all, and a failed
pass is logged and retried on the next tick rather than ending the
schedule or the process.

Several instances may collect concurrently: each candidate blob row is
re-checked and locked FOR UPDATE inside its own transaction, so
collectors serialize on the row rather than racing. The comment says so,
so nobody adds a distributed lock that is not needed.

The byte-store selection moves to lib/persistence/asset-byte-store so the
collector reclaims through the same layer the request path wrote through.
A collector on the PostgreSQL byte store while the route wrote to S3
would drop the blob row and orphan the object permanently.

Refs #1007

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
- feat(storage): bound each collector pass and report whether it filled
An unbounded collect() is sized by however long a deployment ran before
collection was scheduled: one statement selecting every eligible blob,
then a transaction and a byte-layer delete each, in a loop nothing
interrupts. The first pass over the Compose deployment that grew a
backlog before collection was scheduled is exactly that pass.

A pass now takes at most batchSize blobs (default one thousand) and
returns. Ordinary churn between two scheduled passes is far below the
cap, so a healthy deployment behaves as it did when a pass was
unbounded; a capped pass costs the remainder one scheduling interval,
which is what the interval is for.

collect() still answers with the count, and the count alone cannot tell
an empty backlog from a full batch -- a re-referenced or concurrently
taken candidate is skipped, so even a full batch can collect less than
batchSize. collectPass() returns the count together with capped, which
is exactly "this batch was full, run again"; a caller draining a
backlog loops while it is true.

Candidates are taken oldest-unreferenced first, with the content hash
as tiebreaker within one timestamp. Ordering by the hash alone would
starve: digests are uniformly distributed, so a high-sorting blob
waits behind every lower digest stamped after it, and under steady
arrivals those keep coming. The tests pin the ordering against a
planner that happens to answer the right rows without an ORDER BY.

Refs #1007

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
- fix(persistence): keep asset backend faults off the shared handler
Two review findings on the app wiring.

A concrete asset pool instance is single-lifecycle, but
resolveConfiguredAssetPoolStore handed the same object out again after
clearAssetPool() had closed it, reinstalling a dead store as the live
pool. The second handout now refuses loudly and points at the factory
form, which rebuilds on every resolution.

createPersistenceHandler eagerly awaited the asset byte store, so an
invalid ASSET_S3_BUCKET or an unresolvable AWS SDK rejected the shared
persistence handler and took document and runtime traffic down with an
optional backend. Byte-store construction is now deferred to the first
asset byte operation, and a failed construction is not cached, so the
next asset request retries -- the route's own no-poisoned-singleton
rule.

Refs #1007

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Co-authored-by: 杨慎 117187635+cosarah@users.noreply.github.com
Co-authored-by: Claude Opus 5 noreply@anthropic.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 73 Git 标签

feat(storage): wire the asset backend into the app (#1007) (#1089)