* Add external log exporters and fix missing external trace exporters in deployed tasks
* Generate the external traceID correctly and exporter 3rd party logs with the external traceID as well
* WIP clickhouse package with test containers setup
* More clickhouse client setup now with otel and real tests, and the v1 of raw run events
* Add some additional columns to raw_run_events_v1
* WIP runs dashboard service
* Create a new run engine event bus event for the runs dashboard to hook into
* Track run events in the run engine
* make sure engine v1 runs get synced to CH
* Update the attemptNumber of v3 task runs
* Restructure the run events to be more sparse
* emit more stuff
* Setup replication package
* scaffold the replication package
* replication wip
* resolve conflicts
* more replication stuff
* Add ability to drop the replication slot completely on teardown
* Use the new single replacingmergetree task events table for replication
* get it working
* insert payloads into their own table only on insert and then join
* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh
* Handover WIP and tests
* Testing the replication service
* Remove the runs dashboard stuff that we aren't using anymore
* Added a test for large payloads
* hacky typecheck fix
* Fix new internal package typecheck issues and start adding telemetry to the replication service
* tracing over spans, some other improvements
* Improvements to the runs replication service, now ready for testing
* Some fixes and cleanups
* Don't need this code anymore
* move transaction types into the runs replication service
* only send spans where there are transaction events
* A couple of suggested tweaks
* Initial commit with a plan for what we’re going to do
* Some initial types and improved plan
* Add Waitpoint resolver
* Add resolver + status index
* Remove type + status index
* Only drop if exists
* Remove type index
* Update waitpoint list presenter to use resolver
* Added resolver to the engine
* Made the existing waitpoint list presenter more flexible
* Initial implentation ofr wait.forHttpCallback()
* Added the callback endpoint (no API rate limit)
* schema version
* Added jsdocs, removed schema version because of errors
* Show callback URL if it’s set
* Dashboard pages and panels
* Remove todos
* Added temporary icon
* Added a blank state
* Some tweaks and added a Replicate example
* Implement unwrap() for httpCallback
* Added unwrap to wait.forToken() as well
* Improved jsdocs
* Added docs
* Added unwrap to the token docs
* Show a dash if there are no tags
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Make the timeout error safer
* Fixed migrations… should use id desc not createdAt desc
* Fixed page title
* Fixed migration so it only adds them if they don’t exist. This allows us to manuall run in cloud first
* Respect the max content length by getting the length of the body
* Added more docs details about the callback format
* Remove code comment
* Improved the error
* Added a hash to the HTTP callback URLs
* Add the apiKey to the API input type to fix TS error
* Return the error responses. They were being caught and not preserved
* The content-length header is required. Deal with an empty body
* Removed unused types
* Added some new span icons
* Reworked http callback to be a create call then just use wait.forToken()
* Added a changeset
* Updated the docs
* Updated the wait overview docs
* Simplify to just a call
* WIP stripping right back to waitpoints just having a URL associated with them…
* More deletions
* Remove missing icon
* Updated the changeset
* Add URL to the token return types
* Remove wait for http callback page
* Updated docs
* More tidying
* Type and import fix
* Remove unused import
* Some type fixes for the retrieve
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* remove dead code
* rename managed to shared runtime manager
* rename to resolve waitpoint for clarity
* add resolver id helper
* store and correctly resolve waipoints that come in early
* fix ipc message type change
* branded type for resolver ids
* add fixme comments
* remove more unused ipc schemas
* fix entitlement validation when client doesn't exist
* restore hello world reference workspace imports
* runtime manager debug logs
* prefix engine run logs
* managed run logger accepts nested props
* runtime suspendable state and improved logs
* require suspendable state for checkpoints, fix snapshot processing queue
* add terminal link as cli module so we can more easily patch it
* apply cursor patch
* add license info
* remove terminal-link package and add deprecation notice
* remove old patch
* remove terminal-link from sdk
* rename snapshot module
* add cli test tsconfig
* add run logger base type
* add snapshot manager tests
* fix cli builds
* improve QUEUED_EXECUTING test
* changeset
* make testcontainers wait until container has stopped
* require unit tests for publishing again
* avoid mutation during iteration when resolving pending waitpoints
* improve debug logs and make them less noisy
* always update poller snapshot id for accurate logs
* detach task run process handlers
* check for env overrides in a few more places and add verbose logs
* log when poller is still executing when we stop it
* add supervisor to publish workflow
* always print full deploy logs in CI
* Revert "avoid mutation during iteration when resolving pending waitpoints"
This reverts commit 87b0ce1e5b.
* disable pre
* print prerelease script errors
* Revert "disable pre"
This reverts commit 9403409637.
* misc fixes
* better debug logs
* add snapshots since methods and route
* prep for snapshots since
* improve deprecated execution detection
* update supervisor and schema
* properly log http server errors
* detect restore after failed snapshot fetch
* run and snapshot id can be overridden
* fix restore detection
* fix deprecation checks, move into snapshot manager
* less logs
* rename snapshot manager stop
* restore detection was moved into snapshot manager
* fix notifier logs
* make runtime manager status a debug log
* no need to attach runtime status twice
* findUnique -> findFirst
* sort snapshots by created at everywhere
* realtime: fix streams missing chunks when streams are longer than 5 minutes and receive 408 Request Timeout errors. Also now support multiple client streams being sent to a single stream key
* Safely release the reader lock
* Locked task runs will now require queues and tasks to be in the locked version
* Client errors caught in a run function now will skip retrying
* Extracted out the trigger queues logic
* extract validation, idempotency keys, payloads to concerns
* Extracted out a bunch of more stuff and getting trigger tests to work
* Add queue and locked version tests
* Deadlock detection WIP
* more deadlock detection
* Only detect deadlocks when the parent run is waiting on the child run
* Improve the error experience around deadlocks
* A couple tweaks to make CodeRabbit happy and fixing the tests in CI
* Fixed failing test
* Changeset
* wip
* Make sure to scope queries to the runtime env
* multiple queue consumer in the same supervisor instance
* disable pre
* Revert "disable pre"
This reverts commit 4b154398982758d09667531e3a64761da83664eb.
* WIP on secret env vars
* Editing individual env var values is working
* Sort the env vars by the key
* Deleting values
* Allowing setting secret env vars
* Added medium switch style
* Many style changes to the env var form
* “Copy text” -> “Copy”
* Draw a divider between hidden buttons
* Env var tweaks
* Don’t show Dev:you anymore
* Grouping the same env var keys together
* Styles improved
* Improved styling of edit panel
* Fix bun detection, dev flushing, and init command (#1914)
* update nypm to support text-based bun lockfiles
* add nypm changeset
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add changesets
* chore: remove pre.json after exiting pre mode
* init command to install v4-beta packages
* Revert "chore: remove pre.json after exiting pre mode"
This reverts commit f5694fde9314114c74a220c2213d19667bca1a6c.
* make init default to cli version for all packages
* Release 4.0.0-v4-beta.1 (#1916)
* chore: Update version for release (v4-beta)
* Release 4.0.0-v4-beta.1
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Both run engines will only lock to versions they can handle (#1922)
* run engine v1 will only lock to v1 deployments
* run engine v2 will only lock to managed v2 deployments
* test: create background worker and deployment with correct engine version
* Add links to and from deployments (#1921)
* link from deployments tasks to filtered runs view
* jump to deployment
* don't add version links for dev (yet)
* Fix current worker deployment getter (#1924)
* only return last v1 deployment in the shared queue consumer
* be explicit about only returning managed deployments
* Add a docs page for the human-in-the-loop example project (#1919)
* Add a docs page for the human-in-the-loop example project
* Order guides, example projects and example tasks alphabetically in the docs list
* Managed run controller revamp (#1927)
* update nypm to support text-based bun lockfiles
* fix retry spans
* only download debug logs if admin
* add nypm changeset
* pull out env override logic
* use runner env gather helper
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add logger
* add execution heartbeat service
* add snapshot poller service
* fix poller
* add changesets
* create socket in constructor
* enable strictPropertyInitialization
* deprecate dequeue from version
* start is not async
* dependency injection in prep for tests
* add warm start count to all controller logs
* add restore count
* pull out run execution logic
* temp disable pre
* add a controller log when starting an execution
* refactor execution and squash some bugs
* cleanup completed docker containers by default
* execution fixes and logging improvements
* don't throw afet abort cleanup
* poller should use private interval
* rename heartbeat service file
* rename HeartbeatService to IntervalService
* restore old heartbeat service but deprecate it
* use the new interval service everywhere
* Revert "temp disable pre"
This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.
* add changeset
* replace all run engine find uniques with find first
* Release 4.0.0-v4-beta.2 (#1928)
* chore: Update version for release (v4-beta)
* Release 4.0.0-v4-beta.2
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Remove batch ID carryover for non-batch waits (#1930)
* add failing test case
* do not carry over previous batch id when blocking with waitpoint
* delete irrelevant test
* Delete project (#1913)
* Delete project
- Don’t schedule tasks if the project is deleted
- Delete queues from the master queues
- Add the old delete project UI back in
* Mark the project as deleted last
* Fix for overriding local variable
* Added a todo for deleting env queues
* Remove todo
* Improve usage flushing (#1931)
* add flush to global usage api
* enable controller debug logs
* initialize usage manager after env overrides
* add previous run id to more debug logs
* add changeset
* For secret env vars, don’t return the value
* Added a new env var repository function for getting secrets with redactions
* Test task for env vars
* Delete heartbeat file, merge mess up
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Saadi Myftija <saadi.myftija@gmail.com>
* add flush to global usage api
* enable controller debug logs
* initialize usage manager after env overrides
* add previous run id to more debug logs
* add changeset
* update nypm to support text-based bun lockfiles
* fix retry spans
* only download debug logs if admin
* add nypm changeset
* pull out env override logic
* use runner env gather helper
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add logger
* add execution heartbeat service
* add snapshot poller service
* fix poller
* add changesets
* create socket in constructor
* enable strictPropertyInitialization
* deprecate dequeue from version
* start is not async
* dependency injection in prep for tests
* add warm start count to all controller logs
* add restore count
* pull out run execution logic
* temp disable pre
* add a controller log when starting an execution
* refactor execution and squash some bugs
* cleanup completed docker containers by default
* execution fixes and logging improvements
* don't throw afet abort cleanup
* poller should use private interval
* rename heartbeat service file
* rename HeartbeatService to IntervalService
* restore old heartbeat service but deprecate it
* use the new interval service everywhere
* Revert "temp disable pre"
This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.
* add changeset
* replace all run engine find uniques with find first
* attach all run controller logs to the run
* make controller-level pre-suspend wait duration configurable
* snapshot status should remain EXECUTING for short retry delays
* add suspended process error
* ensure clean slate before waiting for next run
* treat immediate retries as warm starts
* fix for finished runs waiting forever
* don't hardcode max old space size in deployment images
* flags: treat underscores as hyphens
* append attempt number to runner name if >1
* improve retry spans for oom errors
* Make sure releaseConcurrencyOnWaitpoint is set on taskQueue
* Allow task queue to have releaseConcurrencyOnWaitpoint (required to fix custom queues)
* Test for releaseConcurrencyOnWaitpoint
* Added a warning if you use the same queue twice with different settings
* Improved the error some more
* Better trace icon
* Better Waitpoint token icon
* Fix for bad jsx
* Warm and cold start icons
* Tooltips now use a <Portal> so they appear on top
* Warm start components
* Added warm start markers to the Run page and inspector
* Fix for getting the correct value from the metadata
* Better trace icon, with fallback to a passed in one
* Removed unused isWarmStart function
* make worker nodetype configurable
* make max dequeue count configurable from supervisor
* deterministic runner ids
* update runner id on restore
* disable resource monitor
* explicit run controller type imports
* enhance debug logs
* v4: eagerly fork child process before warm start
Also fixes an issue where the attempt span events weren't coming through in the partial spans
* Making flushing a safer operation
* Add changeset
* Improve attempt metrics when in a warm start, and pass the STYLE_VARIANT for a warm or cold start on attempt spans
* remove dummy metrics from heartbeat
* fix heartbeat timeouts..
* reset clock when resolving waitpoints
* optionally set metadata url when scheduling run
* don't use global vitest as it's outdated
* don't run test files in parallel
* handle special graceful shutdown code
* exit codes for success and failure can now be set and overridden
* ensure immediate cleanup in tests
* update lockfile after conflict
* AI SDK sql generator and approval example WIP
* Create some nice spans and extract the tools out so we can get the type of the streams
* Enable AI SDK telemetry
* Adding ai.tool and removing toolTask, 3rd party telemetry spans now wil create partials, better ai SDK telemetry icons
* Created a separate d3-chat example, split out from the openai-agents example
* Fixed the tool options being passed to metadata in `ai.tool`
* Add a link to the run
* Slightly improved design
* Add a crawler task using crawl4ai
* Use a tool to get the userId
* Couple of tweaks
* Adding markdown rendering to assistant messages and added an e2b based chart rendering task
* Backup to anthropic
* Add changeset
* logs for optional services
* print env vars on startup in debug mode
* routes need to explicitly ask to keep connection alive
* log indicators for now
* make workload api listen host configurable
* expose supervisor metrics and make more configurable
* configurable pull secrets, no defaults
* remove restore route
* run controller to handle queued executing
* fix v3 deploys in v4 project
* update admin worker route
* only start pod cleaner et al in k8s mode
* set new worker group as default if none yet
* make image ref optional
* checkpoint image ref is optional for output as well
* export feature flag const
* one last image ref type fix
* make runner intervals configurable
* ability to set arbitrary env vars on new runners
* set default runtime back to node 21
* move all runner env vars to the same section
* logs for optional services
* print env vars on startup in debug mode
* routes need to explicitly ask to keep connection alive
* log indicators for now
* make workload api listen host configurable
* expose supervisor metrics and make more configurable
* configurable pull secrets, no defaults
* remove restore route
* run controller to handle queued executing
* fix v3 deploys in v4 project
* update admin worker route
* only start pod cleaner et al in k8s mode
* set new worker group as default if none yet
Eric Allam