Commit Graph

4002 Commits

Author SHA1 Message Date
Matt Aitken 4ca6506655 Delay reschedule fix (v3 & v4) (#2180)
* Fix for v3 rescheduling bug…

* v4 rescheduling working
2025-06-17 18:22:15 +01:00
Matt Aitken 9f9e74b0b6 Fix hydration error (#2179)
* Remove unused hook

* The Impersonation button was causing DOM hydration issues
2025-06-17 11:17:17 +01:00
Eric Allam ea6f66bdfc Improved schedule engine (#2174) 2025-06-17 06:48:05 +01:00
Eric Allam a42c9acf44 v4: Improved run locking (#2173) 2025-06-17 06:45:59 +01:00
Matt Aitken 5f6fd8e235 Tasks page didn't include Delayed runs (#2176)
* Delayed runs weren’t shown in the Tasks page mini graph

* The Tasks page Queued count didn’t include Delayed runs (since ClickHouse switchover)
2025-06-16 14:14:09 +01:00
Matt Aitken a0815c8e3e Clickhouse performance improvements (#2175)
* Runs filter by org id and add created at to ordering

* CopyableText can accept an alternative value for copying

* The runs table now shows the ID instead of run number

* Paginating back/forwards fix

* The task stats need org id and project id too
2025-06-14 11:10:28 +01:00
Matt Aitken c3dca5835d Fix for the Kapa widget not working with SSR (#2170)
* Move all the logic into a single component, preparing for client only

* Use ClientOnly

* Fix for search param not working
2025-06-12 22:17:22 +01:00
James Ritchie 20a8e7c6c1 Updated kapa sdk to 0.1.3 (#2169) 2025-06-12 16:21:34 +01:00
James Ritchie e12c82b444 Use the Kapa AI SDK instead of the Kapa AI widget (#2113)
* Install the kapa sdk

* WIP using the SDK for the Kapa Ask AI widget

* Removes old kapa from root

* Now rendering everything inside the dialog component

* Fixes min-height of dialog content

* Remove kapa from root

* prevents kapa using reCaptcha

* Adds more functionailty with temporary UI placement for now

* Reset conversation button

* Adds a new sparkle list icon

* Adds some example questions as a blank state

* Animate in the example questions

* use “marked” package to render markdown

* Improve some animations

* Submit a question from the URL param

* adds custom scroll bar styling

* fixes modal to correct height after re-opening it

* Add button to stop generating answer mid-stream

* Adds buttons states to show submitting, generating, submittable

* Adds a helpfull sentence in the blank state

* Show a message if the chat returns an error

* Adds reset chat and feedback buttons to the bottom of an answer

* Makes sure you can give feedback in the different states of chat

* Adds a suble background to the dialog

* Fix a button inside button error

* Improve the shortcut esc key on dialog and sheet component

* Fix classname error

* organize imports

* Use our custom focus-visible

* Move the Tooltip for the button into the AskAI component

* Improved error message

* Organize imports

* Animated the modal gradient

* Small layout improvements

* Adds most asked questions from Kapa

* border glow tweak

* AskAI component is now a hook that can take a question

* remove kapa script

* Add a delay before the modal opens when usign the URL params

* Remove old component

* Update to the latest Kapa version

* Rephrased error message

* Use correct types for conversation

* Fixed types for addFeedback

* Adds DOMPurify package

* removed unused const

* Removed unnecessary platform specification

* Reset the timeout when the ai panel pops up

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Fix for coderabbit bad commit

* Clean up imports

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-12 14:09:03 +01:00
Eric Allam 48e1e7250b v4: fix stuck batch issue when processing happens before transaction commits (#2163)
* v4: fix stuck batch issue when processing happens before transaction commits

* Fix flaky test
2025-06-10 17:02:34 +01:00
Eric Allam 3f2b748a80 Delete the secure param before passing it to the ClickHouse client (#2161) 2025-06-10 13:55:19 +01:00
Eric Allam e6fb32194e Add environment filter to declarative schedules (#2148) 2025-06-10 12:11:11 +01:00
Eric Allam b38405cb88 Realtime and task run performance improvements (#2158)
* Add createdAt filter to realtime subscribing with tags

* Filter realtime colums and expose ability to skip some columns

* Add sharding support for electric

* Use unkey cache for the created at filter caching

* Remove 2 unused indexes on TaskRun

* Run list now filters by a single runtime environment

* Remove project ID indexes

* Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters)

* WIP clickhouse powered runs list
stuff

* Improve the query to get the latest tasks for the task list presenter

* Update the usage task list to use clickhouse

* Implement next runs list powered by clickhouse

* Add new index for TaskRun for the runs list, by environment ID

* Add runTags gin index

* Handle possibly malicious inputs

* Ignore claude settings

* Better handling not finding an environment on the schedule page

* Use ms since epoch in test, not seconds

* Remove unused function

* Fix test

* Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day)

* Fixed the query builder to correct the group by / order by order

* Make sure runs.list still works

* Create small-birds-arrive.md
2025-06-10 12:11:01 +01:00
nicktrn 47b19a49d4 A few small v4 fixes (#2153)
* remove unused env vars

* actually use ALERT_REPLY_TO_EMAIL for alerts (non-breaking)

* increase fallback branch limit to 100M

* more unused env vars and example

* hide usage page when self-hosted

* fix for init flow

* set default or concurrency to 300 to match 100 on env

* fail fast when registry env vars are empty strings
2025-06-06 10:15:07 +01:00
Eric Allam 5301239e09 chore: remove v2 models from schema (#2107)
* Round 1 of v2 model removals

* Round 2

* Remove trigger http endpoint env

* Round 3

* Round 4

* Round 5

* Round 6

* Round 7

* Round 8

* Round 9

* Round 10

* Removed the remainder v2 code from the webapp

* GitButler WIP Commit

---------

Co-authored-by: GitButler <gitbutler@gitbutler.com>
2025-06-05 22:57:29 +01:00
Eric Allam 261dce102a v4: fix batchTriggerAndWait completion issues by processing batch chunks sequentially (#2152)
* v4: fix batchTriggerAndWait completion issues by processing batch chunks sequentially

* Remove unnecessary test
2025-06-05 17:56:56 +01:00
nicktrn e7795a06ad Fix: fixes and prerequisites for v4 self-hosting (#2150)
* remove pgadmin

* remove V3_ENABLED

* v3 is always enabled

* enfore docker machine presets by default

* rename autoremove env var

* prefix more k8s-specific env vars

* same prefix for all docker settings

* improve profile switcher copy

* supervisor can load token from file

* optional webapp worker group bootstrap

* fix error message

* fix app origin fallback for otlp endpoint

* use pnpm cache for webapp docker builds

* increase default org and env concurrency limit to 100

* optional machine preset overrides

* improve s3 pre-signing errors

* fix DOCKER_ENFORCE_MACHINE_PRESETS bool coercion

* shard unit tests

* fix for s3-compatible services

* optional object store region

* Update apps/supervisor/src/workerToken.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix DEPLOY_REGISTRY_HOST example

* fix platform mock

* remove remaining v3Enabled refs

* fix error type.. bad bot

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-04 18:02:38 +01:00
Eric Allam 2b3ea692fe v4: dequeue performance improvements (split concurrency from dequeue) (#2127)
* WIP

* Run queue now works with the worker queue / master queue split

* Acking should also cause the master queue to be processed

* Convert run engine tests and run engine to use runQueue changes

* Include the util files in the test tsconfig

* coordinator target should be es2020 as well

* providers target 2020

* Fix the triggerTask tests in the webapp

* v4 now working with the new worker queues, and added the legacy master queue migration stuff

* report worker queue lengths via opentelemetry metrics

* Adding lock metrics

* Release concurrency bucket metrics

* • Updated RunQueue.removeEnvironmentQueuesFromMasterQueue() method signature to take runtimeEnvironmentId instead of masterQueue parameter
• Added automatic master queue shard calculation using this.keys.masterQueueKeyForEnvironment(runtimeEnvironmentId, this.shardCount) 
• Updated RunEngine wrapper method to use new runtimeEnvironmentId parameter
• Updated DeleteProjectService to call the method once per environment instead of once per master queue
• Simplified API by encapsulating master queue sharding logic within RunQueue class

* metrics now working, configure the run queue settings, additional metrics for run engine and redis-worker

* Fix CodeRabbit suggestions

* return undefined from dequeueFromWorkerQueue, not null

* Remove message from worker queue in certain circumstances when acking

* Update log

* Ensure master queue consumers cannot stop from a processing error, and make the consumer interval configurable via an env var

* Change how the run queue master queue consumers are disabled internally

* Fixed tests

* process the queue on nack

* Fix more tests

* Fix priority tests

* Fixed dequeueing test
2025-06-04 17:18:46 +01:00
nicktrn f603725393 Feat: unified deploys for self-hosted and cloud users incl. multi-platform support (#2138)
* remove registry proxy

* remove --self-hosted flag

* automatically set network build flag

* update syncEnvVars debug log

* improve switch command

* always display deploy errors if they exist

* fix stuck deploy command after finalize error

* webapp-driven deploys, multi-platform support, lots of fixes

* add worker deployment migration

* rename image platform env var

* only try to sync parent env vars for preview deployments

* add KEEP_TMP_DIRS

* supervisor: docker api version lock, auth, multi-platform

* set image ref on create, validate digest

* use metadata for digest, fix local multi-platform builds

* print git meta branch before commit

* improve push and load flag handling

* make runs after local builds compatible with load and push

* small improvement for platform overrides

* add image platform to dequeued message

* remove deprecated init request body fields

* fix fail deployment id param

* remove build debug logs

* pass report merge with no tests

* structured run debug logs

* add required env var for tests

* should not be an error log

* add changeset
2025-06-04 15:54:10 +01:00
nicktrn a61951049f Enhance webapp version with build info (#2146)
* improve app version output

* set build info

* fix typo

* always show additional build info when self-hosting

* Update apps/webapp/app/routes/_app.orgs.$organizationSlug.settings/route.tsx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Update apps/webapp/app/components/navigation/OrganizationSettingsSideMenu.tsx

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix build timestamp name

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-06-04 15:49:07 +01:00
nicktrn 1b62b348a9 Fix tar-fs dependabot alerts (#2143)
* update dockerode

* override tar-fs for remix dev
2025-06-03 14:10:35 +01:00
nicktrn 84d8fb575f Fix syncing parent env vars for non-preview deployments (#2131)
* only try to sync parent env vars for preview deployments

* add changeset
2025-06-02 10:40:06 +01:00
Matt Aitken 80d449bfdb Preview branch alerts (#2136)
* Preview branch alerts

* Show “Preview” not the branch for new alert modal
2025-06-02 10:37:37 +01:00
Matt Aitken 5fdc7e66af Preview branch fixes (#2126)
* SDK: auto-detect the branch using the VERCEL_GIT_COMMIT_REF env var

* Waitpoint token completion with HTTP is now working
2025-05-29 21:23:11 +01:00
Matt Aitken ea853b243e Remove onsole.log… oops (#2125) 2025-05-29 14:39:36 +01:00
Matt Aitken 12f4653cd6 Fix long preview branch names (#2124)
* Truncate long branch names, don’t let the icon shrink

* Truncate and show tooltip for long branch names
2025-05-29 14:36:24 +01:00
Matt Aitken 7ecaf2ce53 Sync env vars (parent env vars) (#2120)
* Prefer branch variables, but load preview envs too

* WIP with vercel preview env var syncing

* Return success if either child or parent env vars are synced

* Removed log of env vars

* Remove another log
2025-05-29 12:18:25 +01:00
nicktrn 7b463710f5 Fix metrics format (#2119)
* ensure eof marker is always at the end

* strip empty lines from prisma metrics
2025-05-29 09:52:42 +01:00
nicktrn fda9565eb0 Optionally disable run debug logs (#2116)
* disable run debug logs by default

* lightweight webapp health check

* disable debug logs for dev runs

* disable run debug logs for supervisor client

* add changeset
2025-05-29 09:27:08 +01:00
Eric Allam 021d6d8f20 Improve efficiency of cancelling in progress TaskEvent records in v3 and v4 (#2112) 2025-05-28 11:06:19 +01:00
Matt Aitken 068c024477 Preview branches (#2086)
* Initial preview migrations

* Modified the staging endpoint to create preview environments

* Added isBranchableEnvironment to RuntimeEnvironment

* Staging = yellow Preview = orange

* Changed the env sort order

* Set isBranchableEnvironment correctly. Create preview for new projects

* Very basic branch menu

* Creating branches from the dashboard

* Fix for string icons on project delete page

* Don’t show branch API keys

* WIP on the manage branches page

* RuntimeEnvironment added projectId index

* Only create the parentEnvironmentId column if it doesn’t exist already

* Improved the limit wording

* Add search to the branch list

* contains in both places

* Many style improvements

* Branch dropdown and v4 badge

* Arching/unarchive branches working in the dashboard

* Tidied imports

* Change preview slug from `prev` to `preview`

* Use correct color for side menu preview branch icon

* Upsert the branch and use the shortcode as a unique constraint

* Upserting working with nice messages in the dashboard

* Better errors when upserting branches

* Button shortcut, don’t allow event to propagate

* Better duplicate error message

* Filter out archived branches from the env selector

* Archiving/creating tweaked some more

* Add an archived banner to the app, fixes for archived branches and upsells

* Fixed pagination

* Disable editing schedules, pausing queues, testing tasks

* Don’t allow replaying if the env is archived

* When deploying detect the correct environment

* Get the projectClient when there’s a branch

* createGitMeta function, most code from the vercel CLI repo

* Deploy, getting the correct environment client

* Added git column to WorkerDeployment

* Add GitMeta to core schemas

* Create branch when deploying

* WIP on branch support in the API

* Delete old createTaskRunAttempt fn

* apiAuth remove export from internal functions

* Rename env var to “TRIGGER_PREVIEW_BRANCH”

* Add TRIGGER_PREVIEW_BRANCH to resolved env vars for runs

* First preview deploy and run working

* Set the preview branch in the main SDK

* Added git links to the preview branches table

* Better errors when replaying/testing archived branches

* Don’t dequeue archived environments

* Env var resolution with parent environment

* Hello world default machine small-2x to save my memory

* Fix for more env var functions

* Only return non-archived envs

* Switch to controlled state for the checkboxes

* Uncheck everything when PREVIEW is checked

* WIP on branch UI

* Show the preview branch label on the env vars list

* Fix for overriding env vars

* Adding preview branch env vars working

* Progress on new env vars

* Only allow selecting a single branch

* Layout fix when there are errors

* Set the defaultValue so there are some fields

* Conform fix for team invite page

* Archived environments don’t run scheduled tasks

* Added Git data to deployments

* Added git data to the deployment inspector

* Don’t allow upserting schedules when archived

* Deduplicate and blacklist some env vars

* Fix for wrong conform function being used

* Show a better error if all vars were blacklisted

* Added environment variable search (by key and value)

* Improved preview branch icon

* Replay now supports branches

* Schedule page render branches properly

* Show the env icon in bottom-left of the test page

* When editing older schedules (that have multi-env) show preview branches correctly

* Fix for incorrect disallowed branch name character

* Extract and improve the directory verification code

* WIP for CLI preview archive command

* Improved the preview branch action buttons

* Redirect to the project if we don’t find a matching env

* Archiving branch via the CLI working

* Fix for archiving branches

* Public access token test task

* JWTs working are with preview branches

* Add branch and git data to the Run ctx

* Updated GitMeta functions to work in CI

* Added pullRequestState

* Archive when deploying if the PR is closed/merged

* Fix for the changesets guide

* Fix for CLI dev bug introduced

* CLI promote now supports preview branches

* Add PR title. Reordered them and added tooltips

* syncEnvVars working with branches

* Added preview branch support to syncVercelEnvVars()

* Detect the branch from Vercel env var (set during build)

* Allow passing a branch in

* Use process.env.VERCEL_TOKEN as well… this used in Vercel CI

* Temp delete

* Improved regenerate api key modal

* Added Accordion component (with styles)

* Redesigned the API keys page

* Revert "Temp delete"

This reverts commit 177b92cd935a6161456bde65d01294e23ecfd47f.

* Changeset

* Fixed docs link

* The new branch panel closes when a branch is created

* Update apps/webapp/app/services/upsertBranch.server.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Removed findUniques from WorkerGroupTokenService

* Made the parentEnvironmentId migrations safe

* Latest lockfile

* Update packages/cli-v3/src/commands/workers/build.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Move isValidGitBranchName to a separate file

* Move the sanitize fn too

* removeBlacklistedVariables moved to a separate file

* Moved deduplicateVariableArray to a separate file…

* Fix broken sanitizeBranchName import

* Another import fix…

* Improved blacklisted error message

* SImplified migration to use `ADD COLUMN IF NOT EXISTS "parentEnvironmentId" TEXT`

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-05-28 11:02:56 +01:00
Eric Allam b74c000fb6 Improve metadata flushing efficiency by collapsing operations (fix #2104) (#2106)
* Improve metadata flushing efficiency by collapsing operations (fix #2104)

* Add changeset

* A couple of style tweaks
2025-05-27 14:08:15 +01:00
Eric Allam 0410316e14 Enable prisma metrics and add them to the /metrics endpoint (#2111)
* Enable prisma metrics and add them to the /metrics endpoint

* Add support for bearer token auth on metrics endpoint
2025-05-27 13:28:45 +01:00
Eric Allam 3fc5630be7 fix: calculating next schedule should always just be from now (#2102) 2025-05-23 16:55:52 +01:00
Eric Allam 13d378f6df v4: batches can only complete when all their runs have been created (#2101)
* v4: batches can only complete when all their runs have been created

* Fix flaky runs replication test
2025-05-23 16:53:50 +01:00
Eric Allam 1a6a9ca97a perf fix: calculateNextScheduledTimestamp speed improvement (#2099)
* Simulate "next steps" when calculating the next timestamp to prevent millions of iterations on dev schedues

* Add additional tests

* Fix typescript issue
2025-05-23 16:24:33 +01:00
nicktrn 7c791dd519 Improve unit test workflow performance (#2096)
* shard unit tests

* temp enable for all pushes

* fix test workflow

* update to latest vitest and only add to root package.json

* additionally use default reporter

* gather reports before uploading

* split up slow replication tests

* split up unit tests workflow

* move workflows to parent dir

* use new paths in parent workflow

* prevent artifact clashes

* we always need to create the reports dir

* speed up merge reports

* gather reports even when tests fail

* fix artifact patterns

* increase shards

* disable push trigger again

* improve dequeue snapshot test reliability
2025-05-23 12:22:14 +01:00
Eric Allam a306545952 Always find debug logs when in the span sidebar (#2093) 2025-05-22 14:22:12 +01:00
Eric Allam 1bb1fc4c8f v4 run engine debug logs now filtered out in SQL by default (#2092) 2025-05-22 10:44:06 +01:00
nicktrn 8112c12092 Improve email whitelisting and extend to GitHub auth (#2090)
* make github auth respect email whitelist

* display github login errors

* fix smtp secure env vars

* prevent sending magic link if email not allowed

* fix MARQS_DISABLE_REBALANCING and RUN_ENGINE_DEBUG_WORKER_NOTIFICATIONS
2025-05-22 10:30:00 +01:00
Eric Allam 14e081e17d release concurrency system reliability improvements (#2081)
* Refill release concurrency tokens when a run is cancelled

* Improved release concurrency accounting system + a sweeper to auto-refill tokens for snapshots that are no longer the latest snapshot on a run (e.g. the run has moved to a new snapshot state)

* Fix order of arguments to the releasings sweeper

* Add a heartbeat for SUSPENDED snapshots, where when stalled will attempt to continue the run if unblocked
2025-05-21 13:21:17 +01:00
Eric Allam 500a173d67 Set default values on the invite and new env var forms to fix an issue after updating conform (#2084) 2025-05-21 13:20:16 +01:00
Eric Allam 0334df70ad Prevent socket hangup error by removing the aggressive clickhouse keep alive ttl env var default (#2082) 2025-05-21 13:15:15 +01:00
Eric Allam 8471169b30 Fix various dependabot alerts (#2063)
* Upgrade posthog-node to clear axios vulns

* Upgrade @slack/web-api to use a secure version of axios

* Update parse-duration to fix security vulns

* Mitigate against the ws DoS vuln

* Upgrade body-parser to 1.20.3 in the webapp

* Upgrade react-use to 17.5.1 to remove the fast-loops transitive dep

* Remove unused babel dev deps and config file that's no longer used

* Upgrade prismjs to 1.30.0 and bundle parse-duration now

* upgrade express to 4.20.0 to fix issue with XSS when redirecting

* Upgrade @conform/zod to 0.9.2
2025-05-19 12:01:51 +01:00
nicktrn 558a39c644 Remove supervisor docker binary requirement and other tweaks (#2062)
* don't require docker binary

* more structured logs

* remove docker type dep

* add core changeset

* use implicit DOCKER_HOST instead

* disable resource monitor for now

* support attaching docker runners to multiple networks

* warn if dequeue interval > idle dequeue interval

* verbose logs

* add changeset
2025-05-17 12:33:41 +01:00
Eric Allam a69621bdcc Allow self-hosted deploys locally by pointing to docker.host.internal on macOS (#2064) 2025-05-16 15:20:48 +01:00
Eric Allam ad4daa3301 Fix run replication payloads/outputs trying to send invalid types like BigInt to clickhouse (#2065) 2025-05-16 11:02:30 +01:00
Eric Allam bb7173b4ab Improve run replication logging (#2060)
* Remove the tcp buffer monitors

* Improve run replication logging
2025-05-15 14:12:01 +01:00
Eric Allam d1e40643fc Allow creating and monitoring run replication services with different settings (#2055)
* Allow creating and monitoring run replication services with different settings

* Fix test
2025-05-14 20:40:47 +01:00
Eric Allam eb3929880f runs replication leader lock expiration fix (#2050)
* runs replication leader lock expiration fix

* Allow configuring the container image --max-old-space-size using NODE_MAX_OLD_SPACE_SIZE

* Ability to configure the clickhouse keep alive settings

* Add some logging because we might not be able to do telemetry
2025-05-14 15:49:48 +01:00