Commit Graph

3186 Commits

Author SHA1 Message Date
nicktrn c7fee76f14 Merge branch 'main' into v3/worker-attempt-creation 2024-05-03 12:03:01 +01:00
Eric Allam 9d6c424236 Add deploy admin debug tooltips 2024-05-03 11:22:51 +01:00
nicktrn bce398b892 increase indexing specs 2024-05-03 09:36:16 +01:00
nicktrn 90153c31f6 fix merge 2024-05-02 17:32:42 +01:00
nicktrn 4e18a421b3 Merge branch 'main' into v3/worker-attempt-creation 2024-05-02 17:19:08 +01:00
nicktrn dcc9745597 fix typecheck 2024-05-02 17:13:27 +01:00
nicktrn 1919b6f866 add exception event to failed run spans 2024-05-02 16:41:12 +01:00
nicktrn f53004de6b resurrect prod stack traces 2024-05-02 16:40:25 +01:00
Eric Allam ae99807047 Allow admin tooltips while impersonating 2024-05-02 16:39:01 +01:00
nicktrn c75bbfdf2c lazy attempt creation for prod workers 2024-05-02 15:58:02 +01:00
Eric Allam ff6e2461d5 Add admin tooltips to make support/debugging easier for admin users 2024-05-02 14:12:16 +01:00
Eric Allam 3ed016bc2b Fixed SharedQueueConsumer internal otel logging 2024-05-02 12:31:55 +01:00
Eric Allam 68dd3804a7 Improve the fair queue selection strategy by paginating through all the queues in the parent queue 2024-05-02 11:56:35 +01:00
Eric Allam 95f0c7b780 increase queue selection count to 36 2024-05-01 22:41:50 +01:00
Eric Allam 339c39d739 Fix type issues and unit tests 2024-05-01 22:23:38 +01:00
Eric Allam bc0ca01319 Fix fair queue priority strategy off-by one error 2024-05-01 22:13:52 +01:00
nicktrn 14992f43ae start adding lazy attempts to prod 2024-05-01 15:38:01 +01:00
Eric Allam 9e5382951b Improve the display of non-object return types in the run trace viewer 2024-05-01 13:52:05 +01:00
Eric Allam 1bba5d5617 Allow admins to view any project metrics 2024-05-01 10:41:50 +01:00
Eric Allam 5ca9e5667c Add an admin API endpoint to get info about the shared marqs queue 2024-05-01 10:41:50 +01:00
Eric Allam 3b3b07aec2 “Borrow” the code from alerts PR to get self hosted deployments working 2024-05-01 10:41:50 +01:00
Eric Allam f124d94efc Fixed task runs with unsanitized queue names 2024-05-01 10:41:50 +01:00
Eric Allam e24631e2dd Move the visibility queue stuff into a graphile job 2024-05-01 10:41:50 +01:00
Eric Allam d1bdd0cf5d Handling failing task runs that cannot create an attempt for whatever reason 2024-05-01 10:41:50 +01:00
Eric Allam 5ed700dad8 WIP worker TaskRunAttempt creation 2024-05-01 10:41:50 +01:00
nicktrn 62700245a3 v3: fix consecutive waits (#1073)
* fix spacing for delete hints

* don't try to resume deps on wait resume

* sending duration wait resumes is not an error anymore

* set correct status with new wait resume flow

* cancel checkpoint schema v2

* don't mix messages and schemas

* prevent unintended case fallthrough in tree view

* completely switch to platform-led duration wait resumes

* prevent infinite restores

* some entries for the catalog

* add pg to additional packages

* add checkpoint safe timeout

* prevent duplicate spans after restore

* wait for post start

* add sdk version to deploy tab

* fail on impossible checkpoint scenarios

* remove debug logs
2024-05-01 10:33:30 +01:00
Matt Aitken 0f0a6884e8 Environment variables pasting uses dotenv (#1075)
* Add dotenv package to the webapp frontend, required some polyfills

* Use dotenv to parse the pasted env vars. Make the panel wider on larger screens
2024-04-30 14:23:49 +01:00
Matt Aitken 7ff8f0ebab Task and run page improvements (#1076)
* TaskListPresenter: if there are no tasks then don’t do stats queries

* RunListPresenter, use BasePresenter and the read replica

* Added populate script

* Simplified the Runs list query, added live timer

* Added TaskRun indexes for the RunList

* Status can’t be null now we’re using the TaskRun status

* Use defer so the page loads and shows a spinner

* Improved the loading style

* Get rid of latest run info from the tasks table super slow

* Fix for the activity graph tooltip getting clipped

* Added a code comment crediting the GitHub issue with the portal fix

* Add search to the tasks list

* Padding

* Fix for the schedules columns not being UTC

* Remove unused function
2024-04-30 14:19:50 +01:00
Matt Aitken 68455c796a Fixes/run filtered keyboard nav (#1074)
* Ensure each switch condition returns a state

* Fix for scrolling when filtered

* Fix for up/down navigation when filtered
2024-04-29 18:52:46 +01:00
Eric Allam b0a2c42e0e Add some additional logging around nacking messages 2024-04-29 17:53:38 +01:00
Matt Aitken ed8d24fd3d Run page performance improvements (#1072)
* lotsOfLogs task now outputs much larger logs

* Removed tree view collapse/expand animation

* JSDocs for useDebounce

* WIP moving filtering into the state

* Reworked the reducer to do the filtering
2024-04-29 17:05:26 +01:00
nicktrn d0ef36260a stop trying to pull init image when already present 2024-04-29 15:33:46 +01:00
Matt Aitken 4fdb7f8288 Bulk insert environment variables (#1069)
* Fix for #1066. Correct environment username if dev

* WIP changing the form

* WIP on the form

* WIP on repository

* Adding environment variables en masse is working

* WIP on pasting

* useList hook with reducer

* Got bulk insert working with pasting… it was a pain

* Allow overwriting of values

* Set a max height on the new env var form
2024-04-28 18:31:31 +01:00
Matt Aitken 37b9b056c4 Upload payload packets outside of the db transaction 2024-04-28 18:18:33 +01:00
Matt Aitken 801c86bf73 Added search to the test tasks list 2024-04-28 16:10:46 +01:00
Matt Aitken a1de11a001 Default the test page to the “DEV” tab 2024-04-28 15:59:45 +01:00
Eric Allam 40ba8ad0ee Fix the test page when a scheduled task has recent run payloads that aren’t scheduled payloads 2024-04-26 18:20:04 +01:00
Matt Aitken 96168eb383 Fixes for test page problems 2024-04-26 17:45:06 +01:00
Matt Aitken de188a22c5 Option for users to opt-in to a staging environment (#1064) 2024-04-26 11:29:20 +01:00
Matt Aitken fae93ac747 New tasks page (with activity) and multi-user fixes (#1063)
* Fix for “overridden” spelling mistake

* Make the task function names grey instead of yellow

* The task page now shows each task once

* Blankline

* WIP on activity bars

* Correct size of the graphs

* Add data for all days, even if there are no runs

* Activity graph is working and has a tooltip

* No activity label

* Got the bar bg working

* Activity chart now has a bg

* Added a compound id to make the activity graphs appear faster

* Improvements to the activity graph

* Tried to disable the bg from animating

* Added read replica support to the app, using Nick’s code from his batching PR

* Added the read replica to BasePresenter

* Added the queued and running columns to the Tasks list

* Added avg duration column

* Included completed but failed runs in the average duration

* Get rid of the old useDevEnvironment hook and remove API keys from the projects query

* Unified getting environment info to using a function and doing it in presenters

* Removed the path from the tasks table

* Onboarding for dev for 2nd+ user. Improved environment sorting.

* Removed log

* Improved the spacing

* Enable staging for new v3 projects
2024-04-26 10:44:39 +01:00
Eric Allam 0a5aa2dc15 Fix slow job retries by optimizing how errored tasks are deleted 2024-04-24 22:07:12 +01:00
Eric Allam ff7fa9e19a Clear out errored tasks when re-attempting a job run (instead of using the cached error) 2024-04-24 13:00:02 +01:00
nicktrn 74d1e61e42 v3: fix for fresh CLI logins after CLI token revoke (#1056)
* prettier login command output

* fix for trying to login again after revoking cli token

* changeset

* improve errors when logging in with revoked or invalid token

* fix builds.. again
2024-04-24 10:44:34 +01:00
nicktrn 75a54540a4 use full image refs for init containers 2024-04-24 10:13:28 +01:00
Eric Allam b68012f81c v3: Various fixes for Next.js projects and projects that use v2 and v3 together (#1051)
* Fixes an issue that was treating v2 trigger directories as v3

* Make msw a normal dependency (for now) to fix Module Not Found error in Next.js.

* Extract out all the zod* stuff from core so the SDK does not import it

* Add a changeset

* Fixing typecheck errors in the webapp

* Export the Task and TaskOptions types

* Extract additional exports from core/v3 that aren’t used in the SDK

* Move to our global system from AsyncLocalStorage for the current task context storage

* Update the esbuild core bundling plugin for the new core v3 exports

* Fix v3 CLI telemetry

* Add support for tasks located in subdirectories inside trigger dirs

* Remove the env var check during deploy (too many false negatives)
2024-04-24 10:09:36 +01:00
Matt Aitken 3913e57ef4 Trace view fixes and improvements (#1046)
* Query param for span using history.replaceState is working

* When clicking again on a node, don’t collapse it

* Close the span view using the same replacing of the search param

* Conditional rendering of the resize panels was causing the tree view re-rendering and collapsing…

* Live reloading moved to where the parent label is

* Span action bar is now deeper

* WIP on trace view navigation changes with shortcuts

* Shortcuts for expanding and collapsing en masse

* Number keys expand/collapse levels

* Changed duration toggle to a shortcut key

* Option + click expands/collapse at that level

* Option/alt left/right expands/collapse at that level

* Removed unused imports

* Link from the runs table to the specific span

* Latest lockfile

* Sorted imports

* When doing a test link directly to a span

* Replay links to the span

* CLI log links go directly to a span

* Keyboard shortcuts are in a popover if the width is narrow

* If holding alt only collapse level

* Don’t expand the individual node if you’re holding alt
2024-04-22 13:46:08 +01:00
James Ritchie 26f310397a Fixed the grid layout for v2 runs page 2024-04-22 13:06:56 +01:00
Eric Allam 26093896d2 v3: Fixes for using (batch)triggerAndWait with idempotency keys (#1043)
* Fixes various issues with triggerAndWait and batchTriggerAndWait

When using idempotency keys, triggerAndWait and batchTriggerAndWait will still work even if the existing runs have already been completed (or even partially completed, in the case of batchTriggerAndWait)

- TaskRunExecutionResult.id is now the run friendlyId, not the attempt friendlyId
- A single TaskRun can now have many batchItems, in the case of batchTriggerAndWait while using idempotency keys
- A run’s idempotencyKey is now added to the ctx as well as the TaskEvent and displayed in the span view
- When resolving batchTriggerAndWait, the runtimes no longer reject promises, leading to an error in the parent task

* Remove the default queue concurrency limit as we now have env and org concurrency limits

* Use the run friendlyId in the completion result id

* Added some error logging
2024-04-19 10:54:43 +01:00
nicktrn 584c7da5df v3: prod worker graceful shutdown (#1034)
* graceful exit with timeout

* handle and display graceful timeout errors

* fix for very long waits

* changeset

* increase termination grace period to an hour
2024-04-18 15:46:14 +01:00
nicktrn c9e1a3e9c5 v3: fix dependency trigger and wait (#1030)
* fix dev dependecy resumes

* add catalog entry with multiple trigger waits

* update docker provider lifecycle command

* switch to x for clearing run filters

* Revert "fix dev dependecy resumes"

This reverts commit b4061f2ae71f21889adcf061928e2e77c436f0ed.

* fix dependency resumes, properly this time

* add catalog entry for dependency waits in loops

* advice in docs re parallel dependency waits

* fix link from v3 to v2 docs

* move lifecycle command logging to debug only

* Removed batchOptions from the trigger options and the docs

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-04-18 14:48:43 +01:00