* CLI create-integration command now accepts an Open AI api key
* Create integration docs separated into multiple pages
* Initial Airtable integration commit, with OpenAI generated code
* OAuth page coming soon
* Export DisplayProperty from the SDK
* TSConfig made to match GitHub’s with paths
* getRecords
* Removed duplicate Stripe job from the catalog
* Renamed Airtable apiKey option to token
* First Airtable job
* Export Collaborator and Attachment field types
* A typesafe example that uses runTask
* WIP on new integration tasks… not working yet
* Attempt with class
* Revert "Attempt with class"
This reverts commit 93a48330019f754c3216c5b49964fa4b0218bd3f.
* WIP changing how tasks work
* Mock of async local storage
* Moved client creation from constructor
* New approach with a clone method on TriggerIntegration
* Added runTask to Airtable which is used by integration tasks
* Added the Airtable icon and connection when using runTask
* base().table() is working
* runTask options moved to the 3rd param, made optional with optional name
* Added some generic arguments
* Added generic type to table
* Removed old comment
* We don’t need to repeat the icon
* getRecords and getRecord now returning the right data and types
* Creating records
* Update records
* Delete records
* The internal properties of integrations are now hidden by the TypeScript types
* Sprinkled a Prettify in there
* Improved the types
* Added Airtable to the integration catalog
* Early work on Airtable webhook registration
* More progress with webhooks
* connectionKey needs to be cloned for webhooks to work
* connectionKey needs to be cloned for webhooks to work
* It was unclear that the ActivateSourceService was using a graphileJob id
* ActivateSourceService optionally takes a jobId, if missing it generate a unique id
* When retrying trigger registration, don’t pass an id so it is generated
* Removed Airtable webhooks tasks from the job-catalog example
* Added TriggerSourceOption, removed TriggerSourceEvent
* WIP with new ExternalSource options
* ExternalSourceTrigger setup
* DynamicTrigger changed to options, will need some more work
* filter gets options passed to it
* SourceMetadata v2 renamed to SourceMetadataV2, kept original
* Started versioning the backend
* Moved param order on io.getEvent and io.cancelEvent
* The runTask stuff that allows unknown to work is back
* Indexing for v1 and v2, with version on “activateSource” schema
* Added todos, to deal with Airtable SDK calls inside the webhook handler
* “deliverHttpSourceRequest” queueName changed to the source id so they process in order
* ActivateSource changes to deal with old and new data formats
* Update existing TriggerSources to v2
* Fix for dynamic.ts typescript errors, need to revisit this later
* UpdateSourceService v1 and v2, with new v2 API endpoint
* Removed unused imports
* More progress on v1 and v2
* Airtable webhooks are now triggering a job
* Moved webhooks to a new file
* You can do API calls in the webhook handler now, Airtable webhook data is being processed
* Airtable events coming through
* Defined the Airtable table payload type
* TriggerSource metadata is being stored and used
* Removed some logs
* Added filtering and don’t allow any webhooks that use automated sources
* Resend switched to new integration
* Moved Resend test jobs to the catalog, and tested it worked
* WIP on Slack, there are compile errors
* Created a generic type that strips out indexes
* Slack updated to use new integration
* SendGrid migrated over
* Integration runTask is now allowing regular types
* Changed io.runTask types so it only allows Json-able types
* OpenAI models tasks working
* Added Airtable changes to runTask
* Removed the index signature crap from the Slack integration
* Don’t need to cast the callback result
* Updated Resend
* Re-ordered runTask params
* WIP on openai
* onAccountUpdated is Connect only
* Removed RunTaskResult
* Handle Resend errors, the official SDK doesn’t expose them properly at the moment
* Removed OmitIndexSignature
* OpenAI converted to new integration, with backwards compatible functions
* Put the openai catalog back to what it was originally
* Export a standard retry with backoff, to be used
* Use the standard exponential backoff in the integrations
* Retry options moved earlier so they can be overriden by a task
* GitHub tasks migrated
* Added sources, fixed one bundling issue
* Added GitHub jobs to catalog
* Remove duplicate options
* Deduplicate events
* Removed duplicate Job
* Switched Plain over
* Set the Plain icon
* Converted Stripe over
* Supabase adapted
* Typeform working
* Added dynamic-schedule to catalog
* Added background-fetch job catalog
* Created dynamic-triggers catalog file
* Fixed old general file with runTask param order
* Dynamic triggers working
* SendGrid updated to use the same tsconfig as other integrations
* Removed Airtable webhook, until we have batch support
* Added OAuth airtable auth example
* Created beta changeset tag
* Beta changesets for most packages
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
Performance degradation came from the syntax highlighting of large code blocks and from doing that on the server and the client, so fixed this in a couple of ways:
1. Stream the task details data using defer and Suspense/Await
2. Skipped syntax highlighting code blocks with more than 1k lines
* #344 Upgrading the openai package to v4
* Create neat-roses-wait.md
* Fixed a couple of issues with the openai v4 upgrade
* Added fine tuning job tasks
* WIP job run performance improvements
- Added a `perf` tool to better measure job run performance under heavy load
- Removed `runFinished` job (not really needed)
- startQueuedRuns now uses a jobKey with replace
- Fixed an issue with ZodWorker when using jobKey
* Publish improvement docker images
* fixed the improvement docker publishing
* Downgrade back to prisma 4.16.0 because 5.1.x broke docker builds
* Changes to how queued runs work
- Split the worker into two different workers, one dedicated to performRunExecution
- Schedule performRunExecution in a single place, with a queue and using a round robin manually controlled concurrency
- Remove startQueuedRuns
- All runs are queued before they are started
- Setting the worker maxPoolSize to the same as the worker concurrency
- Starting to be able to split the docker image
* Remove queue name from startRun graphile job
* Make the prisma connection pool stuff configurable through env vars
* Hardcode (for now) the max concurrent runs limit
* Rewrite performRunExecution to be more performant
PerformRunExecutionV2:
- Does not create and manage jobRunExecution records
- Does not reimplement retrying, uses graphile worker retrying instead
I’ve kept around PerformRunExecutionV1 so this works when deploying. Definitely needs LOTS of testing
* Fix issues with cached tasks
- Limit the size of the cached tasks sent when executing a run, using the knapsack problem dynamic programming approach
- Actually USE the cached tasks in IO by using the idempotencyKey instead of the task ID
- Remove output from all logs
- Added a stress test job catalog
* Forgot to commit the logger updates
* Never log connectionString
* Login to docker hub to get around rate limits
* Add additional logging to the graphile workers
* Fix the *_ENABLED env vars
* Allow adding and removing jobs to be done from the webapp
* Don’t set the job to failed if it’s being retried
* Deprecated queue options in the job and removed startPosition. Now using the job/env combo as the job queue name
* Dequeung jobs doesn’t check if the runner is initialized
* Fixed issues with retrying a run getting stuck on a cancelled task, and errors from parsing the results of dequeing a job
* Remove queued round robin thing that isn’t used anymore
* Added slack to job catalog
* Better forwards compat
* Added long delay
* Fixed lock file
* test: add test to no-duplicated-task-keys rule
* feat: implement no-duplicated-task-keys
* chore: create testing scripts
* refactor: fill in rule meta
* feat: export no-duplicated-task-keys rule
* chore: bump package version to 0.0.1
* chore: create eslint plugin
* feat: create no-duplicated-task-keys rule
* refactor: delete no-duplicated-task-keys from config-custom
* refactor: rename eslint-plugin folder
* feat: cover case from nextjs-example
* feat: cover additional cases from examples/package-tester
* chore: add eslint-plugin on nextjs-example
* feat: add more cases from `nextjs-example`
* revert: rollback changes on eslint-config-custom
* Set version to 2.0.9, inline with other packages
* Create chilly-pianos-try.md
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* FIx: [TRI-1006] New @trigger.dev/cli whoami command.
* revert dev.ts zod schema to original
* Removed telemetry
* remove all telemetry
* Made the clientId optional again
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* feature: initial work for sendgrid integration
* fix: fixed the ts-error and added sendgrid integration to the job-catalogue
* SendGrid added to dependencies
* Made the from email an env var so it can be tested by non-Trigger.dev team members
* Refactor: Modify sendEmail function to return void and remove SendEmailResponse type
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Create funny-kangaroos-glow.md
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
* Setup project-wide prettier
* Remove old workspace file
* Remove old debugging directives
* New top-level .prettierignore
* Updated Prettier config settings
* Contrubuting guide: Fix for some bad code blocks
* Added more ignores
* Improved the format script command
* printWidth set to 100
* Formatted entire repo (pnpm run format)
* Changeset should ignore the example projects
* Removed note in Contributing instructions about not adding a changeset for internal
* Renamed @trigger.dev/internal to @trigger.dev/core. Set sdk and internal to be ES2020, so we don’t get errors about private identifiers
* Env vars in nextjs-example use square bracket syntax to avoid Turbo Repo errors
* Set the tsconfigs back for core and sdk
* @examples/nextjs compile error with undefined tasks
* Set the example projects to use ES2015 to avoid private modifier complaints
* package-tester example, which will use built packages
* package-tester package.json
* Created a readme for the package-tester
* Added all the packages to package-tester
* Create .env.local.example and readme instructions
* Added name to the package.json
* Upgraded @types/react and @types/react-dom everywhrre, so we can use server actions
* Upgraded @types/react and @types/react-dom everywhrre, so we can use server actions
* Reworked the react package build, so it generates separate files
* The SDK no longer bundles core
* package-tester setup with a server action and react hooks
* Added new SDK methods with logging
* Working tsup settings for client and server
* Explicit react hooks return types
* Added all the hooks for testing
* Added OpenAI step to the job to check types are still ok in integrations
* Get rid of rogue Changeset ignores
* Changeset: @trigger.dev/core is now a separate package
* Exited prerelease mode, added a changeset
* Changed @next to @latest
* Deleted seed.js, this shouldn’t be committed
* Ignore seed.js
* The webapp was importing @trigger.dev/core with a folder path rather package name…
* More webapp imports instead of @trigger.dev/core were a folder path
* Accidentally edited Stripe internal API url in the comment
* Added “sideEffects”: false so @trigger.dev/core is tree shaken by Remix
* WIP supabase integration
* supabase oauth working
* Supabase database triggers
* Specify postgres:14
* Limit refreshOAuthToken jobs to 10 attempts
* Better displaying types and removing onChange for now
* WIP on the supabase db client
* Finishing the supabase-js integration
* Adding changeset
* Added supabase to the integration catalogs, and added an optional icon to Integrations
* Reworking how we handle types for the triggers (wip)
* Update fully over to the new way to define supabase triggers
* Go back to using the type for the event name
* Add back in the icon to the JobListPresenter since it was moved from the ProjectPresenter
* Remove unused import
* chore(contrib): added a few jobs
* chore(contrib): adds sample jobs for repo setup
* chore(contrib): refactor examples/jobs-starter
* docs(contrib): add steps to add starter jobs
* fix(contrib): typo
* chore(contrib): minor edits
* chore(contrib): update guide and starter code
Matt Aitken