Added LLM evaluator to example projects (#1589)

* Added repo links * Added links to remix / nextjs webhooks examples * Added a Realtime learn more snippet * Added the batch llm evaluator docs * Improved snippet * Updated intro
2025-01-08 21:16:56 +00:00
parent 3ca0c15161
commit d52e3d7d57
10 changed files with 121 additions and 11 deletions
@@ -0,0 +1,49 @@
+---
+title: "Batch LLM Evaluator"
+sidebarTitle: "Batch LLM Evaluator"
+description: "This example project evaluates multiple LLM models using the Vercel AI SDK and streams updates to the frontend using Trigger.dev Realtime."
+---
+
+import RealtimeLearnMore from "/snippets/realtime-learn-more.mdx";
+
+## Overview
+
+This demo is a full stack example that uses the following:
+
+- A [Next.js](https://nextjs.org/) app with [Prisma](https://www.prisma.io/) for the database.
+- Trigger.dev [Realtime](https://trigger.dev/launchweek/0/realtime) to stream updates to the frontend.
+- Work with multiple LLM models using the Vercel [AI SDK](https://sdk.vercel.ai/docs/introduction). (OpenAI, Anthropic, XAI)
+- Distribute tasks across multiple tasks using the new [`batch.triggerByTaskAndWait`](https://trigger.dev/docs/triggering#batch-triggerbytaskandwait) method.
+
+## GitHub repo
+
+<Card
+  title="View the Batch LLM Evaluator repo"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/batch-llm-evaluator"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
+## Video
+
+<video
+  controls
+  className="w-full aspect-video"
+  src="https://content.trigger.dev/batch-llm-evaluator.mp4"
+></video>
+
+## Relevant code
+
+- View the Trigger.dev task code in the [src/trigger/batch.ts](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/trigger/batch.ts) file.
+- The `evaluateModels` task uses the `batch.triggerByTaskAndWait` method to distribute the task to the different LLM models.
+- It then passes the results through to a `summarizeEvals` task that calculates some dummy "tags" for each LLM response.
+- We use a [useRealtimeRunsWithTag](https://trigger.dev/docs/frontend/react-hooks/realtime#userealtimerunswithtag) hook to subscribe to the different evaluation tasks runs in the [src/components/llm-evaluator.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/llm-evaluator.tsx) file.
+- We then pass the relevant run down into three different components for the different models:
+  - The `AnthropicEval` component: [src/components/evals/Anthropic.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/evals/Anthropic.tsx)
+  - The `XAIEval` component: [src/components/evals/XAI.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/evals/XAI.tsx)
+  - The `OpenAIEval` component: [src/components/evals/OpenAI.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/evals/OpenAI.tsx)
+- Each of these components then uses [useRealtimeRunWithStreams](https://trigger.dev/docs/frontend/react-hooks/realtime#userealtimerunwithstreams) to subscribe to the different LLM responses.
+
+<RealtimeLearnMore />
@@ -4,6 +4,8 @@ sidebarTitle: "Realtime image gen with Fal.ai"
 description: "This example project generates an image from a prompt using Fal.ai and shows the progress of the task on the frontend using Trigger.dev Realtime."
 ---

+import RealtimeLearnMore from "/snippets/realtime-learn-more.mdx";
+
 ## Overview

 This full stack Next.js project showcases the following:
@@ -40,3 +42,5 @@ This video walks through the process of creating this task in a Next.js project.
    allowfullscreen
  />
 </div>
+
+<RealtimeLearnMore />
@@ -4,6 +4,17 @@ sidebarTitle: "Fal.ai with Realtime"
 description: "This example task generates an image from a prompt using Fal.ai and shows the progress of the task on the frontend using Trigger.dev Realtime."
 ---

+## GitHub repo
+
+<Card
+  title="View the project on GitHub"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/realtime-fal-ai-image-generation"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
 ## Walkthrough

 This video walks through the process of creating this task in a Next.js project.
@@ -11,6 +11,17 @@ import VercelDocsCards from "/snippets/vercel-docs-cards.mdx";
 - [A Next.js project, set up with Trigger.dev](/guides/frameworks/nextjs)
 - [cURL](https://curl.se/) installed on your local machine. This will be used to send a POST request to your webhook handler.

+## GitHub repo
+
+<Card
+  title="View the project on GitHub"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/nextjs-webhooks/my-app"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
 ## Adding the webhook handler

 The webhook handler in this guide will be an API route.
@@ -9,6 +9,17 @@ description: "Learn how to trigger a task from a webhook in a Remix app."
 - [A Remix project, set up with Trigger.dev](/guides/frameworks/remix)
 - [cURL](https://curl.se/) installed on your local machine. This will be used to send a POST request to your webhook handler.

+## GitHub repo
+
+<Card
+  title="View the project on GitHub"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/remix-webhooks"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
 ## Adding the webhook handler

 The webhook handler in this guide will be an API route. Create a new file `app/routes/api.webhook-handler.ts` or `app/routes/api.webhook-handler.js`.
@@ -17,11 +17,6 @@ import NextjsTroubleshootingButtonSyntax from "/snippets/nextjs-button-syntax.md
 import WorkerFailedToStartWhenRunningDevCommand from "/snippets/worker-failed-to-start.mdx";
 import SupabaseDocsCards from "/snippets/supabase-docs-cards.mdx";

-<Info>
-  The project created in this guide can be found in this [GitHub
-  repo](https://github.com/triggerdotdev/example-projects/tree/main/supabase).
-</Info>
-
 ## Overview

 Supabase edge functions allow you to trigger tasks either when an event is sent from a third party (e.g. when a new Stripe payment is processed, when a new user signs up to a service, etc), or when there are any changes or updates to your Supabase database.
@@ -36,6 +31,17 @@ This guide shows you how to set up and deploy a simple Supabase edge function ex
 - [Create a Trigger.dev account](https://cloud.trigger.dev)
 - [Create a new Trigger.dev project](/guides/dashboard/creating-a-project)

+## GitHub repo
+
+<Card
+  title="View the project on GitHub"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/supabase-edge-functions"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
 ## Initial setup

 <Steps>
@@ -13,11 +13,6 @@ import NextjsTroubleshootingButtonSyntax from "/snippets/nextjs-button-syntax.md
 import WorkerFailedToStartWhenRunningDevCommand from "/snippets/worker-failed-to-start.mdx";
 import SupabaseDocsCards from "/snippets/supabase-docs-cards.mdx";

-<Info>
-  The project created in this guide can be found in this [GitHub
-  repo](https://github.com/triggerdotdev/example-projects/tree/main/supabase).
-</Info>
-
 ## Overview

 Supabase and Trigger.dev can be used together to create powerful workflows triggered by real-time changes in your database tables:
@@ -25,6 +20,7 @@ Supabase and Trigger.dev can be used together to create powerful workflows trigg
 - A Supabase Database Webhook triggers an Edge Function when a row including a video URL is inserted into a table
 - The Edge Function triggers a Trigger.dev task, passing the `video_url` column data from the new table row as the payload
 - The Trigger.dev task then:
+
  - Uses [FFmpeg](https://www.ffmpeg.org/) to extract the audio track from a video URL
  - Uses [Deepgram](https://deepgram.com) to transcribe the extracted audio
  - Updates the original table row using the `record.id` in Supabase with the new transcription using `update`
@@ -38,6 +34,17 @@ Supabase and Trigger.dev can be used together to create powerful workflows trigg
 - [Create a new Trigger.dev project](/guides/dashboard/creating-a-project)
 - [Create a new Deepgram account](https://deepgram.com/) and get your API key from the dashboard

+## GitHub repo
+
+<Card
+  title="View the project on GitHub"
+  icon="GitHub"
+  href="https://github.com/triggerdotdev/examples/tree/main/supabase-edge-functions"
+>
+  Click here to view the full code for this project in our examples repository on GitHub. You can
+  fork it and use it as a starting point for your own project.
+</Card>
+
 ## Initial setup

 <Steps>
@@ -40,6 +40,7 @@ Example projects are full projects with example repos you can fork and use. Thes
 | Example project                                                              | Description                                                                                                    | Framework | GitHub Repo                                                                                           |
 | :--------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------- | :-------- | :---------------------------------------------------------------------------------------------------- |
 | [Realtime Fal.ai image generation](/guides/example-projects/realtime-fal-ai) | Generate an image from a prompt using Fal.ai and show the progress of the task on the frontend using Realtime. | Next.js   | [View the repo](https://github.com/triggerdotdev/examples/tree/main/realtime-fal-ai-image-generation) |
+| [Batch LLM Evaluator](/guides/example-projects/batch-llm-evaluator)          | Evaluate multiple LLM models and stream the results to the frontend.                                           | Next.js   | [View the repo](https://github.com/triggerdotdev/examples/tree/main/batch-llm-evaluator)              |

 ## Example tasks

@@ -348,7 +348,10 @@
    },
    {
      "group": "Example projects",
-      "pages": ["guides/example-projects/realtime-fal-ai"]
+      "pages": [
+        "guides/example-projects/realtime-fal-ai",
+        "guides/example-projects/batch-llm-evaluator"
+      ]
    },
    {
      "group": "Example tasks",
@@ -0,0 +1,7 @@
+## Learn more about Trigger.dev Realtime
+
+To learn more, take a look at the following resources:
+
+- [Realtime docs](https://trigger.dev/docs/realtime) - learn more about Realtime.
+- [Batch Trigger docs](https://trigger.dev/docs/triggering) - learn more about Batch Triggers.
+- [React hooks](https://trigger.dev/docs/frontend/react-hooks) - learn more about React hooks.