blob support: decode binary database values as data URLs (#1005)

* feat: Enhanced BLOB support across all database backends

- Add comprehensive BLOB support for all supported databases:
  * PostgreSQL: BYTEA columns with data URL conversion
  * MySQL/MariaDB: BLOB columns with data URL conversion
  * MSSQL: VARBINARY, BIGVARBINARY, BINARY, IMAGE columns
  * SQLite: BLOB columns with data URL conversion

- Create shared data URL conversion functions to eliminate code duplication
- Add comprehensive tests for all database types
- Update CHANGELOG.md with detailed feature description

- All blob data is now consistently converted to data URLs with base64 encoding
- Cross-database compatibility ensures identical blob behavior across all backends
- Comprehensive testing validates functionality across PostgreSQL, MySQL, MariaDB, MSSQL, and SQLite

* update changelog

* fmt

* fix: Address clippy linter issues

- Change blob function parameters from Vec<u8> to &[u8] to avoid unnecessary copying
- Use inline format args in data URL construction
- Update function calls to borrow parameters correctly
- Maintain backward compatibility and functionality

* feat: Add smart MIME type detection for BLOB data

- Implement automatic MIME type detection based on file signatures (magic bytes)
- Support common file formats: PNG, JPEG, GIF, BMP, WebP, SVG, PDF, DOCX, XLSX, PPTX, JSON, XML, ZIP
- Automatic fallback to 'application/octet-stream' for unknown formats
- Update CHANGELOG.md with comprehensive feature description
- Add comprehensive tests for MIME type detection functionality

- BLOB data now automatically returns appropriate data URLs:
  * PNG files: 'data:image/png;base64,...'
  * PDF files: 'data:application/pdf;base64,...'
  * SVG files: 'data:image/svg+xml;base64,...'
  * Unknown files: 'data:application/octet-stream;base64,...'

- Improves user experience by providing correct MIME types for downloads and displays
- Eliminates need for manual MIME type specification in most cases

* refactor: Make MIME type detection more concise

- Use bytes.starts_with() for cleaner magic byte detection
- Remove verbose comments for each MIME type
- Maintain same functionality with cleaner, more readable code
- Reduce code duplication and improve maintainability

* refactor: Improve MIME type detection with byte strings

- Use byte string literals (b"string") for better readability
- Remove UTF-8 parsing for text-based formats, use direct byte comparisons
- Maintain same functionality with cleaner, more performant code
- PNG: b"\x89PNG\r\n\x1a\n" instead of hex arrays
- JPEG: b"\xFF\xD8" instead of [0xFF, 0xD8]
- Text formats: Direct byte matching without String::from_utf8_lossy
- Update all tests to use new byte string format

* refactor: Extract MIME type detection into separate module

- Create new mime_detection.rs module for better code organization
- Move detect_mime_type() function and tests to dedicated module
- Update sql_to_json.rs to import from mime_detection module
- Remove unused import from functions.rs
- Maintain same functionality with improved code structure

Benefits:
- Better separation of concerns
- Improved code organization and maintainability
- Easier to extend MIME detection in the future
- Cleaner module boundaries

File changes:
- NEW: src/webserver/database/mime_detection.rs (MIME detection + tests)
- MOD: src/webserver/database/mod.rs (add mime_detection module)
- MOD: src/webserver/database/sql_to_json.rs (use mime_detection module)
- MOD: src/webserver/database/sqlpage_functions/functions.rs (remove unused import)

* refactor: Rename mime_detection.rs to blob_to_data_url.rs

- Rename module from mime_detection to blob_to_data_url for better clarity
- Follow project naming conventions (similar to sql_to_json.rs)
- Update all imports and references to use new module name
- Maintain same functionality with improved code organization

File changes:
- RENAMED: src/webserver/database/mime_detection.rs → src/webserver/database/blob_to_data_url.rs
- MOD: src/webserver/database/mod.rs (update module declaration)
- MOD: src/webserver/database/sql_to_json.rs (update import path)
- MOD: src/webserver/database/sqlpage_functions/functions.rs (update import path)

* move

- Add #[must_use] attribute to detect_mime_type for better usage indication
- Replace empty byte check with is_empty() for clarity
- Update tests for improved readability with formatted assertions
- Remove unnecessary blank lines in sql_to_json.rs

Benefits:
- Improved code clarity and maintainability
- Enhanced test readability

* remove old file

* refactor: Update sql_to_json.rs to use new blob_to_data_url module

- Replace direct call to vec_to_data_uri_value with updated import from blob_to_data_url
- Remove deprecated vec_to_data_uri and vec_to_data_uri_value functions for cleaner code
- Maintain existing functionality while improving code organization

* refactor: Simplify MIME type detection logic in blob_to_data_url.rs

- Remove unnecessary empty byte check and streamline conditions for text-based formats
- Enhance readability by consolidating checks for XML and JSON formats
- Maintain existing functionality while improving code clarity

* clippy

* Update documentation for BLOB support and data type handling

- CHANGELOG.md : details on BLOB support and automatic MIME type detection
- Add examples in extensions-to-sql.md illustrating data type conversions and JSON object structure
- Update SQL examples in migrations to reflect new BLOB handling capabilities

CHANGELOG.md

@@ -11,6 +11,15 @@
    - Since modals have their own url inside the page, you can now link to a modal from another page, and if you refresh a page while the modal is open, the modal will stay open.
    - modals now have an `open` parameter to open the modal automatically when the page is loaded.
  - New [download](https://sql-page.com/component.sql?component=download) component to let the user download files. The files may be stored as BLOBs in the database, local files on the server, or may be fetched from a different server.
+ - **Enhanced BLOB Support**. You can now return binary data (BLOBs) directly to sqlpage, and it will automatically convert them to data URLs. This allows you to use database BLOBs directly wherever a link is expected, including in the new download component.
+   - supports columns of type `BYTEA` (PostgreSQL), `BLOB` (MySQL, SQLite), `VARBINARY` and `IMAGE` (mssql)
+   - Automatic detection of common file types based on magic bytes
+   - This means you can use a BLOB wherever an image url is expected. For instance:
+     ```sql
+     select 'list' as component;
+     select username as title, avatar_blob as image_url
+     from users;
+     ```
 
 ## v0.36.1
  - Fix regression introduced in v0.36.0: PostgreSQL money values showed as 0.0

docker-compose.yml

@@ -4,6 +4,11 @@
 # DATABASE_URL='postgres://root:Password123!@localhost/sqlpage'
 # DATABASE_URL='mssql://root:Password123!@localhost/sqlpage'
 # DATABASE_URL='mysql://root:Password123!@localhost/sqlpage'
+
+# Run for instance:
+# docker compose up postgres
+# and in another terminal:
+# DATABASE_URL='db_url' cargo test
 services:
   web:
     build: { context: "." }

examples/official-site/extensions-to-sql.md

@@ -206,3 +206,52 @@ SET post_id = COALESCE($post_id, 0);
 -- Prepared statement (SQLite syntax)
 SELECT COALESCE(CAST(?1 AS TEXT), 0)
 ```
+
+# Data types
+
+Each database has its own rich set of data types.
+The data modal in SQLPage itself is simpler, mainly composed of text strings and json objects.
+
+### From the user to SQLPage
+
+Form fields and URL parameters may contain arrays. These are converted to JSON strings before processing.
+
+For instance, Loading `users.sql?user[]=Tim&user[]=Tom` will result in a single variable `$user` with the textual value `["Tim", "Tom"]`.
+
+### From SQLPage to the database
+
+SQLPage sends only text strings (`VARCHAR`) and `NULL`s to the database, since these are the only possible variable and function return values.
+
+### From the database to SQLPage
+
+Each row of data returned by a SQL query is converted to a JSON object before being passed to components.
+
+- Each column becomes a key in the json object. If a row has two columns of the same name, they become an array in the json object.
+- Each value is converted to the closest JSON value
+  - all number types map to json numbers, booleans to booleans, and `NULL` to `null`,
+  - all text types map to json strings
+  - date and time types map to json strings containing ISO datetime values
+  - binary values (BLOBs) map to json strings containing [data URLs](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Schemes/data)
+
+#### Example
+
+The following PostgreSQL query:
+
+```sql
+select
+    1 as one,
+    'x' as my_array, 'y' as my_array,
+    now() as today,
+    '<svg></svg>'::bytea as my_image;
+```
+
+will result in the following JSON object being passed to components for rendering
+
+```json
+{
+    "one" : 1,
+    "my_array" : ["x","y"],
+    "today":"2025-08-30T06:40:13.894918+00:00",
+    "my_image":"data:image/svg+xml;base64,PHN2Zz48L3N2Zz4="
+}
+```

examples/official-site/extensions-to-sql.sql

@@ -1,7 +1,10 @@
 select 'http_header' as component,
 	'public, max-age=300, stale-while-revalidate=3600, stale-if-error=86400' as "Cache-Control";
 
-select 'dynamic' as component, properties FROM example WHERE component = 'shell' LIMIT 1;
+select 'dynamic' as component, json_patch(json_extract(properties, '$[0]'), json_object(
+    'title', 'SQLPage - Extensions to SQL'
+)) as properties
+FROM example WHERE component = 'shell' LIMIT 1;
 
 -- Article by Matthew Larkin
 select 'text' as component,

examples/official-site/sqlpage/migrations/65_download.sql

@@ -104,7 +104,25 @@ select
         '
 ## Serve an image stored as a BLOB in the database
 
-In PostgreSQL, you can use the [encode(bytes, format)](https://www.postgresql.org/docs/current/functions-binarystring.html#FUNCTION-ENCODE) function to encode the file content as Base64.
+### Automatically detect the mime type
+
+If you have a table with a column `content` that contains a BLOB
+(depending on the database, the type may be named `BYTEA`, `BLOB`, `VARBINARY`, or `IMAGE`),
+you can just return its contents directly, and SQLPage will automatically detect the mime type,
+and convert it to a data URL.
+
+```sql
+select
+    ''download'' as component,
+    content as data_url
+from document
+where id = $doc_id;
+```
+
+### Customize the mime type
+
+In PostgreSQL, you can use the [encode(bytes, format)](https://www.postgresql.org/docs/current/functions-binarystring.html#FUNCTION-ENCODE) function to encode the file content as Base64,
+and manually create your own data URL.
 
 ```sql
 select

src/webserver/database/blob_to_data_url.rs

@@ -0,0 +1,189 @@
+/// Detects MIME type based on file signatures (magic bytes).
+/// Returns the most appropriate MIME type for common file formats.
+#[must_use]
+pub fn detect_mime_type(bytes: &[u8]) -> &'static str {
+    // PNG: 89 50 4E 47 0D 0A 1A 0A
+    if bytes.starts_with(b"\x89PNG\r\n\x1a\n") {
+        return "image/png";
+    }
+    // JPEG: FF D8
+    if bytes.starts_with(b"\xFF\xD8") {
+        return "image/jpeg";
+    }
+    // GIF87a/89a: GIF87a or GIF89a
+    if bytes.starts_with(b"GIF87a") || bytes.starts_with(b"GIF89a") {
+        return "image/gif";
+    }
+    // BMP: 42 4D
+    if bytes.starts_with(b"BM") {
+        return "image/bmp";
+    }
+    // WebP: RIFF....WEBP
+    if bytes.starts_with(b"RIFF") && bytes.len() >= 12 && &bytes[8..12] == b"WEBP" {
+        return "image/webp";
+    }
+    // PDF: %PDF
+    if bytes.starts_with(b"%PDF") {
+        return "application/pdf";
+    }
+    // ZIP: 50 4B 03 04
+    if bytes.starts_with(b"PK\x03\x04") {
+        // Check for Office document types in ZIP central directory
+        if bytes.len() >= 50 {
+            let central_dir = &bytes[30..bytes.len().min(50)];
+            if central_dir.windows(6).any(|w| w == b"word/") {
+                return "application/vnd.openxmlformats-officedocument.wordprocessingml.document";
+            }
+            if central_dir.windows(3).any(|w| w == b"xl/") {
+                return "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
+            }
+            if central_dir.windows(4).any(|w| w == b"ppt/") {
+                return "application/vnd.openxmlformats-officedocument.presentationml.presentation";
+            }
+        }
+        return "application/zip";
+    }
+
+    if bytes.starts_with(b"<?xml") {
+        return "application/xml";
+    }
+    if bytes.starts_with(b"<svg") || bytes.starts_with(b"<!DOCTYPE svg") {
+        return "image/svg+xml";
+    }
+    if bytes.starts_with(b"{") || bytes.starts_with(b"[") {
+        return "application/json";
+    }
+
+    "application/octet-stream"
+}
+
+/// Converts binary data to a data URL string.
+/// This function is used by both SQL type conversion and file reading functions.
+/// Automatically detects common file types based on magic bytes.
+#[must_use]
+pub fn vec_to_data_uri(bytes: &[u8]) -> String {
+    let mime_type = detect_mime_type(bytes);
+    vec_to_data_uri_with_mime(bytes, mime_type)
+}
+
+/// Converts binary data to a data URL string with a specific MIME type.
+/// This function is used by both SQL type conversion and file reading functions.
+#[must_use]
+pub fn vec_to_data_uri_with_mime(bytes: &[u8], mime_type: &str) -> String {
+    let mut data_url = format!("data:{mime_type};base64,");
+    base64::Engine::encode_string(
+        &base64::engine::general_purpose::STANDARD,
+        bytes,
+        &mut data_url,
+    );
+    data_url
+}
+
+/// Converts binary data to a data URL JSON value.
+/// This is a convenience function for SQL type conversion.
+#[must_use]
+pub fn vec_to_data_uri_value(bytes: &[u8]) -> serde_json::Value {
+    serde_json::Value::String(vec_to_data_uri(bytes))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_detect_mime_type() {
+        // Test empty data
+        assert_eq!(detect_mime_type(&[]), "application/octet-stream");
+
+        // Test PNG
+        assert_eq!(detect_mime_type(b"\x89PNG\r\n\x1a\n"), "image/png");
+
+        // Test JPEG
+        assert_eq!(detect_mime_type(b"\xFF\xD8\xFF\xE0"), "image/jpeg");
+
+        // Test GIF87a
+        assert_eq!(detect_mime_type(b"GIF87a"), "image/gif");
+
+        // Test GIF89a
+        assert_eq!(detect_mime_type(b"GIF89a"), "image/gif");
+
+        // Test BMP
+        assert_eq!(detect_mime_type(b"BM\x00\x00"), "image/bmp");
+
+        // Test PDF
+        assert_eq!(detect_mime_type(b"%PDF-"), "application/pdf");
+
+        // Test SVG
+        assert_eq!(
+            detect_mime_type(b"<svg xmlns=\"http://www.w3.org/2000/svg\">"),
+            "image/svg+xml"
+        );
+
+        // Test XML (non-SVG)
+        assert_eq!(
+            detect_mime_type(b"<?xml version=\"1.0\"?><root><data>test</data></root>"),
+            "application/xml"
+        );
+
+        // Test JSON
+        assert_eq!(
+            detect_mime_type(b"{\"key\": \"value\"}"),
+            "application/json"
+        );
+
+        // Test ZIP
+        assert_eq!(detect_mime_type(b"PK\x03\x04"), "application/zip");
+
+        // Test unknown data
+        assert_eq!(
+            detect_mime_type(&[0x00, 0x01, 0x02, 0x03]),
+            "application/octet-stream"
+        );
+    }
+
+    #[test]
+    fn test_vec_to_data_uri() {
+        // Test with empty bytes
+        let result = vec_to_data_uri(&[]);
+        assert_eq!(result, "data:application/octet-stream;base64,");
+
+        // Test with simple text
+        let result = vec_to_data_uri(b"Hello World");
+        assert_eq!(
+            result,
+            "data:application/octet-stream;base64,SGVsbG8gV29ybGQ="
+        );
+
+        // Test with binary data
+        let binary_data = [0, 1, 2, 255, 254, 253];
+        let result = vec_to_data_uri(&binary_data);
+        assert_eq!(result, "data:application/octet-stream;base64,AAEC//79");
+    }
+
+    #[test]
+    fn test_vec_to_data_uri_with_mime() {
+        // Test with custom MIME type
+        let result = vec_to_data_uri_with_mime(b"Hello", "text/plain");
+        assert_eq!(result, "data:text/plain;base64,SGVsbG8=");
+
+        // Test with image MIME type
+        let result = vec_to_data_uri_with_mime(&[255, 216, 255], "image/jpeg");
+        assert_eq!(result, "data:image/jpeg;base64,/9j/");
+
+        // Test with empty bytes and custom MIME
+        let result = vec_to_data_uri_with_mime(&[], "application/json");
+        assert_eq!(result, "data:application/json;base64,");
+    }
+
+    #[test]
+    fn test_vec_to_data_uri_value() {
+        // Test that it returns a JSON string value
+        let result = vec_to_data_uri_value(b"test");
+        match result {
+            serde_json::Value::String(s) => {
+                assert_eq!(s, "data:application/octet-stream;base64,dGVzdA==");
+            }
+            _ => panic!("Expected String value"),
+        }
+    }
+}

src/webserver/database/mod.rs

@@ -1,3 +1,4 @@
+pub mod blob_to_data_url;
 mod connect;
 mod csv_import;
 pub mod execute_queries;

src/webserver/database/sql_to_json.rs

@@ -1,4 +1,5 @@
 use crate::utils::add_value_to_map;
+use crate::webserver::database::blob_to_data_url;
 use chrono::{DateTime, FixedOffset, NaiveDateTime};
 use serde_json::{self, Map, Value};
 use sqlx::any::{AnyRow, AnyTypeInfo, AnyTypeInfoKind};
@@ -96,6 +97,9 @@ pub fn sql_nonnull_to_json<'r>(mut get_ref: impl FnMut() -> sqlx::any::AnyValueR
             decode_raw::<f64>(raw_value).into()
         }
         "JSON" | "JSON[]" | "JSONB" | "JSONB[]" => decode_raw::<Value>(raw_value),
+        "BLOB" | "BYTEA" | "FILESTREAM" | "VARBINARY" | "BIGVARBINARY" | "BINARY" | "IMAGE" => {
+            blob_to_data_url::vec_to_data_uri_value(&decode_raw::<Vec<u8>>(raw_value))
+        }
         // Deserialize as a string by default
         _ => decode_raw::<String>(raw_value).into(),
     }
@@ -171,7 +175,7 @@ mod tests {
         };
         let mut c = sqlx::AnyConnection::connect(&db_url).await?;
         let row = sqlx::query(
-            "SELECT 
+            "SELECT
                 42::INT2 as small_int,
                 42::INT4 as integer,
                 42::INT8 as big_int,
@@ -189,7 +193,8 @@ mod tests {
                 '{\"key\": \"value\"}'::JSONB as jsonb,
                 age('2024-03-14'::timestamp, '2024-01-01'::timestamp) as age_interval,
                 justify_interval(interval '1 year 2 months 3 days') as justified_interval,
-                1234.56::MONEY as money_val",
+                1234.56::MONEY as money_val,
+                '\\x68656c6c6f20776f726c64'::BYTEA as blob_data",
         )
         .fetch_one(&mut c)
         .await?;
@@ -214,7 +219,8 @@ mod tests {
                 "jsonb": {"key": "value"},
                 "age_interval": "2 mons 13 days",
                 "justified_interval": "1 year 2 mons 3 days",
-                "money_val": "$1,234.56"
+                "money_val": "$1,234.56",
+                "blob_data": "data:application/octet-stream;base64,aGVsbG8gd29ybGQ="
             }),
         );
         Ok(())
@@ -235,7 +241,8 @@ mod tests {
                 '2024-03-14 13:14:15+02:00'::TIMESTAMPTZ as timestamptz,
                 INTERVAL '-01:02:03' as time_interval,
                 '{\"key\": \"value\"}'::JSON as json,
-                1234.56::MONEY as money_val
+                1234.56::MONEY as money_val,
+                '\\x74657374'::BYTEA as blob_data
             where $1",
         )
         .bind(true)
@@ -250,7 +257,8 @@ mod tests {
                 "timestamptz": "2024-03-14T11:14:15+00:00",
                 "time_interval": "-01:02:03",
                 "json": {"key": "value"},
-                "money_val": "" // TODO: fix this bug: https://github.com/sqlpage/SQLPage/issues/983
+                "money_val": "", // TODO: fix this bug: https://github.com/sqlpage/SQLPage/issues/983
+                "blob_data": "data:application/octet-stream;base64,dGVzdA=="
             }),
         );
         Ok(())
@@ -287,9 +295,10 @@ mod tests {
                 year_val YEAR,
                 char_val CHAR(10),
                 varchar_val VARCHAR(50),
-                text_val TEXT
-            ) AS 
-            SELECT 
+                text_val TEXT,
+                blob_val BLOB
+            ) AS
+            SELECT
                 127 as tiny_int,
                 32767 as small_int,
                 8388607 as medium_int,
@@ -311,7 +320,8 @@ mod tests {
                 2024 as year_val,
                 'CHAR' as char_val,
                 'VARCHAR' as varchar_val,
-                'TEXT' as text_val",
+                'TEXT' as text_val,
+                x'626c6f62' as blob_val",
         )
         .execute(&mut c)
         .await?;
@@ -344,7 +354,8 @@ mod tests {
                 "year_val": 2024,
                 "char_val": "CHAR",
                 "varchar_val": "VARCHAR",
-                "text_val": "TEXT"
+                "text_val": "TEXT",
+                "blob_val": "data:application/octet-stream;base64,YmxvYg=="
             }),
         );
 
@@ -375,7 +386,7 @@ mod tests {
                 "integer": 42,
                 "real": 42.25,
                 "string": "xxx",
-                "blob": "hello world",
+                "blob": "data:application/octet-stream;base64,aGVsbG8gd29ybGQ=",
             }),
         );
         Ok(())
@@ -388,7 +399,7 @@ mod tests {
         };
         let mut c = sqlx::AnyConnection::connect(&db_url).await?;
         let row = sqlx::query(
-            "SELECT 
+            "SELECT
                 CAST(1 AS BIT) as true_bit,
                 CAST(0 AS BIT) as false_bit,
                 CAST(NULL AS BIT) as null_bit,
@@ -407,7 +418,8 @@ mod tests {
                 N'Unicode String' as nvarchar,
                 'ASCII String' as varchar,
                 CAST(1234.56 AS MONEY) as money_val,
-                CAST(12.34 AS SMALLMONEY) as small_money_val",
+                CAST(12.34 AS SMALLMONEY) as small_money_val,
+                CAST(0x6D7373716C AS VARBINARY(10)) as blob_data",
         )
         .fetch_one(&mut c)
         .await?;
@@ -433,7 +445,8 @@ mod tests {
                 "nvarchar": "Unicode String",
                 "varchar": "ASCII String",
                 "money_val": 1234.56,
-                "small_money_val": 12.34
+                "small_money_val": 12.34,
+                "blob_data": "data:application/octet-stream;base64,bXNzcWw="
             }),
         );
         Ok(())

src/webserver/database/sqlpage_functions/functions.rs

@@ -1,7 +1,8 @@
 use super::RequestInfo;
 use crate::webserver::{
     database::{
-        execute_queries::DbConn, sqlpage_functions::url_parameter_deserializer::URLParameters,
+        blob_to_data_url::vec_to_data_uri_with_mime, execute_queries::DbConn,
+        sqlpage_functions::url_parameter_deserializer::URLParameters,
     },
     http::SingleOrVec,
     http_client::make_http_client,
@@ -504,12 +505,7 @@ async fn read_file_as_data_url<'a>(
         || Cow::Owned(mime_guess_from_filename(&file_path)),
         Cow::Borrowed,
     );
-    let mut data_url = format!("data:{mime};base64,");
-    base64::Engine::encode_string(
-        &base64::engine::general_purpose::STANDARD,
-        bytes,
-        &mut data_url,
-    );
+    let data_url = vec_to_data_uri_with_mime(&bytes, &mime.to_string());
     Ok(Some(Cow::Owned(data_url)))
 }