LearningCircuit
e56d7b6747
* fix(security): resolve open code-scanning alerts across four scanners Resolves 31 open code-scanning alerts (#8038-#8073 range) with real fixes where possible and convention-following baselining for historical findings. Scorecard Token-Permissions (#8039, #8040): - playwright-notes-tests.yml: drop checks:write / pull-requests:write and remove the cosmetic EnricoMi publish-unit-test-result step, following the same fix already applied to playwright-webkit-tests.yml (PR #3704). Failing tests still fail the job; the uploaded artifact keeps the report. - release.yml: trim the matching caller-side grants on notes-test-gate and remove the dead checks:write on test-gate (leftover from PR #3704 -- playwright-webkit-tests.yml no longer publishes check runs). Bearer XSS (#8041-#8052): - notes.js setSynthesizeButtonBusy, showNotesLoadError and unified_search.js showUnifiedSearchError: rebuild via createElement/textContent/replaceChildren, eliminating the innerHTML sinks outright (same pattern as the adjacent loading-state helpers). - notes.js renderNotes/synthesis previews and unified_search.js results render: add the missing bearer:disable directives; these sinks were already audited (escapeHtml on every interpolation / DOMPurify-backed renderMarkdown / escape-first basicMarkdownRender) but notes.js and unified_search.js never got the Bearer half of the repo's paired suppression convention. - annotation_surface.js: bearer:disable on the three replaceWith(...childNodes) unwrap calls -- Node-typed arguments are never HTML-parsed; rule false positive. DevSkim (#8065-#8067): - test_legacy_cleanup_no_plaintext.py: build the valid-shaped hex ids as "a" * 32 / "b" * 32 (idiom already used elsewhere in the file) instead of token-shaped literals. - test_faiss_safe_load_security.py: reword docstring prose so the literal substring pickle.load no longer trips DS425000 (the rule was matching documentation text, not the deliberate security-test code). Gitleaks credential-literal (#8057-#8064, #8068-#8073): All 14 findings are placeholders in historical commits, re-surfaced by the scheduled all-refs scan after PR #5133 renamed the custom rule (generic-secret -> credential-literal): fingerprints are rule-id-specific, so the existing baseline entries stopped matching. - .gitleaks.toml: allowlist the pre-reorg defaults/llm_config.py path (renamed in 8085f9bd3; same api_key="lm-studio" placeholder as the already-allowlisted current path). - .gitleaksignore: baseline the remaining historical fingerprints (benchmark123 throwaway test passwords, lm-studio dummy key, docstring example values in data_sanitizer.py, sk-ant-... docstring example). - research_functions.py: reword the docstring example to the established your-api-key-here placeholder so future edits don't re-trigger. Verified: gitleaks v8.30.0 (CI-pinned) full-history scan over 15101 commits reports no leaks; 57 tests pass including the bearer-disable format guard; eslint clean (0 errors, warnings pre-exist on main). #8038 (update-npm-dependencies.yml contents:write) is already best practice -- top-level permissions {} with job-scoped writes required by peter-evans/create-pull-request -- matching the won't-fix dismissal of the identical alert #7746. * fix(security): add missing bearer:disable on notes-grid render sink The renderNotes innerHTML assignment (alert #8045) was listed as annotated in the original commit but the directive was dropped; every interpolation in createNoteCard is escapeHtml'd or numeric, same audit as the sibling sinks. * style(annotations): tighten bearer:disable comments to repo convention Single terse audited-date line per site, em-dash to match file style. * fix(security): annotate replaceChildren sinks Bearer still flags A live Bearer v2.0.2 scan showed javascript_lang_dangerous_insert_html also fires on replaceChildren() calls with node arguments, so the four DOM-rebuild sites from the earlier commit would have re-surfaced as new alerts. All children at these sites are createElement/textContent-built Nodes; replaceChildren never parses HTML. Per-file scans of all three touched JS files now report zero findings.
333 lines
16 KiB
TOML
333 lines
16 KiB
TOML
title = "Gitleaks Configuration for Local Deep Research"
|
|
|
|
# NOTE: gitleaks is the sole secret scanner for this repository.
|
|
# detect-secrets was removed (2026-02) because its .secrets.baseline file
|
|
# (line-number-based tracking) caused constant merge conflicts across branches.
|
|
# gitleaks uses path-based and regex-based allowlists that are stable across
|
|
# line changes. CI also runs Semgrep (p/secrets) and Bearer (secrets) scanners.
|
|
# Do not re-add detect-secrets — use .gitleaks.toml allowlists instead.
|
|
|
|
# Extend default rules with custom ones
|
|
[extend]
|
|
useDefault = true
|
|
|
|
# Custom rules for this repository
|
|
[[rules]]
|
|
description = "GitHub personal access token"
|
|
id = "github-pat"
|
|
regex = '''ghp_[a-zA-Z0-9]{36}'''
|
|
keywords = ["ghp_"]
|
|
|
|
[[rules]]
|
|
description = "GitHub app token"
|
|
id = "github-app-token"
|
|
regex = '''(ghu|ghs|ghr)_[a-zA-Z0-9]{36}'''
|
|
keywords = ["ghu_", "ghs_", "ghr_"]
|
|
|
|
[[rules]]
|
|
description = "Google API key"
|
|
id = "google-api-key"
|
|
regex = '''AIza[0-9A-Za-z_-]{35}'''
|
|
keywords = ["AIza"]
|
|
|
|
[[rules]]
|
|
description = "Brave Search API key"
|
|
id = "brave-api-key"
|
|
regex = '''BS[a-zA-Z0-9_-]{32}'''
|
|
keywords = ["BS"]
|
|
|
|
[[rules]]
|
|
description = "SERPAPI key"
|
|
id = "serpapi-key"
|
|
regex = '''[a-zA-Z0-9]{64}'''
|
|
keywords = ["serpapi"]
|
|
|
|
[[rules]]
|
|
description = "Tavily API key"
|
|
id = "tavily-api-key"
|
|
regex = '''tvly-[a-zA-Z0-9_-]{40}'''
|
|
keywords = ["tvly-"]
|
|
|
|
[[rules]]
|
|
description = "SQLCipher database key"
|
|
id = "sqlcipher-key"
|
|
regex = '''PRAGMA key = ['"][a-zA-Z0-9/+]{44}['"]'''
|
|
keywords = ["PRAGMA key"]
|
|
|
|
[[rules]]
|
|
description = "Generic private key"
|
|
id = "private-key"
|
|
regex = '''-----BEGIN (RSA |OPENSSH |DSA |EC |PGP )?PRIVATE KEY-----'''
|
|
keywords = ["BEGIN PRIVATE KEY", "BEGIN RSA PRIVATE KEY", "BEGIN OPENSSH PRIVATE KEY"]
|
|
|
|
# The built-in generic-api-key rule intentionally ignores low-entropy values.
|
|
# Keep a narrow companion rule for quoted credential literals so PR scans still
|
|
# catch values such as `password = "summer2026"` without matching assignments
|
|
# from functions or identifiers such as `password = get_user_password(...)`.
|
|
[[rules]]
|
|
description = "Quoted credential literal"
|
|
id = "credential-literal"
|
|
regex = '''(?i)(?:^|[\s{,(])['"]?(?:(?:[a-z][a-z0-9]*[_-])+(?:password|passwd|pwd|secret)|password|passwd|pwd|secret|token|api[_-]?key|auth[_-]?token|access[_-]?token)['"]?\s*[:=]\s*['"]([^'"\r\n]{8,})['"]'''
|
|
secretGroup = 1
|
|
keywords = ["password", "passwd", "pwd", "secret", "token", "apikey", "api_key", "api-key"]
|
|
|
|
[[rules.allowlists]]
|
|
description = "Credential placeholders used in documentation and examples"
|
|
regexTarget = "secret"
|
|
regexes = [
|
|
'''(?i)^(?:not[-_ ]?a[-_ ]?secret|your[-_ ]?(?:password|secret)|replace[-_ ]?me|change[-_ ]?me|changeme|example[-_ ]?(?:password|secret)|dummy[-_ ]?(?:password|secret)|placeholder)$''',
|
|
]
|
|
|
|
# Allowlist - exclude common false positives
|
|
[allowlist]
|
|
description = "Global allowlist"
|
|
paths = [
|
|
'''\.git''',
|
|
'''\.venv''',
|
|
# detect-secrets baseline file (removed in PR #2476) contains SHA-1 hashes
|
|
# of known false positives — not actual secrets. Still appears in git history.
|
|
'''\.secrets\.baseline$''',
|
|
# Generated HTML documentation files (contain api_key parameter documentation, not secrets)
|
|
'''z_[a-f0-9]+_.*\.html$''',
|
|
'''node_modules''',
|
|
'''\.pytest_cache''',
|
|
'''__pycache__''',
|
|
'''coverage''',
|
|
'''\.coverage''',
|
|
'''test_results''',
|
|
'''screenshots''',
|
|
'''build''',
|
|
'''dist''',
|
|
'''\.mypy_cache''',
|
|
'''\.tox''',
|
|
'''tests/.*\.py$''',
|
|
'''tests/.*\.js$''',
|
|
'''examples/.*\.py$''',
|
|
'''.*test.*\.py$''',
|
|
'''.*test.*\.js$''',
|
|
'''.*example.*\.py$''',
|
|
'''mock.*\.py$''',
|
|
'''fixture.*\.py$''',
|
|
# Note: Workflow files are scanned - use specific commit allowlists for test credentials
|
|
'''src/local_deep_research/security/file_integrity/.*\.py$''',
|
|
'''src/local_deep_research/library/download_management/.*\.py$''',
|
|
'''src/local_deep_research/research_library/services/download_service\.py$''',
|
|
'''src/local_deep_research/research_library/routes/library_routes\.py$''',
|
|
'''src/local_deep_research/research_library/downloaders/.*\.py$''', # Contains API key parameters (not actual secrets)
|
|
'''src/local_deep_research/news/subscription_manager/scheduler\.py$''',
|
|
'''pdm\.lock$''', # Contains SHA256 package hashes (not secrets)
|
|
# LLM and embedding provider implementations use api_key/provider_key variable names (not actual secrets)
|
|
'''src/local_deep_research/llm/providers/.*\.py$''',
|
|
'''src/local_deep_research/embeddings/providers/.*\.py$''',
|
|
# LLM config dispatcher constructs ChatOpenAI/ChatOllama with api_key= kwargs (not actual secrets)
|
|
'''src/local_deep_research/config/llm_config\.py$''',
|
|
# Pre-reorg path of the same file (commit 8085f9bd3 renamed
|
|
# defaults/llm_config.py -> config/llm_config.py). Its historical blobs
|
|
# contain the same api_key="lm-studio" placeholder and still surface on
|
|
# full-history scans (gitleaks.yml --log-opts=--all).
|
|
'''src/local_deep_research/defaults/llm_config\.py$''',
|
|
# Web search engines use api_key variable names (not actual secrets)
|
|
'''src/local_deep_research/web_search_engines/engines/.*\.py$''',
|
|
'''src/local_deep_research/web_search_engines/search_engine_factory\.py$''',
|
|
# search_engines_config resolves api_key from settings_snapshot (not actual secrets)
|
|
'''src/local_deep_research/web_search_engines/search_engines_config\.py$''',
|
|
# Benchmark service uses user_password parameter names (not actual secrets)
|
|
'''src/local_deep_research/benchmarks/web_api/benchmark_service\.py$''',
|
|
'''src/local_deep_research/web_search_engines/search_engine_base\.py$''',
|
|
# HTML templates use csrfToken variable names (not actual secrets)
|
|
'''src/local_deep_research/web/templates/.*\.html$''',
|
|
# Notification flow docs contain example code with parameter names
|
|
'''docs/NOTIFICATION_FLOW\.md$''',
|
|
# UI test files contain test credentials (intentional for testing)
|
|
'''tests/ui_tests/.*\.js$''',
|
|
# Puppeteer E2E test files contain test credentials (intentional for testing)
|
|
'''tests/puppeteer/.*\.js$''',
|
|
# CI scripts contain test credentials (intentional for testing)
|
|
'''scripts/ci/.*\.sh$''',
|
|
# Web routes use api_key/password parameter names (not actual secrets)
|
|
'''src/local_deep_research/web/routes/.*\.py$''',
|
|
'''src/local_deep_research/web/services/.*\.py$''',
|
|
# JavaScript files use csrfToken variable names (not actual secrets)
|
|
'''src/local_deep_research/web/static/js/.*\.js$''',
|
|
# Database session context uses password variable names (not actual secrets)
|
|
'''src/local_deep_research/database/session_context\.py$''',
|
|
# Database middleware passes password variables between Flask g context and session stores (not actual secrets)
|
|
'''src/local_deep_research/web/auth/database_middleware\.py$''',
|
|
# Password utility helper resolves session password (variable names, not actual secrets)
|
|
'''src/local_deep_research/web/auth/password_utils\.py$''',
|
|
# SQLCipher utils has key/password variable names inherent to encryption operations (not actual secrets)
|
|
'''src/local_deep_research/database/sqlcipher_utils\.py$''',
|
|
# Environment setting definitions use key= parameter names (not actual secrets)
|
|
'''src/local_deep_research/settings/env_definitions/.*\.py$''',
|
|
# RAG routes and services use db_password variable names (not actual secrets)
|
|
'''src/local_deep_research/research_library/routes/rag_routes\.py$''',
|
|
'''src/local_deep_research/research_library/services/library_rag_service\.py$''',
|
|
# RAG service factory passes db_password variable names (not actual secrets)
|
|
'''src/local_deep_research/research_library/services/rag_service_factory\.py$''',
|
|
# Background scheduler passes db_password variable names to DB sessions and
|
|
# the RAG factory (not actual secrets)
|
|
'''src/local_deep_research/scheduler/background\.py$''',
|
|
# Encrypted DB manager uses key derivation and password parameters (not actual secrets)
|
|
'''src/local_deep_research/database/encrypted_db\.py$''',
|
|
# Metrics search tracker uses password variable names for DB credentials (not actual secrets)
|
|
'''src/local_deep_research/metrics/search_tracker\.py$''',
|
|
# Auto-generated configuration docs reference env var names like ENCRYPTION_KEY (not actual secrets)
|
|
'''docs/CONFIGURATION\.md$''',
|
|
# Chat routes use password retrieval helper (same pattern as web routes, not actual secrets)
|
|
'''src/local_deep_research/chat/routes\.py$''',
|
|
'''src/local_deep_research/research_library/services/research_history_indexer\.py$''',
|
|
'''src/local_deep_research/research_library/search/services/research_history_indexer\.py$''',
|
|
'''src/local_deep_research/web/queue/processor_v2\.py$''',
|
|
# Auth routes pass password variables for DB operations (not actual secrets)
|
|
'''src/local_deep_research/web/auth/routes\.py$''',
|
|
# Backup service uses password variable for encryption key derivation (not actual secrets)
|
|
'''src/local_deep_research/database/backup/.*\.py$'''
|
|
]
|
|
files = [
|
|
"\\.env\\.example$",
|
|
"\\.env\\.template$",
|
|
"config\\.example\\..*$",
|
|
"docker-compose\\.example\\.yml$"
|
|
]
|
|
commits = [
|
|
# AI reviewer documentation commit with API key explanation
|
|
"b31ff6fec15c9fe6f523d1f93f23e321b8803799",
|
|
# PDF upload feature with CSRF token DOM reading
|
|
"e0660dd0da5efed3ddbfcf5cfa1985b39f33df16",
|
|
# Download system integration with encrypted database - legitimate password parameters
|
|
"65ebe6bd522152ebb00a1fce38b8f6acb8fdcc7d",
|
|
# OpenRouter documentation commits with placeholder API keys (fixed in later commits)
|
|
"1395290fa5e8a955d5e01ed056e3ee7bad1b888c",
|
|
"00f48008cfedcc01adedc2b3f84246e0e3d883ef",
|
|
"17e4404af5b891b78e7eda99c196f03710e7107b",
|
|
# Notification system commit with user_password parameter names (false positives)
|
|
"17762f9e1b48db91092a867195aca81cfce6ab9c",
|
|
# PDM lock file update with SHA256 content hashes (false positives)
|
|
"3dd78e593550623af172c4d9f857197fdb1dfc99",
|
|
# Old .env.template commits with placeholder values (not actual secrets)
|
|
"40ca8fff073c20291758841c334081831f75a224",
|
|
"02042634506c377e86161cc2ce038eb8c19f10f3",
|
|
"03eba268505e3f239643041ce9c3d367d9b42700",
|
|
"08eaba75f4e3fb46f72668d3cc1f1d5eb4254f6b",
|
|
# README.md commits with placeholder API key examples (not actual secrets)
|
|
"27539371e7d2b2b6edb63bc238a6c0347fafef0c",
|
|
"792da694efe22304469ec22ff46201080791e3a6",
|
|
# Generated HTML documentation files with api_key parameter docs
|
|
"9acf5c42fb7ef721499e6d3a637f4faac69bc96b",
|
|
# Coverage HTML report files with api_key parameter names in docstrings (false positives)
|
|
"bdf812d1dd0efa81971d7500593e5c25339c676c",
|
|
# full_serp_search_results.py example code with placeholder API key comments
|
|
"f4a447e3cf4bb53ee5a88384dd8fd4c1ce2a9b4c",
|
|
"a831a13c5a43cf34eaf3c52495db2e4c2a0005b7",
|
|
# openai_endpoint provider with api_key = "not-needed" placeholder (not a real secret)
|
|
"e1b7322c73165be9ac651804b7babecc85c0d7df"
|
|
]
|
|
regexes = [
|
|
# Allow placeholder values that are clearly not real secrets
|
|
'''your-.*-key-here''',
|
|
'''your-api-key-here''',
|
|
'''your-openai-key-here''',
|
|
'''your-google-key-here''',
|
|
# Allow comments that explain API key usage but don't contain actual keys
|
|
'''# API_KEY: OpenRouter API key''',
|
|
'''OPENROUTER_API_KEY''',
|
|
# Allow JavaScript variable declarations that happen to use "token" as a variable name
|
|
'''const token = document\.querySelector''',
|
|
'''let token = document\.querySelector''',
|
|
'''var token = document\.querySelector''',
|
|
# Allow CSRF token reading from DOM meta tag (standard web CSRF protection pattern)
|
|
'''const csrfToken = document\.querySelector\(['"](meta\[name=)''',
|
|
# Allow test credentials used in CI/CD workflows
|
|
'''testpass123''',
|
|
'''perf-test-secret-key''',
|
|
# Password parameters and variable assignments (not actual secrets)
|
|
'''self\.password = password''',
|
|
'''self\.username = username''',
|
|
# Flask g context password variable assignment (not actual secrets)
|
|
'''g\.user_password\s*=\s*password''',
|
|
'''password: Optional\[str\]''',
|
|
'''username: str, password: Optional''',
|
|
# Database session context - parameter passing
|
|
'''get_user_db_session\(self\.username, self\.password\)''',
|
|
'''get_user_db_session\(username, password\)''',
|
|
'''username=username, password=password''',
|
|
# Session password storage (encrypted in session)
|
|
'''user_password = session_password_store''',
|
|
'''get_session_password''',
|
|
'''user_password = getattr''',
|
|
# Allow specific password variable patterns with safe context (not arbitrary assignments)
|
|
'''user_password\s*=\s*getattr\(g''',
|
|
'''user_password\s*=\s*session_password_store\.get_session_password''',
|
|
'''user_password\s*=\s*user_password''',
|
|
'''user_password=user_password''',
|
|
'''session_password_store\.get_session_password''',
|
|
'''password\)\s*:''',
|
|
# Allow type hints only (not assignments)
|
|
'''\bpassword:\s*Optional\[str\]\s*[=)]''',
|
|
'''\bpassword:\s*str\s*[=)]''',
|
|
'''password\s*:\s*Optional\[str\]\s*=\s*None''',
|
|
# Allow getattr patterns for password retrieval from Flask g context
|
|
'''getattr\(g,\s*["']user_password["']''',
|
|
# Allow SHA256 hashes in pdm.lock (package integrity hashes)
|
|
'''hash\s*=\s*"sha256:[a-fA-F0-9]{64}"''',
|
|
'''content_hash\s*=\s*"sha256:[a-fA-F0-9]{64}"''',
|
|
# Allow CSRF token meta tag selectors (JavaScript, not actual secrets)
|
|
'''csrf_token\s*=\s*document\.querySelector\(['"](meta\[name=)''',
|
|
# Allow csrfToken variable assignment in JavaScript/HTML
|
|
'''const csrfToken = document\.querySelector''',
|
|
'''let csrfToken = document\.querySelector''',
|
|
# Allow api_key variable assignment from settings (not actual secrets)
|
|
'''api_key\s*=\s*get_setting_from_snapshot''',
|
|
'''self\.api_key\s*=\s*settings_snapshot\.get''',
|
|
'''self\.api_key\s*=\s*get_setting_from_snapshot''',
|
|
# Allow provider_key constant definitions
|
|
'''provider_key\s*=\s*"[A-Z_]+"''',
|
|
# Allow test credentials in test files (pragma comments indicate intentional)
|
|
'''Test@Pass123!''',
|
|
'''TestPass123!''',
|
|
# Allow api_key type hints and docstrings
|
|
'''api_key:\s*Optional''',
|
|
'''api_key\s*=\s*None''',
|
|
# Allow placeholder API keys for local servers without auth.
|
|
# "not-required" is the unified placeholder (providers/base.py
|
|
# OPTIONAL_API_KEY_PLACEHOLDER). The other strings are kept for
|
|
# backward compat with any third-party provider classes that haven't
|
|
# migrated to the constant yet.
|
|
'''api_key\s*=\s*"not-required"''',
|
|
'''api_key\s*=\s*"not-needed"''',
|
|
'''api_key\s*=\s*"lm-studio"''',
|
|
# Allow research_id, user_password in notification flow docs (example code)
|
|
'''research_id,\s*user_password''',
|
|
# Allow CSRF token patterns in HTML templates
|
|
'''csrf[Tt]oken\s*=\s*document\.querySelector''',
|
|
# Thread context password retrieval (session_context.py)
|
|
'''thread_context\.get\("user_password"\)''',
|
|
'''thread_context\["user_password"\]''',
|
|
'''if not password:''',
|
|
# Database password variable patterns (not actual secrets)
|
|
'''db_password\s*=\s*session_password_store\.get_session_password''',
|
|
'''db_password\s*=\s*db_password''',
|
|
'''db_password=db_password''',
|
|
'''db_password=user_password''',
|
|
'''db_password:\s*Optional\[str\]''',
|
|
'''self\._db_password\s*=\s*db_password''',
|
|
'''embedding_manager\.db_password\s*=\s*db_password''',
|
|
'''rag_service\.db_password\s*=\s*db_password''',
|
|
# SQLCipher key derivation function calls (not actual secrets)
|
|
'''key\s*=\s*_get_key_from_password''',
|
|
'''key\s*=\s*_get_key_from_password_cached''',
|
|
# Chat routes / auth helpers (not actual secrets)
|
|
'''auth_data\s*=\s*temp_auth_store\.peek_auth''',
|
|
'''auth_token\s*=\s*session\.get\("temp_auth_token"\)''',
|
|
'''token\s*=\s*session\.get\("temp_auth_token"\)'''
|
|
]
|
|
|
|
# NOTE: a "[target]" section used to live here listing paths/extensions,
|
|
# intending to restrict the scan to src/, scripts/, docs/, .github/ and code
|
|
# extensions. gitleaks has no such config key — it was silently ignored, so
|
|
# the scan already covers the whole tree regardless (verified: a secret in a
|
|
# root .txt file outside every listed path is still flagged with the section
|
|
# present). Removed to avoid implying a scope that was never enforced. Scan
|
|
# scope is set by the CLI (--source / --log-opts); to EXCLUDE paths, add them
|
|
# to [allowlist].paths above.
|