- Build Nmap 7.95 from source instead of using Ubuntu package
- Removes incompatibility with sirius-nse scripts (e.g., shortport.ssh)
- Added automake, autoconf, libtool, perl for build dependencies
- Disabled ndiff (not needed) to avoid Python setuptools dependency
- Applied to both dev and runtime stages
- Adds ~3 minutes to build time but ensures script compatibility
- Add CORS_ALLOWED_ORIGINS env var support in API
- Default to '*' for production flexibility
- Update docker-compose.yaml with CORS configuration
- Fixes demo deployment connectivity issues
Process improvements based on GitHub Issue #60 resolution:
Lessons Learned Section:
- Added comprehensive lessons learned framework to Git operations documentation
- Documented what went well during issue #60 resolution
- Identified 6 key areas for improvement with specific impacts and solutions
- Created quarterly review process for continuous improvement
- Established template for future lessons learned entries
Testing Checklists:
- Created CHECKLIST.testing-by-type.md with issue-type-specific testing procedures
- Added checklists for: Frontend, Backend API, Database, Authentication, Docker, Documentation, Git Operations
- Included specific commands and troubleshooting for each type
- Added post-merge verification checklist
- Documented continuous improvement process for checklists
Key Improvements Identified:
1. Database management practices for testing (resolved in this cycle)
2. Testing checklists by issue type (resolved with new document)
3. GitHub issue comment templates (recommended for next iteration)
4. Documentation index maintenance reminders (noted)
5. Branch strategy documentation clarity (noted)
6. Rollback procedures (recommended)
Related to #60 - Continuous process improvement
- Added SQLite database files to .gitignore to prevent committing test data
- Removed dev.db and db.sqlite from git tracking
- Added Database Management section to sirius-ui README with reset instructions
- Documented default credentials (admin/password)
- Leverages existing Prisma seed script for database reset
Best Practice Implementation:
- Database files are now gitignored (industry standard for SQLite)
- Each developer maintains their own local database state
- Seed script allows easy reset to default credentials after testing
- Clear documentation for resetting database when needed
Related to #60 - Ensures testing password changes doesn't affect other developers
- Fixed type mismatch between NextAuth JWT (string) and Prisma schema (Int)
- Updated user.ts TRPC endpoints to coerce userId from string to integer
- Changed UserProfile interface id type from string to number
- Applied fix to all three affected endpoints: changePassword, updateProfile, getProfile
- Added user-reported issue handling section to Git operations documentation
- Updated documentation index to include missing files
Changes:
- sirius-ui/src/server/api/routers/user.ts: Use z.coerce.number().int().positive() for userId validation
- documentation/dev/operations/README.git-operations.md: Added workflow for handling user-reported issues
- documentation/README.documentation-index.md: Added missing documentation files to index
Testing:
- Requires container rebuild to test changes
- Manual verification of password change functionality needed
- Profile update functionality should also be tested
Closes#60
- Move developer-specific content from README.md to documentation/contributing.md
- Simplify README.md to focus on end-user experience
- Add comprehensive contributing guide with:
- Development environment setup
- Development workflow and best practices
- Testing and quality assurance
- Code standards and Git workflow
- PR submission guidelines
- Update documentation index with new contributing.md
- Update community links to reference local contributing guide
This makes the project more approachable for newcomers while providing
clear guidance for developers who want to contribute.
Fixes#77
This merge includes:
- Fixed go-api v0.0.10 with proper error checking
- Updated sirius-api to use working go-api version
- Added Dependabot for automated dependency updates
- Created bulk update script for manual dependency management
- Fixed system monitor data display in UI
- Removed docker-compose.prod.yaml in favor of single config
All tests passing, system monitor verified working in production.
- Add Dependabot configuration for Go modules, Docker, and GitHub Actions
- Create bulk update script (scripts/update-go-api.sh) for manual updates
- Update production go.mod/go.sum for sirius-api with go-api v0.0.10
- Enables automatic PR creation when go-api releases new versions
- Prevents recurring issues with stale dependencies
- Updates go-api from v0.0.9 to v0.0.10
- Fixes system monitor GetValue() returning empty strings
- Resolves issue where system metrics were not displaying in UI
Removed:
- docker-compose.prod.yaml (base docker-compose.yaml is production-ready)
Updated:
- testing/container-testing/test-build.sh (removed prod yaml test)
- testing/container-testing/Makefile (prod commands use base config)
- testing/container-testing/test-integration.sh (prod uses base config)
- testing/container-testing/test-integration-simple.sh (prod uses base config)
- testing/container-testing/test-health.sh (prod uses base config)
- testing/README.md (updated documentation)
Base docker-compose.yaml is configured for production:
- sirius-ui: target production
- sirius-api: target runner
- sirius-engine: target runtime
docker-compose.dev.yaml provides development overrides when needed.
Related to #77
Changes:
- Created sirius-api/start-dev.sh with conditional service startup
- Updated sirius-api/Dockerfile to use startup script instead of raw CMD
- Script checks if /system-monitor and /app-administrator exist before starting
- Continues to main API regardless of optional service availability
- Cleaned up temporary workflow files from tmp/
- Updated git-operations.md to require human approval before merging to main
Process Improvements:
- Added explicit human validation step in workflow (step 7)
- Added requirement to get approval before merging (step 8)
- Added cleanup step for temporary files (step 12)
- Updated best practices to mandate testing and approval
Testing:
- Handles missing volume mounts gracefully
- Shows warnings for unavailable services
- Main API starts successfully regardless
Related to #77
Fixes regression from #76
Production Mode Changes:
- Updated sirius-api Dockerfile to fix binary paths and add administrator build
- Updated sirius-engine Dockerfile to build and copy administrator to all stages
- Updated sirius-ui Dockerfile to build and copy administrator
- Fixed all binary paths to be consistent across containers
Development Mode Changes:
- Updated sirius-api CMD to start administrator with go run
- Updated sirius-engine start-enhanced.sh to detect and run source code in dev mode
- Updated sirius-ui start-dev.sh to start administrator with go run
Testing Strategy:
- Production mode: Verify binaries built and services start
- Development mode: Verify source mounted and services start with go run
- Process monitoring: Confirm both services appear in ps aux
- Log validation: Check logs show successful startup
Affected Components:
- sirius-api/Dockerfile (development and runner stages)
- sirius-engine/Dockerfile (builder, development, runtime stages)
- sirius-engine/start-enhanced.sh (administrator startup logic)
- sirius-ui/Dockerfile (production stage)
- sirius-ui/start-dev.sh (add administrator startup)
- documentation/dev/operations/README.git-operations.md (GitHub workflow)
Related to #76
- Update version to 0.4.0 in package.json
- Add comprehensive CHANGELOG.md with v0.4.0 features
- Update README.md with new monitoring capabilities
- Document system monitoring dashboard and observability features
- Highlight enhanced reliability and troubleshooting capabilities
This release introduces:
- Real-time system health monitoring
- Centralized logging infrastructure
- Performance metrics collection
- Enhanced container builds and testing
- SSH troubleshooting capabilities
- Comprehensive error handling and recovery
- Add sirius-demo branch to CI workflow triggers
- Add dispatch-demo-deployment job to automatically trigger sirius-demo repository deployment
- Uses repository_dispatch to connect main Sirius repository with sirius-demo infrastructure
- Ensures container build fixes automatically deploy to demo environment
- Fixes the disconnect between development and demo deployment workflows
- Remove validation for docker-compose.local.example.yaml (outdated template)
- Update CI workflow to check for absence of outdated template files
- Fixes CI pipeline validation failure caused by old implementation artifacts
- Ensures CI validates current Docker configuration approach
- Fix Go module dependency version conflicts between go-api and sirius-api
- Update go-api to v0.0.9 with valkey-go v1.0.60 for version consistency
- Update app-scanner to use go-api v0.0.9 to resolve build conflicts
- Implement industry best practices for Go module management in Docker
- Fix RabbitMQ health check logic in test scripts
- Clean up and optimize Dockerfile configurations
- Remove unnecessary local cloning approach in favor of published modules
- Update sirius-api go.mod.prod to use latest go-api version
- Ensure all container builds pass comprehensive testing
- All 41 tests now passing (9 build + 13 health + 19 integration)
This resolves the container build failures and ensures production readiness.
- Created PerformanceDashboard component with comprehensive metrics display
- Added performance monitoring tab to system monitor page
- Implemented mock performance data for testing and demonstration
- Added performance metrics API handler (backend route needs debugging)
- Features implemented:
* Performance summary cards (total requests, avg response time, error rate)
* Top endpoints analysis with request counts and performance metrics
* Detailed performance metrics table with timestamps and response times
* Time range filtering (1h, 24h, 7d, 30d)
* Real-time refresh functionality
* Color-coded performance indicators (green/yellow/red)
* Status code badges and error rate monitoring
The performance monitoring dashboard is now accessible via the new 'Performance' tab
in the system monitor page, providing comprehensive system performance visibility.
Note: Backend API route needs debugging - currently using mock data for demonstration.
- Updated all task statuses to 'done' in system-monitoring.json
- Verified system monitor page functionality
- Confirmed health check APIs working correctly
- Validated logging system with submission and retrieval
- All phases (0-4) completed successfully:
* Phase 0: Project Foundation - System monitor page, components, mock data
* Phase 1: Service Health Monitoring - Real health check APIs and frontend integration
* Phase 2: Centralized Logging Infrastructure - Valkey storage and log management
* Phase 3: Log Viewer and Dashboard Integration - Advanced filtering and real-time updates
* Phase 4: Testing and Documentation - Component testing and system validation
System monitoring dashboard is now fully functional with:
- Real-time service health monitoring for all 6 microservices
- Centralized logging system with search, filtering, and pagination
- Comprehensive API endpoints for health checks and log management
- Responsive UI with proper error handling and loading states
Comprehensive Logging API Features:
- POST /api/v1/logs - Submit new log entries with validation
- GET /api/v1/logs - Retrieve logs with advanced filtering and pagination
- GET /api/v1/logs/stats - Get comprehensive log statistics and metrics
- DELETE /api/v1/logs/clear - Clear all logs for maintenance
- Full CRUD operations for log management
- Advanced filtering (service, level, time range, search)
- Pagination support for large datasets
- Third-party application integration support
API Capabilities:
- Structured log entry validation using SDK types
- Environment-based configuration support
- Performance optimization with key limiting
- Comprehensive error handling and validation
- RESTful API design for external consumption
- No direct Valkey access required for third parties
Benefits:
- Enables external applications to consume Sirius logs
- Provides clean REST API interface for log management
- Supports log analysis and monitoring tools
- Maintains security by abstracting Valkey access
- Standardized log format across all services
Updated go-api to v0.0.8 with comprehensive logging API endpoints.
All go.mod files updated to use the latest version.
SDK Logging Refactor:
- Created sirius/logging package in go-api with structured types and client
- Implemented async buffering, retry logic, and environment configuration
- Added convenience methods for business events, performance metrics, and errors
- Updated API to use SDK-based middleware instead of custom logging
- Updated App Scanner to use SDK logging instead of custom implementation
- Removed old logging middleware and performance metric spam
- Added proper error logging for 404s and other HTTP errors
- Maintained selective logging (only errors and slow requests)
Benefits:
- Consistent logging interface across all services
- Centralized configuration and maintenance
- Better performance with async buffering
- Reduced log volume (no health check spam)
- Standardized log format and metadata structure
This provides a solid foundation for logging across all Sirius services.
Development Workflow Documentation:
- Added hot reloading information for extended development mode
- Documented that minor-projects changes are automatically reflected
- Explained that engine services use 'go run' for hot reloading with Air
- Clarified that changes are not tracked by main Sirius repository
- Added note about no need to rebuild containers for scanner/terminal/agent code
This documentation update reflects the successful integration of
centralized logging into the App Scanner service, which was
developed and tested using the extended development workflow
with hot reloading capabilities.
Performance Issues Fixed:
- Reduced logging frequency to only log errors, health checks, and slow requests
- Increased Valkey timeout from 10s to 30s for log maintenance operations
- Made log maintenance less aggressive (only runs every 10th log entry)
- Prevented logging cascade that was causing 8-20+ second response times
Changes:
- LoggingMiddleware now only logs important events (errors, health checks, slow requests)
- Log maintenance runs less frequently to avoid Valkey timeout issues
- System response times back to normal (4-21ms instead of 8-20+ seconds)
- All services showing healthy status again
The system is now stable and performant while still maintaining
comprehensive logging for important events and errors.
Phase 3 Progress: API Integration Complete
API Logging Middleware:
- Created LoggingMiddleware for automatic request/response logging
- Added ErrorLoggingMiddleware for comprehensive error capture
- Implemented PerformanceMetricsMiddleware for response time tracking
- Added request ID generation and tracking
- Integrated HTTP client for automatic log submission to centralized system
Features:
- Automatic logging of all API requests with detailed metadata
- Performance tracking (response times, request counts, response sizes)
- Error logging with full context and stack traces
- Request ID tracking for request correlation
- Slow request detection and warning (threshold: 1000ms)
- Recursion prevention (excludes logging endpoints from being logged)
- Asynchronous log submission to avoid blocking requests
Log Data Captured:
- Request method, path, status code, duration
- User agent, remote IP, content length, response size
- Request body (for small requests < 1KB)
- Error details and context
- Performance metrics and timing
Current Status:
- 393+ logs automatically captured and stored
- All API endpoints now have comprehensive logging
- Performance monitoring active
- Error tracking operational
Ready for Engine Sub-Services Integration
- Fixed LogDashboard component imports to use correct ~/components/lib/ui/ paths
- Replaced missing Alert component with custom styled div
- Added missing downServices variable calculation in system monitor
- System monitor page now loads correctly without errors
All UI components now properly imported and system monitor is functional
Phase 2 Complete: Centralized Logging Infrastructure
Backend:
- Created comprehensive logging API with log submission, retrieval, and stats
- Implemented log storage in Valkey with size limits (10,000 logs) and auto-rotation
- Added filtering by service, level, subcomponent, and search functionality
- Built pagination support for large log datasets
- Created log statistics endpoint with service and level breakdowns
Frontend:
- Built LogService with retry logic, polling, and error handling
- Created LogDashboard component with TanStack Table integration
- Added real-time log monitoring with auto-refresh
- Implemented advanced filtering (service, level, search)
- Added log statistics cards and pagination controls
- Integrated log dashboard into system monitor with tabs
Features:
- Standardized log format with metadata and context support
- Real-time updates every 10 seconds for logs, 30 seconds for stats
- Comprehensive error handling and retry mechanisms
- Responsive design with Shadcn/ui components
- Log level badges with appropriate colors and icons
- Timestamp formatting with relative time display
System monitor now has two tabs:
1. Service Status - Health monitoring (Phase 1)
2. System Logs - Centralized logging dashboard (Phase 2)
Ready for Phase 3: Service Integration
- Fixed UI health check to use container name 'sirius-ui:3000' instead of localhost
- Fixed RabbitMQ health check to use container name 'sirius-rabbitmq:5672' instead of localhost
- Fixed Engine health check to use gRPC port 50051 instead of non-existent port 5174
- All services now correctly show as UP when actually running
- Overall system status now shows 'healthy' instead of 'degraded'
Health monitoring now accurately reflects actual service status
- Fixed 503 status handling in healthCheckService (degraded is valid)
- Fixed service counting logic to show correct total (6 services)
- Updated totalServices calculation to use actual service count
- Health data should now display properly in system monitor
Frontend should now show real service status instead of loading state
- Updated go-api to v0.0.6 to include missing functions
- Fixed go.mod to use correct version with all required functions
- API now successfully builds and runs with comprehensive health checks
- System health endpoint working with real service status data
- Resolved development vs production container configuration issues
Phase 1 complete: Real-time service health monitoring operational
- Enhanced API health handler with comprehensive system health checks
- Added /api/v1/system/health endpoint for monitoring all services
- Implemented health checks for UI, API, Engine, PostgreSQL, Valkey, RabbitMQ
- Created healthCheckService.ts for frontend health check management
- Updated system monitor page to use real health check data
- Added real-time polling with error handling and retry logic
- Fixed go.mod to point to correct go-api path
- Added error display and loading states for better UX
Phase 1 complete: Real-time service health monitoring with backend integration
- Add project plan in documentation/dev-notes/system-monitoring-plan.md
- Add detailed task breakdown in tasks/system-monitoring.json
- Create system monitor page with mock data and basic UI
- Add System Monitor link to Header Settings dropdown
- Implement ServiceStatusCard and LogViewer components
- Include system overview metrics and real-time refresh functionality
Phase 0 complete: Basic UI structure with mock data ready for backend integration
- Add README.new-project.md with standardized development workflow
- Add README.tasks.md with task management system guidelines
- Add README.architecture-quick-reference.md for LLM context
- Add task-management.mdc cursor rule for automatic context inclusion
- Update documentation index with new project management section
- Update main cursor rules to include task management context
This establishes consistent project structure, task tracking, and Git workflows
for all future development sprints.
- Establish operator-first testing philosophy for Sirius
- Define integration-focused testing approach over traditional unit testing
- Document current testing infrastructure (container and documentation testing)
- Outline future testing objectives for next 3 months
- Provide testing standards and best practices specific to Sirius
- Emphasize standalone, real-world validation testing
- Include testing tools, commands, and troubleshooting guidance
- Update documentation index to include new testing philosophy guide
- Follow Sirius documentation standards with complete YAML front matter
- Add consistent spacing and line breaks for better readability
- Improve table formatting for testing strategy section
- Enhance code block formatting and structure
- Add visual separation between sections for better navigation
- Maintain consistent formatting throughout the document
- Create comprehensive CI/CD documentation (README.cicd.md)
- Modernize CI workflow to align with current testing system
- Remove legacy submodule handling and complex repository_dispatch logic
- Optimize pre-commit hooks to only run quick validation (~30s vs ~5-10min)
- Add CI/CD integration section to developer guide
- Update documentation index to include new CI documentation
- Maintain testing infrastructure for developers while moving heavy testing to CI
- Implement staged testing approach: quick checks (pre-commit) + full testing (CI)
- Align CI with new Docker Compose structure and environment switching
- Add README.developer-guide.md with complete development workflows
- Add QUICK-REFERENCE.md for common commands and troubleshooting
- Update README.docker-architecture.md with new environment switching
- Update README.development.md to reference new developer guide
- Update documentation index to include all files
- Document new image tagging strategy and environment switching
- Provide comprehensive troubleshooting and best practices
- Include quick reference for common development tasks
- Add environment-specific image tags (dev, prod) to prevent cache conflicts
- Update docker-compose.dev.yaml to use sirius-sirius-ui:dev tag
- Update docker-compose.prod.yaml to use sirius-sirius-ui:prod tag
- Create scripts/switch-env.sh for convenient environment switching
- Script handles cleanup, rebuild, and proper environment startup
- Resolves issue where cached images prevented dev/prod mode switching
- Now supports seamless switching between development and production modes
- Add detailed section on Docker layer caching problems that cause dev images in prod
- Include symptoms, root cause analysis, and step-by-step solutions
- Add verification steps to ensure correct build targets are used
- Include prevention strategies and long-term best practices
- Add environment switching best practices to prevent cache issues
- Document the specific React hook error case we encountered
- Provide immediate fix commands and verification steps
- Update getBaseUrl() function to use SIRIUS_API_URL environment variable
- Fixes tRPC client connection during server-side rendering in Docker
- Resolves 'Cannot read properties of null (reading useState)' error
- UI now renders properly in both development and production modes
- Create comprehensive documentation system with templates and standards
- Add Docker architecture documentation with complete setup guide
- Implement documentation testing and validation system
- Simplify git operations workflow for practical development
- Reorganize documentation structure with proper categorization
- Add container testing system with Makefile integration
- Archive old documentation and consolidate new structure
- Update cursor rules for documentation and testing workflows
- Fix Makefile paths for documentation linting scripts
- Add Docker architecture doc to documentation index
- Remove unused 'log' import from app-terminal and app-agent main.go files
- Create missing command binaries for app-agent (agent, command-receiver, command-sender)
- Fixes Docker build failures that were preventing container startup
- All app services now have proper Go module structure and build successfully
This resolves the container restart loop issue in production deployment.
0sm0s1z