Compare commits

...

19 Commits

Author SHA1 Message Date
Xiaoxia 9e0f8ae611 refactor: centralize board build configuration 2026-07-27 16:21:07 +08:00
Xiaoxia dd7e931c80 fix: resolve build regressions after board moves 2026-07-27 08:14:50 +08:00
Xiaoxia f475b4cbb1 fix: defer board validation during CMake early expansion 2026-07-27 07:24:54 +08:00
Xiaoxia c78d7f2568 refactor: replace release script with build tool 2026-07-27 07:24:51 +08:00
Xiaoxia 177901c974 fix: enable cross-signed certificate verification globally 2026-07-27 07:04:48 +08:00
Xiaoxia 29bf765947 refactor: organize board metadata and directories 2026-07-27 07:04:48 +08:00
Xiaoxia 90311fc8fb refactor: remove unreliable acoustic provisioning (#2152)
Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
2026-07-27 02:14:53 +08:00
Xiaoxia 3e86f0adaf feat: add ESP32-S31 Function CoreBoard support (#2151)
* feat: add ESP32-S31 Function CoreBoard support

* ci: build ESP32-S31 with IDF 6.1

---------

Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
2026-07-27 00:46:15 +08:00
javen13 5b55606415 feat: add Waveshare ESP32-S3-Touch-LCD-3.49 V2 (#2143) 2026-07-26 21:17:11 +08:00
Xiaoxia 38bc4f443b Update ESP-SR/P4 support and fix LCDWiki IDF 6 build (#2142)
* feat: update esp-sr and enable ESP32-P4 rev1 builds

* fix: build LCDWiki ES3C35P with ESP-IDF 6

---------

Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
2026-07-25 00:35:14 +08:00
sinco-z 324c8098a9 fix(taiji-pi-s3): keep round-screen text in safe area (#2136)
Co-authored-by: sinco-z <sinco-z@users.noreply.github.com>
2026-07-23 02:29:53 +08:00
Eivs ac3212d781 Add support for LCD Wiki ES3C35P board configuration and files (#2112)
* Add support for LCD Wiki ES3C35P (3.5" ESP32-S3) board

* fix: update variant counts in test for new ES3C35P board

---------

Co-authored-by: Eivs <eivs@Eivss-MacBook-Pro.local>
2026-07-22 20:46:57 +08:00
Xiaoxia 5f6c09b893 feat: enhance BoxAudioCodec with input gain and reference channel support
- Updated BoxAudioCodec constructor to include parameters for input gain, reference gain channel, and reference gain.
- Modified EnableInput method to set gain for the reference channel if applicable.
- Adjusted CustomAudioCodec initialization to utilize new gain parameters for improved audio processing.
2026-07-19 22:45:03 +08:00
Xiaoxia 5540258abc feat: integrate press-to-talk functionality in lichuang_dev_board
- Added PressToTalkMcpTool to manage press-to-talk and click-to-talk modes.
- Updated button event handling to support listening state transitions based on press-to-talk status.
- Initialized press-to-talk tool during board setup for enhanced user interaction.
2026-07-19 02:43:04 +08:00
Xiaoxia cb06516bea Refine GitHub issue forms (#2128)
Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
2026-07-19 02:10:46 +08:00
Xiaoxia 3dea6e7f84 Migrate to xiaozhi-fonts 2.0.0 (#2125)
* feat: migrate firmware to xiaozhi-fonts 2.0.0

* fix: restart scrolling subtitles on text change

* docs: add AGENTS.md for project overview, architecture, rules, and commands

- Introduced AGENTS.md to document the XiaoZhi voice-assistant firmware architecture, required rules for development, board configuration process, and validation steps.
- Included detailed sections on project structure, commands for building and testing, and authoritative documentation references.

* feat: finalize glyph push protocol

* fix: remove stale emoji font include

* build: use xiaozhi-fonts 2.0.0

* fix: decouple custom fonts from glyph push

* refactor: improve wake word invocation handling

- Introduced BeginWakeWordInvoke method to streamline the wake word processing flow.
- Updated HandleWakeWordDetectedEvent to utilize the new method, ensuring proper state transitions and scheduling.
- Enhanced error handling to prevent the device from getting stuck in the connecting state.
- Refactored audio channel management to improve responsiveness during wake word detection.

---------

Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
2026-07-18 21:29:00 +08:00
Xiaoxia 0f6c435f66 Complete ESP-IDF 6 migration hardening (#2123)
* Complete ESP-IDF 6 migration hardening

* Fix PR variant diff selection

* Allow Git diff in CI container

---------

Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
2026-07-17 16:32:44 +08:00
Xiaoxia 3e78cd73bd v2.4.0: Migrate to IDF 6.0 (#2120)
* Migrate board builds to ESP-IDF 6.0.1

* Document upstream IDF 6 CI blockers

* Use IDF 6 patched registry components

* Fix board-specific ESP-IDF 6 build errors

* Fix remaining ESP-IDF 6 board builds

* Document final ESP-IDF 6 matrix results [skip ci]

* update IDF 6 component releases

* rebase IDF 6 migration and use upstream SSCMA

* Enhance audio input management in AudioService

- Introduced AS_EVENT_AUDIO_INPUT_STOP_REQUEST to manage audio input stopping more effectively.
- Updated AudioService::Start() to clear the new stop request event.
- Modified AudioService::AudioInputTask() to handle the stop request and ensure proper input disabling.
- Adjusted AdcPdmAudioCodec::Start() to reflect lazy input opening, ensuring start/stop operations run in the same task.

* Refactor audio codec management and configuration

- Added output_device_opened_ flag to track the state of the output device in Es8388 and Es8389 codecs.
- Updated EnableOutput method to prevent unnecessary device reopening and added mute functionality to manage audio output more effectively.
- Enhanced error handling in Es8311AudioCodec by ensuring proper closure and deletion of the codec device.
- Adjusted sample rates in board configurations for atk-dnesp32s3-box2 to 24000 Hz and introduced AUDIO_CODEC_USE_MCLK for improved clock management.

* Update build configurations and documentation for ESP-IDF 6 compatibility

- Added container specification for the build workflow using espressif/idf:v6.0.1.
- Updated the version of the espressif/esp_video component to ^2.3.0.
- Enhanced documentation to clarify the support status of ESP32-P4 variants, specifying that legacy Rev < 3 variants are excluded from the IDF 6 matrix.
- Adjusted board configurations to include IDF version constraints and necessary SDK configurations for legacy support.
- Improved handling of YUV formats in EspVideo to maintain compatibility with previous versions.

* Implement playback drained event handling in Application

- Added MAIN_EVENT_PLAYBACK_DRAINED to manage playback state transitions.
- Introduced callbacks for playback drained events in AudioService to trigger listening state changes.
- Refactored Application::Run() to handle deferred listening start based on playback queue status.
- Enhanced audio processing logic to prevent audio truncation during state changes.
- Updated related methods to ensure proper wake word detection configuration during listening mode.

* Fix variant selection shell in CI

* Update project version to 2.4.0 and adjust component dependencies

- Bump project version in CMakeLists.txt to 2.4.0.
- Change espressif/esp_video component version to ^2.0.1 in idf_component.yml.
- Modify AUDIO_INPUT_REFERENCE setting in config.h to false for m5stack-core-s3.
- Remove unnecessary infinite loops in xmini_c3_board.cc during initialization.

---------

Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
2026-07-17 13:41:52 +08:00
小鹏 2b8e1f34b0 feat(otto): update GIF emoji component to v1.3.0 (#2119)
- Upgrade the Otto GIF emoji component to v1.3.0
- Remove legacy Otto GIF emotion alias mappings
- Use the 21 standard Xiaozhi emotion names directly
2026-07-17 12:53:50 +08:00
598 changed files with 9554 additions and 9541 deletions
@@ -0,0 +1,77 @@
name: Build, setup, or flashing problem
description: Report a reproducible problem setting up, building, or flashing the firmware
title: "[Build] "
labels: ['bug']
body:
- type: markdown
attributes:
value: |
This form is for the open-source firmware. For device unbinding, email [xiaozhi.ai@tenclass.com](mailto:xiaozhi.ai@tenclass.com?subject=Device%20unbinding%20request) privately with the device ID and MAC address. Use the [XiaoZhi AI website](https://xiaozhi.me/) for other account and cloud-service requests.
- type: checkboxes
id: checklist
attributes:
label: Before submitting
options:
- label: I have tested a current source version and searched for an existing issue reporting the same problem.
required: true
- type: input
id: board
attributes:
label: Board or device
description: Enter the physical product/model and the selected board target or build variant.
placeholder: ex. Waveshare ESP32-S3-Touch-LCD-3.5B; target waveshare/esp32-s3-touch-lcd-3.5b
validations:
required: true
- type: input
id: source_version
attributes:
label: XiaoZhi source version
description: Paste the complete output of `git describe --tags --always --dirty`. Do not enter "latest".
placeholder: ex. v2.2.6-15-g1a2b3c4
validations:
required: true
- type: input
id: esp_idf_version
attributes:
label: ESP-IDF version
description: Paste the complete output of `idf.py --version`.
placeholder: ex. ESP-IDF v6.0.2
validations:
required: true
- type: textarea
id: problem
attributes:
label: Problem and reproduction steps
description: State what you tried, the exact commands or steps, what you expected, what happened, and any relevant local changes.
placeholder: |
Operating system:
Command or steps:
Expected result:
Actual result:
Local changes, if any:
validations:
required: true
- type: textarea
id: logs
attributes:
label: Complete error output
description: Paste the output from the first error through the final failure. Remove credentials, tokens, usernames, and other private information.
placeholder: Paste the complete output here, not only the final error line.
render: shell
validations:
required: true
- type: upload
id: attachments
attributes:
label: Optional files
description: Attach logs or a ZIP containing relevant configuration files. Review files for secrets first.
validations:
required: false
accept: ".log,.txt,.json,.zip"
@@ -1,103 +0,0 @@
name: Installation or build bug report
description: Report installation or build bugs
labels: ['bug']
body:
- type: checkboxes
id: checklist
attributes:
label: Answers checklist.
description: Before submitting a new issue, please follow the checklist and try to find the answer.
options:
- label: I have read the documentation [XiaoZhi AI Programming Guide](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb) and the issue is not addressed there.
required: true
- label: I have updated my branch (master or release) to the latest version and checked that the issue is present there.
required: true
- label: I have searched the issue tracker for a similar issue and not found a similar issue.
required: true
- type: input
id: xiaozhi_ai_version
attributes:
label: XiaoZhi AI version.
description: On which XiaoZhi AI version does this issue occur on? Run `git describe --tags` to find it.
placeholder: ex. v1.1.0-44-g140aab8
validations:
required: true
- type: dropdown
id: operating_system
attributes:
label: Operating System used.
multiple: false
options:
- Windows
- Linux
- macOS
validations:
required: true
- type: dropdown
id: build
attributes:
label: How did you build your project?
multiple: false
options:
- Command line with CMake
- Command line with idf.py
- CLion IDE
- VS Code IDE/Cursor
- Other (please specify in More Information)
validations:
required: true
- type: dropdown
id: windows_comand_line
attributes:
label: If you are using Windows, please specify command line type.
multiple: false
options:
- PowerShell
- CMD
validations:
required: false
- type: textarea
id: expected
attributes:
label: What is the expected behavior?
description: Please provide a clear and concise description of the expected behavior.
placeholder: I expected it to...
validations:
required: true
- type: textarea
id: actual
attributes:
label: What is the actual behavior?
description: Please describe actual behavior.
placeholder: Instead it...
validations:
required: true
- type: textarea
id: steps
attributes:
label: Steps to reproduce.
description: 'How do you trigger this bug? Please walk us through it step by step. If this is build bug, please attach sdkconfig file (from your project folder). Please attach your code here.'
value: |
1. Step
2. Step
3. Step
...
validations:
required: true
- type: textarea
id: debug_logs
attributes:
label: Build or installation Logs.
description: Build or installation log goes here, should contain the backtrace, as well as the reset source if it is a crash.
placeholder: Your log goes here.
render: plain
validations:
required: false
- type: textarea
id: more-info
attributes:
label: More Information.
description: Do you have any other information from investigating this?
placeholder: ex. Any more.
validations:
required: false
+57 -95
View File
@@ -1,115 +1,77 @@
name: Runtime bug report
description: Report runtime bugs
name: Device runtime bug
description: Report a reproducible firmware problem while a device is running
title: "[Bug] "
labels: ['bug']
body:
- type: markdown
attributes:
value: |
This form is for the open-source device firmware. For device unbinding, email [xiaozhi.ai@tenclass.com](mailto:xiaozhi.ai@tenclass.com?subject=Device%20unbinding%20request) privately with the device ID and MAC address. Use the [XiaoZhi AI website](https://xiaozhi.me/) for other account and cloud-service requests.
- type: checkboxes
id: checklist
attributes:
label: Answers checklist.
description: Before submitting a new issue, please follow the checklist and try to find the answer.
label: Before submitting
options:
- label: I have read the documentation [XiaoZhi AI Programming Guide](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb) and the issue is not addressed there.
required: true
- label: I have updated my firmware to the latest version and checked that the issue is present there.
required: true
- label: I have searched the issue tracker for a similar issue and not found a similar issue.
- label: I have tested the latest applicable firmware and searched for an existing issue reporting the same problem.
required: true
- type: input
id: xiaozhi_ai_firmware_version
id: device
attributes:
label: XiaoZhi AI firmware version.
description: On which firmware version does this issue occur on?
placeholder: ex. v1.2.1_bread-compact-wifi
label: Device or development board
description: Enter the manufacturer, exact product/model, and hardware revision or SKU when known.
placeholder: ex. LilyGO T-Circle-S3 V1.1 or EchoEar N16R8
validations:
required: true
- type: dropdown
id: operating_system
- type: input
id: firmware_version
attributes:
label: Operating System used.
multiple: false
options:
- Windows
- Linux
- macOS
label: Firmware version
description: Paste the `App version:` value from the startup log. Do not enter a board name or "latest".
placeholder: ex. 2.2.6
validations:
required: true
- type: dropdown
id: build
- type: input
id: firmware_source
attributes:
label: How did you build your project?
multiple: false
options:
- Command line with CMake
- Command line with idf.py
- CLion IDE
- VS Code IDE/Cursor
- Other (please specify in More Information)
label: Firmware source or filename
description: State where the firmware came from and include its filename or board target when available.
placeholder: ex. xiaozhi.me, GitHub release echoear.bin, self-built target df-k10, or vendor firmware
validations:
required: true
- type: dropdown
id: windows_comand_line
- type: textarea
id: problem
attributes:
label: If you are using Windows, please specify command line type.
multiple: false
options:
- PowerShell
- CMD
label: Problem and reproduction steps
description: Describe the starting state, exact actions, expected result, actual result, frequency, and any relevant modifications.
placeholder: |
Steps:
Expected result:
Actual result:
Frequency:
Modifications, if any:
validations:
required: true
- type: textarea
id: evidence
attributes:
label: Logs or other evidence
description: Paste relevant startup and failure logs, including a backtrace or reset reason when applicable. If logs are unavailable, briefly say why. Remove credentials, device IDs, MAC addresses, UUIDs, and other private information.
placeholder: Paste relevant logs here, or explain why logs cannot be collected.
render: shell
validations:
required: true
- type: upload
id: attachments
attributes:
label: Optional photos, videos, or log files
description: Attach files that help identify the hardware or demonstrate the failure. Review files for private information first.
validations:
required: false
- type: dropdown
id: power_supply
attributes:
label: Power Supply used.
multiple: false
options:
- USB
- External 5V
- External 3.3V
- Battery
validations:
required: true
- type: textarea
id: expected
attributes:
label: What is the expected behavior?
description: Please provide a clear and concise description of the expected behavior.
placeholder: I expected it to...
validations:
required: true
- type: textarea
id: actual
attributes:
label: What is the actual behavior?
description: Please describe actual behavior.
placeholder: Instead it...
validations:
required: true
- type: textarea
id: steps
attributes:
label: Steps to reproduce.
description: 'How do you trigger this bug? Please walk us through it step by step. Please attach your code here.'
value: |
1. Step
2. Step
3. Step
...
validations:
required: true
- type: textarea
id: debug_logs
attributes:
label: Debug Logs.
description: Debug log goes here, should contain the backtrace, as well as the reset source if it is a crash.
placeholder: Your log goes here.
render: plain
validations:
required: false
- type: textarea
id: more-info
attributes:
label: More Information.
description: Do you have any other information from investigating this?
placeholder: ex. Any more.
validations:
required: false
accept: ".png,.jpg,.jpeg,.gif,.mp4,.mov,.webm,.log,.txt,.zip"
+37 -21
View File
@@ -1,34 +1,50 @@
name: Feature request
description: Suggest an idea for this project.
description: Propose an improvement to a supported device or the common firmware
title: "[Feature] "
labels: ['enhancement']
body:
- type: markdown
attributes:
value: |
* We welcome any ideas or feature requests! Its helpful if you can explain exactly why the feature would be useful.
* There are usually some outstanding feature requests in the [existing issues list](https://github.com/78/xiaozhi-esp32/labels/enhancement), feel free to add comments to them.
* If you would like to contribute, please read the [contributions guide](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb).
- type: textarea
id: problem-related
Use the **Board support request** form for a new board, hardware revision, or product variant. Account and cloud-product requests belong on the [XiaoZhi AI website](https://xiaozhi.me/).
- type: checkboxes
id: checklist
attributes:
label: Is your feature request related to a problem?
description: Please provide a clear and concise description of what the problem is.
placeholder: ex. I'm always frustrated when ...
label: Before submitting
options:
- label: I have searched the existing feature requests and did not find the same proposal.
required: true
- type: input
id: scope
attributes:
label: Affected device or area
description: Enter the device, board target, or firmware area. Enter "All devices" for a general request.
placeholder: ex. df-k10, audio, display, protocol, or All devices
validations:
required: true
- type: textarea
id: problem
attributes:
label: Problem or use case
description: Explain the current limitation and why the requested change would be useful.
validations:
required: true
- type: textarea
id: solution
attributes:
label: Describe the solution you'd like.
description: Please provide a clear and concise description of what you want to happen.
placeholder: ex. When using XiaoZhi ...
- type: textarea
id: alternatives
attributes:
label: Describe alternatives you've considered.
description: Please provide a clear and concise description of any alternative solutions or features you've considered.
placeholder: ex. Choosing other approach wouldn't work, because ...
label: Requested behavior
description: Describe the result or behavior you would like to see.
validations:
required: true
- type: textarea
id: context
attributes:
label: Additional context.
description: Please add any other context or screenshots about the feature request here.
placeholder: ex. This would work only when ...
label: Additional context
description: Optionally add alternatives, screenshots, diagrams, links, or testing information.
validations:
required: false
@@ -0,0 +1,67 @@
name: Board support request
description: Request support for a new board, product, hardware revision, or SKU
title: "[Board Support] "
labels: ['enhancement']
body:
- type: markdown
attributes:
value: |
Similar-looking products may use different pins and components. Please identify the exact hardware rather than only naming the closest supported board.
- type: checkboxes
id: checklist
attributes:
label: Before submitting
options:
- label: I have searched the repository and existing issues for this exact model and hardware revision.
required: true
- type: textarea
id: hardware
attributes:
label: Exact hardware
description: Provide the manufacturer, model, revision/SKU, main chip and memory, and an official product or hardware documentation link.
placeholder: |
Manufacturer and model:
Revision or SKU:
Chip, flash, and PSRAM:
Official link:
validations:
required: true
- type: textarea
id: request
attributes:
label: Requested support
description: Describe what should work and any known differences from the closest supported board.
validations:
required: true
- type: textarea
id: attempted
attributes:
label: What have you tried?
description: Include the closest board target or firmware, its version, and the result. Enter "None" if no existing target is appropriate.
validations:
required: true
- type: dropdown
id: testing
attributes:
label: Testing available
options:
- I can build, flash, collect logs, and test changes
- I can flash binaries and test changes
- I can provide hardware information only
- I cannot test changes
validations:
required: true
- type: upload
id: attachments
attributes:
label: Optional hardware files, photos, or logs
description: Attach public schematics, board photos, pinouts, or serial logs. Do not upload confidential material or secrets.
validations:
required: false
accept: ".png,.jpg,.jpeg,.pdf,.log,.txt,.json,.zip"
@@ -0,0 +1,42 @@
name: Firmware development question
description: Ask a focused technical question about this repository or supported hardware
title: "[Question] "
labels: ['question']
body:
- type: markdown
attributes:
value: |
This form is for technical questions about the open-source firmware. For device unbinding, email [xiaozhi.ai@tenclass.com](mailto:xiaozhi.ai@tenclass.com?subject=Device%20unbinding%20request) privately with the device ID and MAC address. Other account and cloud-service requests belong on the [XiaoZhi AI website](https://xiaozhi.me/).
- type: checkboxes
id: checklist
attributes:
label: Before submitting
options:
- label: I have searched the repository and existing issues and did not find an answer.
required: true
- type: input
id: context
attributes:
label: Relevant device, version, or code area
description: Enter the device/model, board target, source or firmware version, file, or subsystem that matters to the question.
placeholder: ex. zhengchen-cam on v2.2.6, or main/audio on current main
validations:
required: true
- type: textarea
id: question
attributes:
label: Goal and question
description: Explain what you are trying to accomplish and state the specific technical question.
validations:
required: true
- type: textarea
id: attempted
attributes:
label: What have you tried?
description: Optionally include relevant files, commands, experiments, logs, screenshots, or a minimal example.
validations:
required: false
+6 -6
View File
@@ -1,8 +1,8 @@
blank_issues_enabled: true
blank_issues_enabled: false
contact_links:
- name: 小智 AI 官方网站
- name: Unbind a previously owned device
url: https://github.com/78/xiaozhi-esp32/blob/main/.github/SUPPORT.md#unbind-a-previously-owned-device
about: If the device is bound to another account, email its device ID and MAC address privately. Do not post these identifiers in a public issue.
- name: Other cloud services and account requests
url: https://xiaozhi.me/
about: 激活设备、配置 AI、声纹识别、声音克隆等应有尽有,DIY 属于你自己的小智
- name: 小智 AI 聊天机器人百科全书
url: https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb
about: 开发文档、硬件制作、烧录教程、FAQ尽在小智百科
about: Use the XiaoZhi AI website for activation, agent configuration, voiceprint, voice cloning, and other cloud services
+49
View File
@@ -0,0 +1,49 @@
# Account and Device Support
This GitHub repository tracks the open-source XiaoZhi device firmware. Account recovery, device ownership, and cloud-service requests are handled outside the public issue tracker.
## Unbind a Previously Owned Device
If you purchased or received a used device that is still bound to another person's account:
1. Collect the device ID and MAC address using the instructions below.
2. Email [xiaozhi.ai@tenclass.com](mailto:xiaozhi.ai@tenclass.com?subject=Device%20unbinding%20request) with both identifiers.
3. Use an email subject such as `Device unbinding / 解绑设备 - Device ID XXX - MAC address XXX`.
For multiple devices, attach a list containing the device ID and MAC address for each device.
> **Privacy:** Send device IDs and MAC addresses only by email. Do not post them in a public GitHub issue, discussion, screenshot, or log.
### Why Both Identifiers Are Required
- The **MAC address** uniquely identifies the physical device hardware.
- The **device ID** is obtained by asking the AI running on the device. Providing the value reported by the device helps demonstrate that the requester has the device in hand and can operate it.
Together, these identifiers help the support team locate the correct binding record and avoid unbinding the wrong device.
### Find the Device ID
If the device can connect and have a conversation, ask it:
> What is my device ID?
Record the complete value reported by the device. Both the device ID and MAC address are required for the standard unbinding process. If the device cannot have a conversation or does not return a device ID, explain this in the email; the support team may require other proof of possession. Do not guess or substitute a temporary activation code.
### Find the MAC Address
1. Connect the powered-on device to a computer with a data-capable USB cable.
2. Open the device's serial port with a serial terminal. Developers with an ESP-IDF environment can use `idf.py monitor`.
3. Keep the serial terminal open and restart the device so that the complete startup log is captured.
4. Search the log for a line similar to:
```text
wifi:mode : sta (aa:bb:cc:dd:ee:ff)
```
5. The value inside the parentheses is the Wi-Fi MAC address to include in the email.
If that line does not appear, search the startup log for `MAC` or `mac_address`. If you still cannot identify it, attach the exact device model and explain the situation in the private email. Do not upload the unredacted startup log to a public GitHub issue because it may contain the MAC address, UUID, Wi-Fi name, and other device information.
## Other Account and Cloud-Service Requests
For verification codes, password recovery, activation, agent configuration, voiceprint, voice cloning, and other cloud services, use the [XiaoZhi AI website](https://xiaozhi.me/).
+63 -37
View File
@@ -16,6 +16,8 @@ jobs:
prepare:
name: Determine variants to build
runs-on: ubuntu-latest
container:
image: espressif/idf:v6.0.1
outputs:
variants: ${{ steps.select.outputs.variants }}
steps:
@@ -24,16 +26,17 @@ jobs:
with:
fetch-depth: 0
- name: Install jq
run: sudo apt-get update && sudo apt-get install -y jq
- name: Test build tooling
run: python -m unittest discover -s scripts/tests -v
- id: list
name: Get all variant list
run: |
echo "all_variants=$(python scripts/release.py --list-boards --json)" >> $GITHUB_OUTPUT
echo "all_variants=$(python scripts/build.py --list-boards --json)" >> $GITHUB_OUTPUT
- id: select
name: Select variants based on changes
shell: bash
env:
ALL_VARIANTS: ${{ steps.list.outputs.all_variants }}
run: |
@@ -46,41 +49,20 @@ jobs:
fi
# pull_request 场景
BASE_SHA="${{ github.event.pull_request.base.sha }}"
HEAD_SHA="${{ github.event.pull_request.head.sha }}"
echo "Base: $BASE_SHA, Head: $HEAD_SHA"
# actions/checkout checks out GitHub's synthetic PR merge commit. Its
# first and second parents are the exact base and head commits used
# for this run, even when the event SHAs are not exposed as refs in
# the job container.
git config --global --add safe.directory "$GITHUB_WORKSPACE"
BASE_SHA="HEAD^1"
HEAD_SHA="HEAD^2"
echo "Base: $(git rev-parse "$BASE_SHA"), Head: $(git rev-parse "$HEAD_SHA")"
CHANGED=$(git diff --name-only $BASE_SHA $HEAD_SHA || true)
CHANGED=$(git diff --name-only "$BASE_SHA" "$HEAD_SHA")
echo -e "Changed files:\n$CHANGED"
NEED_ALL=0
declare -A AFFECTED
while IFS= read -r file; do
if [[ "$file" == main/* && "$file" != main/boards/* ]]; then
NEED_ALL=1
fi
if [[ "$file" == main/boards/common/* ]]; then
NEED_ALL=1
fi
if [[ "$file" == main/boards/* ]]; then
board=$(echo "$file" | cut -d '/' -f3)
AFFECTED[$board]=1
fi
done <<< "$CHANGED"
if [[ "$NEED_ALL" -eq 1 ]]; then
echo "variants=$ALL_VARIANTS" >> $GITHUB_OUTPUT
else
if [[ ${#AFFECTED[@]} -eq 0 ]]; then
echo "variants=[]" >> $GITHUB_OUTPUT
else
BOARDS_JSON=$(printf '%s\n' "${!AFFECTED[@]}" | sort -u | jq -R -s -c 'split("\n")[:-1]')
FILTERED=$(echo "$ALL_VARIANTS" | jq -c --argjson boards "$BOARDS_JSON" 'map(select(.board as $b | $boards | index($b)))')
echo "variants=$FILTERED" >> $GITHUB_OUTPUT
fi
fi
SELECTED=$(printf '%s\n' "$CHANGED" | python scripts/build.py --select-changed)
echo "variants=$SELECTED" >> $GITHUB_OUTPUT
build:
name: Build ${{ matrix.full_name }}
@@ -92,7 +74,7 @@ jobs:
include: ${{ fromJson(needs.prepare.outputs.variants) }}
runs-on: ubuntu-latest
container:
image: espressif/idf:v5.5.2
image: espressif/idf:v6.0.1
steps:
- name: Checkout
uses: actions/checkout@v4
@@ -101,7 +83,7 @@ jobs:
shell: bash
run: |
source $IDF_PATH/export.sh
python scripts/release.py ${{ matrix.board }} --name ${{ matrix.name }}
python scripts/build.py ${{ matrix.board }} --name ${{ matrix.name }}
- name: Upload artifacts
uses: actions/upload-artifact@v4
@@ -109,3 +91,47 @@ jobs:
name: xiaozhi_${{ matrix.full_name }}_${{ github.sha }}
path: build/merged-binary.bin
if-no-files-found: error
esp32s31-idf61:
name: Build ESP32-S31 with IDF 6.1
runs-on: ubuntu-latest
container:
image: espressif/idf:release-v6.1
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build ESP32-S31 preview variant
shell: bash
run: |
source $IDF_PATH/export.sh
python scripts/build.py espressif/esp32-s31-function-coreboard-1 \
--name esp32-s31-function-coreboard-1
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: xiaozhi_espressif-esp32-s31-function-coreboard-1_${{ github.sha }}
path: build/merged-binary.bin
if-no-files-found: error
blufi-idf6:
name: IDF 6 BluFi option
runs-on: ubuntu-latest
container:
image: espressif/idf:v6.0.1
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build BluFi-enabled firmware
shell: bash
run: |
source $IDF_PATH/export.sh
idf.py -B build-blufi \
-DSDKCONFIG=build-blufi/sdkconfig \
-DSDKCONFIG_DEFAULTS="sdkconfig.defaults;scripts/ci/blufi.sdkconfig.defaults" \
-DIDF_TARGET=esp32s3 \
-DBOARD_NAME=bread-compact-wifi \
-DBOARD_TYPE=bread-compact-wifi \
build
+94
View File
@@ -0,0 +1,94 @@
# AGENTS.md
## Project
XiaoZhi is an ESP-IDF C/C++ voice-assistant firmware supporting many chips, boards, displays, audio devices, and network transports. A build selects exactly one board implementation.
Use ESP-IDF v6.0.2 when possible. IDF 5.5.x is retained only for documented legacy boards.
## Architecture
- `main/application.*`: main event loop, protocol lifecycle, and high-level behavior.
- `main/device_state_machine.*`: legal runtime state transitions.
- `main/boards/common/`: board interfaces and reusable hardware/network helpers.
- `main/boards/**/`: board-specific pins, initialization, and build variants.
- `main/audio/`: codecs, audio tasks, engines, wake words, and queues.
- `main/protocols/`: transport-neutral API plus WebSocket and MQTT/UDP.
- `main/display/` and `main/led/`: reusable UI implementations.
- `main/mcp_server.*`: common device-side MCP tools and dispatch.
- `main/Kconfig.projbuild`: board and feature configuration.
- `main/CMakeLists.txt`: source, board, locale, font, and asset selection.
- `scripts/build.py`: canonical board/variant build entry point.
Read the closest existing implementation before adding a new one. Prefer the narrowest owning layer; do not put board-specific behavior into core modules.
## Required Rules
- Preserve unrelated worktree changes and keep patches focused.
- A build must export exactly one board factory through `DECLARE_BOARD(...)`.
- Never alter an existing board's pins to support different hardware. Add a uniquely named board or release variant; board identity affects OTA compatibility.
- Core code depends on `Board` interfaces, never a concrete board class or board `config.h`.
- Treat camera, backlight, display, LED, battery, and similar capabilities as optional.
- Change runtime state through `Application::SetDeviceState()` and the state machine.
- Callbacks may run outside the main task. Schedule application mutations with `Application::Schedule()` or event bits.
- Do not block the main event loop or audio tasks. Avoid unbounded queues and repeated large allocations in audio paths.
- Keep shared message semantics in `Protocol`; verify both transports when changing its contract.
- Validate network input and preserve `cJSON` ownership. NVS keys are persistent API and require migration when changed.
- Guard target-specific features with Kconfig/component rules. Do not assume every target has PSRAM or S3/P4 resources.
- Do not manually edit generated/vendor output: `build/`, `releases/`, `managed_components/`, `components/`, `sdkconfig*`, `main/assets/lang_config.h`, or generated mmap headers.
- Format only touched C/C++ files with the repository `.clang-format`; avoid unrelated mass formatting.
## Boards and Configuration
Board selection is a coupled chain:
`config.json` -> `scripts/build.py` -> `main/Kconfig.projbuild` -> `main/CMakeLists.txt` -> board source and `config.h`.
When adding a board or variant, update every relevant link in that chain. Include a unique board identity, correct chip target, flash/partition settings, exactly one `DECLARE_BOARD`, and board documentation. Follow `docs/custom-board.md`.
## Commands
Source the intended ESP-IDF environment first:
```sh
source /path/to/esp-idf/export.sh
idf.py --version
```
```sh
# Discover exact board and variant names
python3 scripts/build.py --list-boards
# Canonical variant build
python3 scripts/build.py <board-directory> --name <variant-name>
# Host-side build tests
python3 -m unittest discover -s scripts/tests -v
# Format/check touched files
clang-format -i <files>
clang-format --dry-run -Werror <files>
```
The build script changes local `sdkconfig` and build state. Do not assume the build directory still represents a previous target.
## Validation
- Board-only change: build affected variants and smoke-test changed hardware.
- Core, common-board, audio, protocol, display, dependency, Kconfig, or CMake change: run host tests and build representative affected chip/network paths.
- Protocol changes: verify WebSocket and MQTT/UDP when shared behavior changes.
- Audio changes: verify capture, playback, wake/VAD, interruption, reconnect, and applicable AEC modes.
- UI/assets changes: verify applicable no-display/OLED/LVGL paths and partition size.
- Always report what was tested and what still needs physical hardware. A successful build is not hardware validation.
## Authoritative Documentation
- Overview and SDK policy: `README.md`
- SDK compatibility: `docs/esp-idf-6-migration.md`
- Board guide: `docs/custom-board.md`
- Audio design: `main/audio/README.md`
- Code style: `docs/code_style.md`
- Protocols: `docs/websocket.md`, `docs/mqtt-udp.md`, `docs/mcp-protocol.md`
- CI matrix: `.github/workflows/build.yml`
Keep detailed or fast-changing information in those files, not here. Add a nested `AGENTS.md` only when a subsystem needs specialized instructions.
+1 -1
View File
@@ -9,5 +9,5 @@ include($ENV{IDF_PATH}/tools/cmake/project.cmake)
# "Trim" the build. Include the minimal set of components, main, and anything it depends on.
idf_build_set_property(MINIMAL_BUILD ON)
set(PROJECT_VER "2.3.0")
set(PROJECT_VER "2.4.0")
project(xiaozhi)
+18 -18
View File
@@ -12,26 +12,25 @@ As a voice interaction entry, the XiaoZhi AI chatbot leverages the AI capabiliti
<img src="docs/mcp-based-graph.jpg" alt="Control everything via MCP" width="320">
## Version Notes
## Recent Updates
The current v2 version is incompatible with the v1 partition table, so it is not possible to upgrade from v1 to v2 via OTA. For partition table details, see [partitions/v2/README.md](partitions/v2/README.md).
All hardware running v1 can be upgraded to v2 by manually flashing the firmware.
The stable version of v1 is 1.9.2. You can switch to v1 by running `git checkout v1`. The v1 branch will be maintained until February 2026.
- The mainline now targets ESP-IDF v6.0 or later, with v6.0.2 as the preferred stable SDK. The previous 157-variant baseline was validated on ESP-IDF v6.0.1; the current matrix contains 171 variants, of which 170 support IDF 6.0.x and the ESP32-S31 variant requires IDF 6.1 or later.
- MQTT and BluFi cryptographic code has migrated to PSA Crypto. IDF 6 component splits and third-party dependency compatibility have also been addressed.
- Audio pipeline concurrency, MQTT/UDP packet validation, and release-matrix selection have been hardened.
- ESP-IDF v5.5 is retained only for documented legacy boards. ESP32-P4 Rev1 and Rev3 are both supported on IDF 6 with ESP-SR 2.4.7; see the [ESP-IDF 6.0 Migration Guide](docs/esp-idf-6-migration.md) for full compatibility and board-validation details.
### Features Implemented
- Wi-Fi / ML307 Cat.1 4G
- Offline voice wake-up [ESP-SR](https://github.com/espressif/esp-sr)
- Supports two communication protocols ([Websocket](docs/websocket.md) or MQTT+UDP)
- Uses OPUS audio codec
- Voice interaction based on streaming ASR + LLM + TTS architecture
- Wi-Fi, wired Ethernet, USB RNDIS, and ML307/EC801E or NT26 Cat.1 4G networking; supported boards can switch between Wi-Fi and 4G
- Offline voice wake-up with [ESP-SR](https://github.com/espressif/esp-sr), including customizable wake words
- Two communication transports: [WebSocket](docs/websocket.md) and [MQTT + UDP](docs/mqtt-udp.md)
- Opus audio streaming with conventional streaming ASR + LLM + TTS pipelines and Realtime end-to-end voice models; AEC-capable hardware supports realtime full-duplex interaction
- Speaker recognition, identifies the current speaker [3D Speaker](https://github.com/modelscope/3D-Speaker)
- OLED / LCD display, supports emoji display
- OLED / LCD displays with emoji and rich expression support, plus camera vision input on supported boards
- Battery display and power management
- Multi-language support (Chinese, English, Japanese)
- Supports ESP32-C3, ESP32-S3, ESP32-P4 chip platforms
- 38 interface languages, with localized voice prompts where available and English fallback
- ESP32, ESP32-C3, ESP32-C5, ESP32-C6, ESP32-S3, and ESP32-P4 chip platforms
- Wi-Fi provisioning through hotspot or BluFi
- Device-side MCP for device control (Speaker, LED, Servo, GPIO, etc.)
- Cloud-side MCP to extend large model capabilities (smart home control, PC desktop operation, knowledge search, email, etc.)
- Customizable wake words, fonts, emojis, and chat backgrounds with online web-based editing ([Custom Assets Generator](https://github.com/78/xiaozhi-assets-generator))
@@ -48,10 +47,10 @@ Breadboard demo:
![Breadboard Demo](docs/v1/wiring2.jpg)
### Supports 70+ Open Source Hardware (Partial List)
### Supports 138 Board Directories and 171 Release Variants (Partial List)
- <a href="https://oshwhub.com/li-chuang-kai-fa-ban/li-chuang-shi-zhan-pai-esp32-s3-kai-fa-ban" target="_blank" title="LiChuang ESP32-S3 Development Board">LiChuang ESP32-S3 Development Board</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="Espressif ESP32-S3-BOX3">Espressif ESP32-S3-BOX3</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="Espressif ESP32-S3-BOX-3">Espressif ESP32-S3-BOX-3</a>
- <a href="https://docs.m5stack.com/zh_CN/core/CoreS3" target="_blank" title="M5Stack CoreS3">M5Stack CoreS3</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">M5Stack AtomS3R + Echo Base</a>
- <a href="https://gf.bilibili.com/item/detail/1108782064" target="_blank" title="Magic Button 2.4">Magic Button 2.4</a>
@@ -115,12 +114,13 @@ The firmware connects to the official [xiaozhi.me](https://xiaozhi.me) server by
### Development Environment
- Cursor or VSCode
- Install ESP-IDF plugin, select SDK version 5.4 or above
- Install the ESP-IDF plugin. [ESP-IDF v6.0.2](https://github.com/espressif/esp-idf/releases/tag/v6.0.2) is preferred; use a stable v6.0 or later release. ESP-IDF v5.5.2 is retained only for legacy board compatibility
- Linux is better than Windows for faster compilation and fewer driver issues
- This project uses Google C++ code style, please ensure compliance when submitting code
### Developer Documentation
- [ESP-IDF 6.0 Migration Guide](docs/esp-idf-6-migration.md) - SDK compatibility, component changes, legacy hardware support, and board validation status
- [Custom Board Guide](docs/custom-board.md) - Learn how to create custom boards for XiaoZhi AI
- [MCP Protocol IoT Control Usage](docs/mcp-usage.md) - Learn how to control IoT devices via MCP protocol
- [MCP Protocol Interaction Flow](docs/mcp-protocol.md) - Device-side MCP protocol implementation
@@ -170,4 +170,4 @@ If you have any ideas or suggestions, please feel free to raise Issues or join o
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date" />
</picture>
</a>
</a>
+17 -17
View File
@@ -12,26 +12,25 @@
<img src="docs/mcp-based-graph.jpg" alt="MCPであらゆるものを制御" width="320">
## バージョンノート
## 最近の更新
現在のv2バージョンはv1パーティションテーブルと互換性がないため、v1からv2へOTAでアップグレードすることはできません。パーティションテーブルの詳細については、[partitions/v2/README.md](partitions/v2/README.md)をご参照ください
v1を実行しているすべてのハードウェアは、ファームウェアを手動で書き込むことでv2にアップグレードできます
v1の安定版は1.9.2です。`git checkout v1`でv1に切り替えることができます。v1ブランチは2026年2月まで継続的にメンテナンスされます。
- メインラインはESP-IDF v6.0以降へ移行し、推奨安定版はv6.0.2です。従来の157リリースバリアントはESP-IDF v6.0.1でビルド検証済みです。現在のマトリクスは171バリアントで、そのうち170はIDF 6.0.xに対応し、ESP32-S31バリアントにはIDF 6.1以降が必要です
- MQTTとBluFiの暗号処理をPSA Cryptoへ移行し、IDF 6のコンポーネント分割およびサードパーティ依存関係にも対応しました。
- オーディオパイプラインの並行処理、MQTT/UDPパケット検証、リリースマトリクス選択処理を強化しました
- ESP-IDF v5.5は、文書で明記された旧式ボード向けにのみ残しています。ESP-SR 2.4.7を使用すると、ESP32-P4 Rev1とRev3の両方がIDF 6に対応します。詳細な互換性とボード検証状況は、[ESP-IDF 6.0移行ガイド](docs/esp-idf-6-migration.md)を参照してください。
### 実装済み機能
- Wi-Fi / ML307 Cat.1 4G
- オフライン音声ウェイクアップ [ESP-SR](https://github.com/espressif/esp-sr)
- 2種類の通信プロトコルに対応([Websocket](docs/websocket.md) または MQTT+UDP
- OPUSオーディオコーデックを採用
- ストリーミングASR + LLM + TTSアーキテクチャに基づく音声インタラクション
- Wi-Fi、有線Ethernet、USB RNDIS、およびML307/EC801EまたはNT26 Cat.1 4Gに対応し、一部のボードではWi-Fiと4Gを切り替え可能
- [ESP-SR](https://github.com/espressif/esp-sr)によるオフライン音声ウェイクアップとカスタムウェイクワード
- 2種類の通信方式:[WebSocket](docs/websocket.md)と[MQTT + UDP](docs/mqtt-udp.md)
- Opusオーディオストリーミングにより、従来のストリーミングASR + LLM + TTS構成とRealtimeエンドツーエンド音声モデルの両方に対応。AEC対応ハードウェアではリアルタイム全二重対話が可能
- 話者認識、現在話している人を識別 [3D Speaker](https://github.com/modelscope/3D-Speaker)
- OLED / LCDディスプレイ、表情表示対応
- OLED / LCDディスプレイで絵文字や豊かな感情表現を表示し、一部のボードではカメラによる視覚入力にも対応
- バッテリー表示と電源管理
- 多言語対応(中国語、英語、日本語)
- ESP32-C3、ESP32-S3、ESP32-P4チッププラットフォーム対応
- 38言語の画面表示に対応し、音声プロンプトはローカライズ済みリソースを優先して、未収録時は英語へフォールバック
- ESP32、ESP32-C3、ESP32-C5、ESP32-C6、ESP32-S3、ESP32-P4チッププラットフォーム
- ホットスポットまたはBluFiによるWi-Fiプロビジョニング
- デバイス側MCPによるデバイス制御(音量・明るさ調整、アクション制御など)
- クラウド側MCPで大規模モデル能力を拡張(スマートホーム制御、PCデスクトップ操作、知識検索、メール送受信など)
- カスタマイズ可能なウェイクワード、フォント、絵文字、チャット背景、オンラインWeb編集に対応 ([カスタムアセットジェネレーター](https://github.com/78/xiaozhi-assets-generator))
@@ -48,10 +47,10 @@ Feishuドキュメントチュートリアルをご覧ください:
![ブレッドボードデモ](docs/v1/wiring2.jpg)
### 70種類以上のオープンソースハードウェアに対応(一部のみ表示)
### 138のボードディレクトリと171のリリースバリアントに対応(一部のみ表示)
- <a href="https://oshwhub.com/li-chuang-kai-fa-ban/li-chuang-shi-zhan-pai-esp32-s3-kai-fa-ban" target="_blank" title="立創・実戦派 ESP32-S3 開発ボード">立創・実戦派 ESP32-S3 開発ボード</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="楽鑫 ESP32-S3-BOX3">楽鑫 ESP32-S3-BOX3</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="楽鑫 ESP32-S3-BOX-3">楽鑫 ESP32-S3-BOX-3</a>
- <a href="https://docs.m5stack.com/zh_CN/core/CoreS3" target="_blank" title="M5Stack CoreS3">M5Stack CoreS3</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">M5Stack AtomS3R + Echo Base</a>
- <a href="https://gf.bilibili.com/item/detail/1108782064" target="_blank" title="マジックボタン2.4">マジックボタン2.4</a>
@@ -115,12 +114,13 @@ Feishuドキュメントチュートリアルをご覧ください:
### 開発環境
- Cursor または VSCode
- ESP-IDFプラグインをインストールし、SDKバージョン5.4以上を選択
- ESP-IDFプラグインをインストールし、[ESP-IDF v6.0.2](https://github.com/espressif/esp-idf/releases/tag/v6.0.2)を優先して使用してください。v6.0以降の安定版を推奨し、ESP-IDF v5.5.2は旧ハードウェアとの互換性維持にのみ使用します
- LinuxはWindowsよりも優れており、コンパイルが速く、ドライバの問題も少ない
- 本プロジェクトはGoogle C++コードスタイルを採用、コード提出時は準拠を確認してください
### 開発者ドキュメント
- [ESP-IDF 6.0移行ガイド](docs/esp-idf-6-migration.md) - SDK互換性、コンポーネント変更、旧ハードウェア対応、ボード検証状況
- [カスタム開発ボードガイド](docs/custom-board.md) - シャオジーAI用のカスタム開発ボード作成方法
- [MCPプロトコルIoT制御使用法](docs/mcp-usage.md) - MCPプロトコルでIoTデバイスを制御する方法
- [MCPプロトコルインタラクションフロー](docs/mcp-protocol.md) - デバイス側MCPプロトコルの実装方法
+17 -17
View File
@@ -12,26 +12,25 @@
<img src="docs/mcp-based-graph.jpg" alt="通过MCP控制万物" width="320">
### 版本说明
## 近期更新
当前 v2 版本与 v1 版本分区表不兼容,所以无法从 v1 版本通过 OTA 升级到 v2 版本。分区表说明参见 [partitions/v2/README.md](partitions/v2/README.md)
使用 v1 版本的所有硬件,可以通过手动烧录固件来升级到 v2 版本
v1 的稳定版本为 1.9.2,可以通过 `git checkout v1` 来切换到 v1 版本,该分支会持续维护到 2026 年 2 月。
- 项目主线现已迁移到 ESP-IDF v6.0 或以上版本,首选稳定版为 v6.0.2;此前的 157 个发布变体已在 ESP-IDF v6.0.1 上通过构建验证。当前矩阵包含 171 个变体,其中 170 个支持 IDF 6.0.xESP32-S31 变体需要 IDF 6.1 或以上版本
- MQTT 和 BluFi 加密已迁移到 PSA Crypto,同时完成了 IDF 6 组件拆分及第三方依赖兼容处理。
- 加固了音频流水线并发、MQTT/UDP 数据包校验和发布矩阵选择逻辑
- ESP-IDF v5.5 仅保留用于文档明确标注的旧版板卡;使用 ESP-SR 2.4.7 时,ESP32-P4 Rev1 和 Rev3 均支持 IDF 6。完整兼容性和板卡验证状态请参阅 [ESP-IDF 6.0 迁移文档](docs/esp-idf-6-migration.md)。
### 已实现功能
- Wi-Fi / ML307 Cat.1 4G
- 离线语音唤醒 [ESP-SR](https://github.com/espressif/esp-sr)
- 支持两种通信协议([Websocket](docs/websocket_zh.md) 或 MQTT+UDP
- 采用 OPUS 音频编解码
- 基于流式 ASR + LLM + TTS 架构的语音交互
- 支持 Wi-Fi、有线以太网、USB RNDIS,以及 ML307/EC801E 或 NT26 Cat.1 4G 网络;部分硬件支持 Wi-Fi 与 4G 切换
- 基于 [ESP-SR](https://github.com/espressif/esp-sr) 的离线语音唤醒,支持自定义唤醒词
- 支持两种通信传输方式:[WebSocket](docs/websocket_zh.md) 和 [MQTT + UDP](docs/mqtt-udp_zh.md)
- 采用 Opus 音频流,既支持传统的流式 ASR + LLM + TTS 方案,也支持 Realtime 端到端语音模型;具备 AEC 的硬件可实现实时全双工交互
- 声纹识别,识别当前说话人的身份 [3D Speaker](https://github.com/modelscope/3D-Speaker)
- OLED / LCD 显示屏,支持表情显示
- OLED / LCD 显示屏,支持表情和丰富的情绪呈现;部分硬件支持摄像头视觉输入
- 电量显示与电源管理
- 支持多语言(中文、英文、日文)
- 支持 ESP32-C3、ESP32-S3、ESP32-P4 芯片平台
- 提供 38 种界面语言;语音提示优先使用本地化资源,缺失时自动回退到英文
- 支持 ESP32、ESP32-C3、ESP32-C5、ESP32-C6、ESP32-S3、ESP32-P4 芯片平台
- 支持热点和 BluFi 两种 Wi-Fi 配网方式
- 通过设备端 MCP 实现设备控制(音量、灯光、电机、GPIO 等)
- 通过云端 MCP 扩展大模型能力(智能家居控制、PC桌面操作、知识搜索、邮件收发等)
- 自定义唤醒词、字体、表情与聊天背景,支持网页端在线修改 ([自定义Assets生成器](https://github.com/78/xiaozhi-assets-generator))
@@ -48,10 +47,10 @@ v1 的稳定版本为 1.9.2,可以通过 `git checkout v1` 来切换到 v1 版
![面包板效果图](docs/v1/wiring2.jpg)
### 支持 70 多个开源硬件(仅展示部分)
### 支持 138 个板卡目录、171 个固件发布变体(仅展示部分)
- <a href="https://oshwhub.com/li-chuang-kai-fa-ban/li-chuang-shi-zhan-pai-esp32-s3-kai-fa-ban" target="_blank" title="立创·实战派 ESP32-S3 开发板">立创·实战派 ESP32-S3 开发板</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="乐鑫 ESP32-S3-BOX3">乐鑫 ESP32-S3-BOX3</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="乐鑫 ESP32-S3-BOX-3">乐鑫 ESP32-S3-BOX-3</a>
- <a href="https://docs.m5stack.com/zh_CN/core/CoreS3" target="_blank" title="M5Stack CoreS3">M5Stack CoreS3</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">M5Stack AtomS3R + Echo Base</a>
- <a href="https://gf.bilibili.com/item/detail/1108782064" target="_blank" title="神奇按钮 2.4">神奇按钮 2.4</a>
@@ -115,12 +114,13 @@ v1 的稳定版本为 1.9.2,可以通过 `git checkout v1` 来切换到 v1 版
### 开发环境
- Cursor 或 VSCode
- 安装 ESP-IDF 插件,选择 SDK 版本 5.4 或以上
- 安装 ESP-IDF 插件,首选 [ESP-IDF v6.0.2](https://github.com/espressif/esp-idf/releases/tag/v6.0.2),建议使用 v6.0 或以上的稳定版本;ESP-IDF v5.5.2 仅保留用于旧版硬件兼容
- Linux 比 Windows 更好,编译速度快,也免去驱动问题的困扰
- 本项目使用 Google C++ 代码风格,提交代码时请确保符合规范
### 开发者文档
- [ESP-IDF 6.0 迁移文档](docs/esp-idf-6-migration.md) - SDK 兼容性、组件变更、旧版硬件支持和板卡验证状态
- [自定义开发板指南](docs/custom-board_zh.md) - 学习如何为小智 AI 创建自定义开发板
- [MCP 协议物联网控制用法说明](docs/mcp-usage_zh.md) - 了解如何通过 MCP 协议控制物联网设备
- [MCP 协议交互流程](docs/mcp-protocol_zh.md) - 设备端 MCP 协议的实现方式
+2 -2
View File
@@ -5,7 +5,7 @@ This document explains how to enable and use BluFi (BLE-based WiFi provisioning)
## Prerequisites
- A chip and firmware configuration that support BLE.
- In `idf.py menuconfig`, enable `WiFi Configuration Method -> Esp Blufi` (`CONFIG_USE_ESP_BLUFI_WIFI_PROVISIONING=y`). If you want to use BluFi, disable the Hotspot option in the same menu; otherwise hotspot provisioning wins by default.
- In `idf.py menuconfig`, enable `WiFi Configuration Method -> ESP-BluFi` (`CONFIG_USE_ESP_BLUFI_WIFI_PROVISIONING=y`). If you want to use BluFi, disable the Hotspot option in the same menu; otherwise hotspot provisioning wins by default.
- Keep the default NVS and event-loop initialization provided by the project's `app_main`.
- Exactly one of `CONFIG_BT_BLUEDROID_ENABLED` / `CONFIG_BT_NIMBLE_ENABLED` must be selected; they are mutually exclusive.
@@ -18,7 +18,7 @@ This document explains how to enable and use BluFi (BLE-based WiFi provisioning)
## Steps
1. **Configure**: turn on `Esp Blufi` in menuconfig, then build and flash the firmware.
1. **Configure**: turn on `ESP-BluFi` in menuconfig, then build and flash the firmware.
2. **Trigger provisioning**: at first boot with no stored WiFi credentials the device enters provisioning automatically.
3. **Phone side**: open the EspBlufi app (or another BluFi client), scan and connect to the device, optionally enable encryption, then enter the WiFi SSID / password and send them.
4. **Observe the result**:
+2 -2
View File
@@ -7,7 +7,7 @@ BluFi
## 前置条件
- 需要支持 BLE 的芯片与固件配置。
-`idf.py menuconfig` 中启用 `WiFi Configuration Method -> Esp Blufi``CONFIG_USE_ESP_BLUFI_WIFI_PROVISIONING=y`
-`idf.py menuconfig` 中启用 `WiFi Configuration Method -> ESP-BluFi``CONFIG_USE_ESP_BLUFI_WIFI_PROVISIONING=y`
)。如果想使用 BluFi,必须关闭同一菜单下的 Hotspot 选项,否则默认使用 Hotspot 配网模式。
- 保持默认的 NVS 与事件循环初始化(项目的 `app_main` 已处理)。
@@ -21,7 +21,7 @@ BluFi
## 使用步骤
1. 配置:在 menuconfig 开启 `Esp Blufi`。编译并烧录固件。
1. 配置:在 menuconfig 开启 `ESP-BluFi`。编译并烧录固件。
2. 触发配网:设备首次启动且没有已保存的 Wi‑Fi 时会自动进入配网。
3. 手机端操作:打开 EspBlufi App(或其他 BluFi 客户端),搜索并连接设备,可以选择是否加密,按提示输入 Wi‑Fi SSID/密码并发送。
4. 观察结果:
+66 -24
View File
@@ -4,7 +4,7 @@ This guide describes how to add a new board to the XiaoZhi AI voice assistant pr
## Important
> **Warning**: for a custom board whose IO configuration differs from an existing board, never overwrite the original board's configuration. Always create a new board type - or use the `builds` array in `config.json` to produce a distinct firmware name with different `sdkconfig` macros. Use `python scripts/release.py [board-directory]` to build and package the firmware.
> **Warning**: for a custom board whose IO configuration differs from an existing board, never overwrite the original board's configuration. Always create a new board type - or use the `builds` array in `config.json` to produce a distinct firmware name with different `sdkconfig` macros. Use `python scripts/build.py [board-directory]` to build the firmware.
>
> Overwriting an existing board's configuration is dangerous because OTA updates may replace your custom firmware with the stock firmware for the original board. Every board must have a unique identity and its own firmware update channel.
@@ -14,7 +14,7 @@ A board directory typically contains:
- `xxx_board.cc` - board-level initialization and glue code.
- `config.h` - pin assignments and board-level settings.
- `config.json` - build configuration consumed by `scripts/release.py`.
- `config.json` - reported board type and release configuration consumed by CMake and `scripts/build.py`.
- `README.md` - board-specific notes.
Boards can live directly under `main/boards/` or be grouped by manufacturer under `main/boards/<manufacturer>/<board>/` (see [Manufacturer Sub-directories](#manufacturer-sub-directories) below).
@@ -89,10 +89,11 @@ Example (from `lichuang-c3-dev`):
#### config.json
`config.json` drives `scripts/release.py`:
`config.json` defines the compatibility-sensitive reported type and drives `scripts/build.py`:
```json
{
"type": "my-custom-board",
"target": "esp32s3",
"builds": [
{
@@ -107,32 +108,40 @@ Example (from `lichuang-c3-dev`):
```
**Fields**:
- `type`: compatibility-sensitive board family reported by the firmware. Keep it stable after release.
- `target`: target chip, must match the real hardware (`esp32`, `esp32s3`, `esp32c3`, `esp32c6`, `esp32p4`, ...).
- `name`: firmware package name; typically matches the directory name.
- `name`: compatibility-sensitive firmware variant name reported by release builds; typically matches `type`.
- `sdkconfig_append`: extra sdkconfig lines merged into the defaults.
Both `type` and `name` must contain only lowercase letters, digits, periods
(`.`), and hyphens (`-`). Underscores, spaces, and uppercase letters are not
allowed.
**Common `sdkconfig_append` entries**:
```json
// Flash size
"CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y"
"CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y"
"CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y"
// Partition table
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/4m.csv\""
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/8m.csv\""
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/16m.csv\""
// Language
"CONFIG_LANGUAGE_EN_US=y"
"CONFIG_LANGUAGE_ZH_CN=y"
// Wake word configuration
// Audio pipeline
"CONFIG_USE_DEVICE_AEC=y" // enable on-device AEC
"CONFIG_WAKE_WORD_DISABLED=y" // disable wake word detection
```
The project defaults to 16MB flash and `partitions/v2/16m.csv` on applicable
targets. Do not repeat values that already match the effective project and
target defaults; use `sdkconfig_append` only for actual board-specific
overrides.
Do not select a language or a specific wake word in a board `config.json`.
Those are user build options and must be configured consistently through
`menuconfig` or build-script parameters so CLI, agent, and online builds can
share the same interface.
### 3. Implement the Board Class
Create `my_custom_board.cc` containing the board-level implementation.
@@ -310,7 +319,7 @@ Open `main/CMakeLists.txt` and extend the board-type chain:
```cmake
elseif(CONFIG_BOARD_TYPE_MY_CUSTOM_BOARD)
set(BOARD_TYPE "my-custom-board") # must match the directory name
set(BOARD_DIR "my-custom-board")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4) # pick a font for the display
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64) // optional, for emoji display
@@ -356,18 +365,51 @@ Emoji collections:
idf.py flash monitor
```
#### Option B - use `release.py` (recommended)
#### Option B - use `build.py` (recommended)
If the board directory contains a `config.json`, you can build and package automatically:
If the board directory contains a `config.json`, you can configure and build it automatically:
```bash
python scripts/release.py my-custom-board
python scripts/build.py my-custom-board
```
Language and wake-word selection are user build options:
```bash
python scripts/build.py my-custom-board \
--language en-US \
--wake-word wn9_jarvis_tts
```
`--language` accepts a locale listed under `main/assets/locales/`.
`--wake-word` accepts an ESP-SR model name, `nihaoxiaozhi` (which selects the
compatible model for the target), or `disabled`. ESP32-C3/C5/C6 targets support
WakeNet9s (`wn9s_*`) models; ESP32-S3/P4/S31 builds automatically use the AFE
wake-word engine.
Query the accepted values in text or machine-readable form:
```bash
python scripts/build.py --list-languages
python scripts/build.py --list-languages --json
python scripts/build.py --list-wake-words
python scripts/build.py --list-wake-words --json
```
The wake-word list is read from the currently resolved ESP-SR component, so
run `idf.py reconfigure` first if `managed_components/` has not been populated.
The script:
- Reads `target` from `config.json` and calls `idf.py set-target`.
- Appends the entries listed in `sdkconfig_append`.
- Builds and packages the firmware.
- Prints help when run without arguments. Use `--list-boards` to list board
types and variants.
- Prompts for a variant when the selected board has multiple builds. In
non-interactive environments, pass `--name <variant>`.
- Reads `target` from `config.json`. It calls `idf.py set-target` only when the
target changes, then regenerates `sdkconfig` from defaults and the selected
build's `sdkconfig_append`.
- Passes the selected build's `name` as the reported firmware variant name.
- Builds `build/merged-binary.bin` without creating a ZIP by default. Pass
`--zip` to recreate `releases/v<version>_<name>.zip`.
### 6. Write the README
@@ -377,18 +419,19 @@ In `README.md`, describe the board, hardware requirements, build instructions, a
Boards can be grouped by manufacturer under `main/boards/<manufacturer>/<board>/`. This is the recommended layout when a single vendor ships several variants - for example `main/boards/waveshare/esp32-p4-nano/` or `main/boards/lceda-course-examples/eda-tv-pro/`.
To enable the layout, set the `MANUFACTURER` variable in `main/CMakeLists.txt` for your board:
For a board in a manufacturer sub-directory, add the same value to `config.json`, for example `"manufacturer": "waveshare"`. The firmware reports it as `board.manufacturer` together with `board.type` and `board.name`. Flat community boards without this field report an empty manufacturer string.
Set `BOARD_DIR` to the complete path relative to `main/boards/`:
```cmake
elseif(CONFIG_BOARD_TYPE_WAVESHARE_ESP32_P4_NANO)
set(MANUFACTURER "waveshare")
set(BOARD_TYPE "esp32-p4-nano")
set(BOARD_DIR "waveshare/esp32-p4-nano")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
```
When `MANUFACTURER` is set, the build system globs source files from `main/boards/${MANUFACTURER}/${BOARD_TYPE}/`. When it is empty, it falls back to the flat `main/boards/${BOARD_TYPE}/` layout.
The build system loads sources from `main/boards/${BOARD_DIR}/` and reads the reported board type from that directory's `config.json`. If `config.json` or its top-level `type` is absent, the full `BOARD_DIR` with `/` replaced by `-` is used as the fallback type.
Rules of thumb:
- Use the manufacturer layout when you have two or more boards from the same vendor that share drivers, assets, or documentation.
@@ -437,7 +480,6 @@ Supported LCD families include:
- `Button` - standard push buttons (click, long-press, multi-click).
- `Knob` - rotary encoder wrapper.
- `PressToTalkMcpTool` - push-to-talk tool that registers itself through MCP.
- `AfskDemod` - AFSK demodulator used by some acoustic provisioning flows.
- `SystemReset` - helper that performs a safe factory reset when a button is held at boot.
### MCP integration
+56 -19
View File
@@ -4,7 +4,7 @@
## 重要提示
> **警告**: 对于自定义开发板,当IO配置与原有开发板不同时,切勿直接覆盖原有开发板的配置编译固件。必须创建新的开发板类型,或者通过config.json文件中的builds配置不同的name和sdkconfig宏定义来区分。使用 `python scripts/release.py [开发板目录名字]` 来编译打包固件。
> **警告**: 对于自定义开发板,当IO配置与原有开发板不同时,切勿直接覆盖原有开发板的配置编译固件。必须创建新的开发板类型,或者通过config.json文件中的builds配置不同的name和sdkconfig宏定义来区分。使用 `python scripts/build.py [开发板目录名字]` 来编译固件。
>
> 如果直接覆盖原有配置,将来OTA升级时,您的自定义固件可能会被原有开发板的标准固件覆盖,导致您的设备无法正常工作。每个开发板有唯一的标识和对应的固件升级通道,保持开发板标识的唯一性非常重要。
@@ -14,7 +14,7 @@
- `xxx_board.cc` - 主要的板级初始化代码,实现了板子相关的初始化和功能
- `config.h` - 板级配置文件,定义了硬件管脚映射和其他配置项
- `config.json` - 编译配置,指定目标芯片和特殊的编译选项
- `config.json` - 上报开发板类型及发布配置,供 CMake 和 `scripts/build.py` 使用
- `README.md` - 开发板相关的说明文档
## 定制开发板步骤
@@ -87,10 +87,11 @@ mkdir main/boards/my-custom-board
#### config.json
`config.json`中定义编译配置,这个文件用于 `scripts/release.py` 脚本自动化编译:
`config.json`中定义兼容性相关的上报类型和自动化编译配置
```json
{
"type": "my-custom-board", // 固件上报的开发板类型,发布后应保持稳定
"target": "esp32s3", // 目标芯片型号: esp32, esp32s3, esp32c3, esp32c6, esp32p4等
"builds": [
{
@@ -107,31 +108,37 @@ mkdir main/boards/my-custom-board
```
**配置项说明:**
- `manufacturer`: 厂商子目录名称;使用厂商目录时必须填写,并作为 `board.manufacturer` 上报。平铺社区板可以省略,上报为空字符串
- `type`: 固件上报的开发板系列类型,发布后应保持稳定
- `target`: 目标芯片型号,必须与硬件匹配
- `name`: 编译输出的固件名称,建议与目录名一致
- `name`: release 构建上报的固件变体名称,通常与 `type` 一致
- `sdkconfig_append`: 额外的 sdkconfig 配置项数组,会追加到默认配置中
`type``name` 只能包含小写字母、数字、点(`.`)和连字符(`-`),
不允许使用下划线、空格或大写字母。
**常用的 sdkconfig_append 配置:**
```json
// Flash 大小
"CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y" // 4MB Flash
"CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y" // 8MB Flash
"CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y" // 16MB Flash
// 分区表
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/4m.csv\"" // 4MB 分区表
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/8m.csv\"" // 8MB 分区表
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/16m.csv\"" // 16MB 分区表
// 语言配置
"CONFIG_LANGUAGE_EN_US=y" // 英语
"CONFIG_LANGUAGE_ZH_CN=y" // 简体中文
// 唤醒词配置
// 音频处理
"CONFIG_USE_DEVICE_AEC=y" // 启用设备端 AEC
"CONFIG_WAKE_WORD_DISABLED=y" // 禁用唤醒词
```
对于适用的目标芯片,项目默认使用 16MB Flash 和
`partitions/v2/16m.csv`。如果板子配置与项目及目标芯片的有效默认值相同,
不要在 `sdkconfig_append` 中重复填写;这里应只保留板子确实需要的覆盖项。
不要在板子的 `config.json` 中选择语言或具体唤醒词。这些属于用户构建选项,
应统一通过 `menuconfig` 或构建脚本参数配置,方便 CLI、Agent 和在线编译接口
共用同一套参数。
### 3. 编写板级初始化代码
创建一个`my_custom_board.cc`文件,实现开发板的所有初始化逻辑。
@@ -321,7 +328,7 @@ endchoice
```cmake
# 在 elseif 链中添加你的开发板配置
elseif(CONFIG_BOARD_TYPE_MY_CUSTOM_BOARD)
set(BOARD_TYPE "my-custom-board") # 与目录名一致
set(BOARD_DIR "my-custom-board") # 相对于 main/boards 的完整路径
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4) # 根据屏幕大小选择合适的字体
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64) # 可选,如果需要表情显示
@@ -374,18 +381,48 @@ endif()
idf.py flash monitor
```
#### 方法二:使用 release.py 脚本(推荐)
#### 方法二:使用 build.py 脚本(推荐)
如果你的开发板目录下有 `config.json` 文件,可以使用此脚本自动完成配置和编译:
```bash
python scripts/release.py my-custom-board
python scripts/build.py my-custom-board
```
语言和唤醒词属于用户构建参数:
```bash
python scripts/build.py my-custom-board \
--language en-US \
--wake-word wn9_jarvis_tts
```
`--language` 接受 `main/assets/locales/` 下已有的 locale`--wake-word`
接受 ESP-SR 模型名、`nihaoxiaozhi`(自动选择与目标芯片兼容的模型)或
`disabled`。ESP32-C3/C5/C6 仅支持 WakeNet9s`wn9s_*`)模型;
ESP32-S3/P4/S31 构建会自动使用 AFE 唤醒引擎。
可以用文本或 JSON 格式查询可用值:
```bash
python scripts/build.py --list-languages
python scripts/build.py --list-languages --json
python scripts/build.py --list-wake-words
python scripts/build.py --list-wake-words --json
```
唤醒词列表从当前已解析的 ESP-SR 组件读取。如果尚未生成
`managed_components/`,请先运行 `idf.py reconfigure`。
此脚本会自动:
- 读取 `config.json` 中的 `target` 配置并设置目标芯片
- 应用 `sdkconfig_append` 中的编译选项
- 完成编译并打包固件
- 不传参数时打印帮助;使用 `--list-boards` 列出所有开发板类型和变体
- 如果开发板有多个变体,交互式提示选择;非交互环境使用 `--name <变体>`
- 读取 `config.json` 中的 `target`,仅在目标芯片变化时调用
`idf.py set-target`,然后根据 defaults 和所选变体的
`sdkconfig_append` 重新生成 `sdkconfig`
- 将所选 build 的 `name` 作为固件上报的变体名称
- 默认生成 `build/merged-binary.bin`,不创建 ZIP;指定 `--zip` 时会重新生成
`releases/v<版本>_<名称>.zip`
### 6. 创建README.md
@@ -450,4 +487,4 @@ python scripts/release.py my-custom-board
- ESP-IDF 文档: https://docs.espressif.com/projects/esp-idf/
- LVGL 文档: https://docs.lvgl.io/
- ESP-SR 文档: https://github.com/espressif/esp-sr
- ESP-SR 文档: https://github.com/espressif/esp-sr
+257
View File
@@ -0,0 +1,257 @@
# ESP-IDF 6.0 Migration and Board Compatibility Status
> Last updated: 2026-07-24
> Validated SDK: ESP-IDF v6.0.1
> Scope: 138 board directories and 171 supported build variants defined by `main/boards/**/config.json`.
## Current Status
The current release matrix contains 171 variants: 170 select on IDF 6.0.x, while the ESP32-S31 variant requires IDF 6.1 or later. The previous 157-variant baseline passed the latest complete GitHub Actions matrix on ESP-IDF 6.0.1. ESP-SR 2.4.7 adds IDF 6 support for ESP32-P4 Rev < 3, so the 14 unsuffixed Rev1 variants now join their existing `-p4x` Rev3 counterparts in the IDF 6 matrix. Component versions that use ranges are resolved from the Component Registry at build time; until per-target lock snapshots are committed, this is a source-reproducible build rather than a bit-for-bit dependency-reproducible build.
| Status | Variants | Meaning |
|---|---:|---|
| ✅ Previously matrix-validated | 157 | All 157 baseline builds completed in GitHub Actions with ESP-IDF 6.0.1; this does not imply hardware or complete peripheral validation |
| 🟡 Net variants added since full-matrix validation | 14 | Includes the 14 newly enabled P4 Rev1 variants and the IDF 6.1-only ESP32-S31 variant, after retired low-use variants were removed; a complete current-matrix CI run remains pending |
| 🟡 Feature-degraded subset | 1 | `esp-vocat` builds on IDF 6, but its PCB capacitive slider/button support is disabled pending compatible touch-sensor components |
| 🔴 Build-blocked | 0 | No supported release variant remains blocked at compile or link time |
There are no known remaining IDF 6 build blockers. [`78/esp_lcd_nv3023 1.0.1`](https://components.espressif.com/components/78/esp_lcd_nv3023/versions/1.0.1) and [`wvirgil123/sscma_client 1.0.3`](https://components.espressif.com/components/wvirgil123/sscma_client/versions/1.0.3/readme) are consumed directly from the Component Registry, so local copies under the ignored `components/` directory are not required. ESP32-P4 Rev1 and Rev3 now share the IDF 6 build path while retaining separate artifact names and silicon-selection settings.
Representative local full-build results are shown below. Firmware size and free space are reported by ESP-IDF 6.0.1 `check_sizes.py`. The authoritative per-variant compatibility result is the full GitHub Actions matrix in the next section.
| Chip | Build variant | Application size | Free space in smallest app partition |
|---|---|---:|---:|
| ESP32 | `bread-compact-esp32` | `0x240560` | 23% |
| ESP32-C3 | `espressif-esp-hi` | `0x20ef10` | 30% |
| ESP32-C3 | `xmini-c3` | `0x2370f0` | 44% |
| ESP32-C5 | `waveshare-esp32-c5-touch-lcd-1.69` | `0x27cd30` | 37% |
| ESP32-C6 | `waveshare-esp32-c6-lcd-0.85` | `0x280ee0` | 15% |
| ESP32-S3 | `doit-s3-aibox` | `0x21edf0` | 46% |
| ESP32-S3 | `lilygo-t-display-s3-pro-mvsrlora` | `0x2adce0` | 32% |
| ESP32-S3 | `lilygo-t-circle-s3` | `0x2a32f0` | 33% |
| ESP32-S3 | `zhengchen-1.54tft-wifi` | `0x2abdc0` | 32% |
| ESP32-S3 | `kevin-yuying-313lcd` | `0x2c4990` | 30% |
| ESP32-S3 | `otto-robot` | `0x37ff70` | 11% |
| ESP32-S3 | `espressif-esp-vocat` | `0x2742f0` | 38% |
| ESP32-S3 | `sensecap-watcher` | `0x2faff0` | 25% |
| ESP32-P4 Rev1 | `espressif-esp32-p4-function-ev-board` | `0x345a90` | 17% |
| ESP32-P4 Rev3 | `espressif-esp32-p4x-function-ev-board` | `0x34a150` | 16% |
| ESP32-P4 v3.x | `m5stack-tab5-p4x` | `0x380790` | 11% |
The old negative result for `esp32-p4-function-ev-board` came from ESP-SR 2.4.6. ESP-SR 2.4.7 supplies the missing ESP32-P4 Rev < 3 libraries for IDF 6, so the project no longer version-gates the P4 variants.
For backward-compatibility regression coverage, `xmini-c3` also completed a full build with ESP-IDF 5.5.4 (application size `0x234920`, 44% free in the smallest app partition). The legacy `esp32-p4-function-ev-board` release variant subsequently completed the same 5.5.4 release flow (application size `0x38a130`, 10% free), including merged-binary packaging. This confirms that the compatibility changes for I2S port numbering, LCD I2C configuration, the UHCI DMA dependency, and the pre-v3 P4 selection did not break the existing 5.5 build chain.
## Full-Matrix CI Validation
GitHub Actions run [29534954031](https://github.com/78/xiaozhi-esp32/actions/runs/29534954031) built the then-current matrix with the `espressif/idf:v6.0.1` container. The matrix-generation job and all 157 board builds passed. This historical run predates the current 170-variant IDF 6.0.x matrix.
Results by chip target:
| Chip target | Variants | Passed | Blocked |
|---|---:|---:|---:|
| ESP32 | 7 | 7 | 0 |
| ESP32-C3 | 9 | 9 | 0 |
| ESP32-C5 | 4 | 4 | 0 |
| ESP32-C6 | 9 | 9 | 0 |
| ESP32-S3 | 114 | 114 | 0 |
| ESP32-P4 v3.x | 14 | 14 | 0 |
| **Total** | **157** | **157** | **0** |
`esp-vocat` remains feature-degraded because IDF 6 builds omit the PCB capacitive slider/button path; its CST816 display touch remains enabled.
## ESP32-P4 Silicon Scope and Naming
The IDF 6 release matrix supports both ESP32-P4 Rev < 3 and Rev >= 3 silicon. The two silicon families keep distinct artifact names and sdkconfig settings:
| SDK | ESP32-P4 Rev < 3 | ESP32-P4 Rev >= 3 |
|---|---|---|
| ESP-IDF < 6 | Original `esp32-p4-` name; adds `CONFIG_ESP32P4_SELECTS_REV_LESS_V3=y` and `CONFIG_ESP32P4_REV_MIN_100=y` | Uses `esp32-p4x-` when the variant starts with the chip name; otherwise retains the established `-p4x` suffix |
| ESP-IDF >= 6 | Original `esp32-p4-` name; adds `CONFIG_ESP32P4_SELECTS_REV_LESS_V3=y` and `CONFIG_ESP32P4_REV_MIN_100=y` | Uses `esp32-p4x-` when the variant starts with the chip name; otherwise retains the established `-p4x` suffix |
ESP-IDF 5.5 and 6.0.x each select 170 variants: 14 P4X artifacts, 14 Rev < 3 P4 artifacts with their original names, and 142 non-P4 variants. ESP-IDF 6.1 or later additionally selects the ESP32-S31 build, for 171 variants in total.
Espressif's current chip-identification table lists v0.0, v1.0, v1.3, v3.0, v3.1, and v3.2, with no v4.0 revision. The public errata history added v3.0/v3.1 information on 2026-02-12 and v3.2 information on 2026-04-20. ESP-IDF 6.0 release notes explicitly describe support for "ESP32-P4 Version3 silicon." See the official [chip revision identification](https://docs.espressif.com/projects/esp-chip-errata/en/latest/esp32p4/01-chip-identification/index.html), [errata revision history](https://docs.espressif.com/projects/esp-chip-errata/en/latest/esp32p4/revision-history/index.html), and [ESP-IDF releases](https://github.com/espressif/esp-idf/releases).
## Component Compatibility
| Component/module | Version or approach for IDF 6 | Status | Notes |
|---|---|---|---|
| `78/uart-uhci` | `0.3.2` | ✅ Upstream support | The registry release compiles under IDF 6.0.1 in GitHub Actions |
| `78/uart-eth-modem` | `0.6.0` | ✅ Upstream support | Pinned because it has the required RF-test event API while supporting both ESP-IDF 5.5.2+ and 6.0.1; version 0.6.1 requires IDF 6.0.1+ |
| `espressif/mqtt` | `1.0.0` | ✅ Upstream support | MQTT moved from a built-in SDK component to a Component Manager dependency in IDF 6 |
| `78/esp-ml307` | `3.6.6` + project-level `espressif/mqtt` | ✅ Builds in CI | The upstream source compiles under IDF 6.0.1 when the project supplies the MQTT component moved out of IDF |
| `espressif/esp_hosted` / `esp_wifi_remote` | `2.12.11` / `1.6.2` | ✅ Upstream support | Used for ESP32-P4 Hosted Wi-Fi |
| `espressif/esp_video` | `^2.0.1` | ✅ Upstream support | Currently resolves to 2.3.0 on S3 and 2.0.1 on P4 due to BSP constraints |
| P4 BSP / LCD drivers | BSP `5.2.3`, EK79007/ST7701 `2.0.x` | ✅ Upstream support | Resolves IDF 6 DPI, color-field, and split-driver-component issues |
| `espressif/esp_lcd_st77916` | `2.0.2` | ✅ Upstream support | Major version 2 uses the IDF 6 panel I/O definitions |
| `espressif/esp_lcd_spd2010` | `^2.0.0` (resolved `2.0.0~1`) | ✅ Upstream support | Major version 2 declares ESP-IDF 6 compatibility |
| `espressif/esp_lcd_co5300` | `2.1.0` | ✅ Upstream support | Used with explicit IDF 6-compatible QSPI I/O configuration in board code |
| `esp_emote_expression` | `1.0.2` | ✅ Upstream support | Replaces the previous dependency on the built-in `json` component |
| `wvirgil123/sscma_client` | [`1.0.3`](https://components.espressif.com/components/wvirgil123/sscma_client/versions/1.0.3/readme) | ✅ Upstream support | Uses `espressif/cjson` and the split driver components on IDF 6 while retaining the legacy component names on older IDF releases |
| `espressif/servo` / `espfriends/servo_dog_ctrl` | `1.0.0` / `0.2.0` | ✅ Upstream support | Uses official registry releases; compilation and linking were validated in the full IDF 6.0.1 build of `esp-hi`, with no local override required |
| `78/esp_lcd_nv3023` | [`1.0.1`](https://components.espressif.com/components/78/esp_lcd_nv3023/versions/1.0.1) | ✅ Upstream-based registry release | Mirrors MakerM0 upstream commit `15dae953`; adds IDF 6 color-order and GPIO compatibility while retaining older IDF branches; validated through the registry in a clean `magiclick-c3` full build |
| `llgok/cpp_bus_driver` | Excluded from the IDF 6 baseline | ⚪ Waiting for upstream | Version `2.1.0` still requires local patches for IDF 6. It is used only by the `LILYGO T-Display-P4` source, and that board has no release `config.json`, so the component has been removed until upstream support is available |
| MQTT protocol AES-CTR | PSA Crypto | ✅ Ported | Replaces the legacy AES context API removed by IDF 6 / Mbed TLS 4 |
| BluFi security negotiation (conditional path) | PSA FFDH + SHA-256 + AES-CTR | ✅ Ported | Uses the ESP-IDF 6 security scheme with ffdhe3072 and passed local full ESP32-S3 builds with BluFi enabled on IDF 5.5.4 and 6.0.1. Legacy 1024-bit BluFi clients are not compatible and must be upgraded |
| `espressif/bmi270_sensor` | [`0.1.2`](https://components.espressif.com/components/espressif/bmi270_sensor/versions/0.1.2/readme?language=en) | ✅ Upstream support | Provides IDF 6.0 prebuilt libraries for ESP32-C5 and ESP32-S3; validated by a full `esp-spot-c5` build |
| `touch_slider_sensor` / `touch_button_sensor` | Disabled for IDF 6 | 🟡 Feature gap | Their manifests require IDF < 6.0, so ESP Vocat omits its PCB capacitive slider/button path on IDF 6. CST816 display touch is unaffected |
| ESP32-P4 Rev < 3 / `espressif/esp-sr` | `~2.4.7` | ✅ Upstream support | The 14 Rev1 variants are available on IDF 5.5 and IDF 6; P4X-specific names continue to select Rev >= 3 |
## Per-Board Progress
In the table below, "Board" is the source directory and "Build variant" is the firmware name used by the build script. Different chip revisions of the same board may have different status, so they are split into separate rows when necessary.
| Chip | Board | Build variant | IDF 6.0 status | Current validation | Blocker/next step |
|---|---|---|---|---|---|
| `esp32` | `atommatrix-echo-base` | `atommatrix-echo-base` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32` | `bread-compact-esp32` | `bread-compact-esp32` | ✅ Full build passed | IDF 6.0.1 `idf.py build` | Hardware smoke/peripheral regression pending |
| `esp32` | `bread-compact-esp32` | `bread-compact-esp32-128x32` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32` | `bread-compact-esp32-lcd` | `bread-compact-esp32-lcd` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32` | `wdmomo/esp32-cgc` | `wdmomo-esp32-cgc` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32` | `wdmomo/esp32-cgc-144` | `wdmomo-esp32-cgc-144` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32` | `waveshare/esp32-touch-lcd-3.5` | `waveshare-esp32-touch-lcd-3.5` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c3` | `espressif/esp-hi` | `espressif-esp-hi` | ✅ Full build passed | IDF 6.0.1 `idf.py build`; `servo_dog_ctrl 0.2.0` | Hardware smoke, servo, and audio peripheral regression pending |
| `esp32c3` | `kevin/c3` | `kevin-c3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c3` | `lckfb/szpi-esp32c3` | `lckfb-lichuang-c3-dev` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c3` | `magiclick-c3` | `magiclick-c3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c3` | `magiclick-c3-v2` | `magiclick-c3-v2` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c3` | `surfer-c3-1.14tft` | `surfer-c3-1.14tft` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c3` | `xmini-c3` | `xmini-c3` | ✅ Full build passed | IDF 6.0.1 `idf.py build` | Hardware smoke/peripheral regression pending |
| `esp32c3` | `xmini-c3-4g` | `xmini-c3-4g` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c3` | `xmini-c3-v3` | `xmini-c3-v3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c5` | `espressif/esp-sensairshuttle` | `espressif-esp-sensairshuttle` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c5` | `espressif/esp-spot` | `espressif-esp-spot-c5` | ✅ Full build passed | IDF 6.0.1 `idf.py build`; `bmi270_sensor 0.1.2` | Hardware smoke, BMI270, audio, and power-management regression pending |
| `esp32c5` | `movecall-moji2-esp32c5` | `movecall-moji2-esp32c5` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c5` | `waveshare/esp32-c5-touch-lcd-1.69` | `waveshare-esp32-c5-touch-lcd-1.69` | ✅ Full build passed | IDF 6.0.1 `idf.py build` | Hardware smoke/peripheral regression pending |
| `esp32c6` | `waveshare/esp32-c6-lcd-0.85` | `waveshare-esp32-c6-lcd-0.85` | ✅ Full build passed | IDF 6.0.1 `idf.py build` | Hardware smoke/peripheral regression pending |
| `esp32c6` | `waveshare/esp32-c6-lcd-1.69` | `waveshare-esp32-c6-lcd-1.69` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c6` | `waveshare/esp32-c6-touch-amoled-1.32` | `waveshare-esp32-c6-touch-amoled-1.32` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c6` | `waveshare/esp32-c6-touch-amoled-1.43` | `waveshare-esp32-c6-touch-amoled-1.43` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c6` | `waveshare/esp32-c6-touch-amoled-1.8` | `waveshare-esp32-c6-touch-amoled-1.8` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c6` | `waveshare/esp32-c6-touch-amoled-2.06` | `waveshare-esp32-c6-touch-amoled-2.06` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c6` | `waveshare/esp32-c6-touch-amoled-2.16` | `waveshare-esp32-c6-touch-amoled-2.16` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c6` | `waveshare/esp32-c6-touch-lcd-1.54` | `waveshare-esp32-c6-touch-lcd-1.54` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32c6` | `waveshare/esp32-c6-touch-lcd-1.83` | `waveshare-esp32-c6-touch-lcd-1.83` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `xorigin/aipi-lite` | `xorigin-aipi-lite` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `alientek/atk-dnesp32s3` | `alientek-atk-dnesp32s3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `alientek/atk-dnesp32s3-box` | `alientek-atk-dnesp32s3-box` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `alientek/atk-dnesp32s3-box0` | `alientek-atk-dnesp32s3-box0` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `alientek/atk-dnesp32s3-box2-4g` | `alientek-atk-dnesp32s3-box2-4g` | ✅ Build and hardware validated | GitHub Actions IDF 6.0.1 full build; maintainer hardware validation | Complete for the IDF 6 migration scope |
| `esp32s3` | `alientek/atk-dnesp32s3-box2-wifi` | `alientek-atk-dnesp32s3-box2-wifi` | ✅ Build and hardware validated | GitHub Actions IDF 6.0.1 full build; maintainer hardware validation | Complete for the IDF 6 migration scope |
| `esp32s3` | `alientek/atk-dnesp32s3-box3` | `alientek-atk-dnesp32s3-box3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `atom-echos3r` | `atom-echos3r` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `atoms3-echo-base` | `atoms3-echo-base` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `atoms3r-cam-m12-echo-base` | `atoms3r-cam-m12-echo-base` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `atoms3r-echo-base` | `atoms3r-echo-base` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `atoms3r-echo-pyramid` | `atoms3r-echo-pyramid` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `bread-compact-ml307` | `bread-compact-ml307`<br>`bread-compact-ml307-128x64` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `bread-compact-nt26` | `bread-compact-nt26` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `bread-compact-wifi` | `bread-compact-wifi`<br>`bread-compact-wifi-128x64` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `df-k10` | `df-k10` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `df-s3-ai-cam` | `df-s3-ai-cam` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `doit-s3-aibox` | `doit-s3-aibox` | ✅ Full build passed | IDF 6.0.1 `idf.py build` | Hardware smoke/peripheral regression pending |
| `esp32s3` | `du-chatx` | `du-chatx` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `electron-bot` | `electron-bot` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `espressif/esp32-s3-box` | `espressif-esp-box` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `espressif/esp32-s3-box-3` | `espressif-esp-box-3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `espressif/esp32-s3-box-lite` | `espressif-esp-box-lite` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `espressif/esp32-s3-lcd-ev-board` | `espressif-esp-s3-lcd-ev-board-1p4`<br>`espressif-esp-s3-lcd-ev-board-1p5` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `espressif/esp32-s3-lcd-ev-board-2` | `espressif-esp32-s3-lcd-ev-board-2` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `espressif/esp-sparkbot` | `espressif-esp-sparkbot` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `espressif/esp-vocat` | `espressif-esp-vocat` | 🟡 Full build passed; feature-degraded | Local ESP-IDF 6.0.1 full build after rebasing onto `origin/main` | PCB capacitive slider/button disabled on IDF 6; CST816 display touch remains enabled; hardware regression pending |
| `esp32s3` | `espressif/esp32-s3-korvo-2-v3.0` | `espressif-esp32s3-korvo2-v3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `espressif/esp32-s3-korvo-2-v3.0-rndis` | `espressif-esp32s3-korvo2-v3-rndis` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `freenove-esp32s3-display-2.8-lcd` | `freenove-esp32s3-display-2.8-lcd` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `genjutech-s3-1.54tft` | `genjutech-s3-1.54tft` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `hu-087` | `hu-087` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `jiuchuan-s3` | `jiuchuan-s3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `kevin/box-2` | `kevin-box-2` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `kevin/sp-v4-dev` | `kevin-sp-v4-dev` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `kevin/yuying-313lcd` | `kevin-yuying-313lcd` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `labplus/ledong-v2` | `labplus-ledong-v2` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `labplus/mpython-v3` | `labplus-mpython-v3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `lceda-course-examples/eda-robot-pro` | `lceda-course-examples-eda-robot-pro` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `lceda-course-examples/eda-super-bear` | `lceda-course-examples-eda-super-bear` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `lceda-course-examples/eda-tv-pro` | `lceda-course-examples-eda-tv-pro` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `lckfb/szpi-esp32s3` | `lckfb-lichuang-dev` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `lilygo-t-cameraplus-s3` | `lilygo-t-cameraplus-s3`<br>`lilygo-t-cameraplus-s3-v1.2` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `lilygo-t-circle-s3` | `lilygo-t-circle-s3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `lilygo-t-display-s3-pro-mvsrlora` | `lilygo-t-display-s3-pro-mvsrlora` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `m5stack-cardputer-adv` | `m5stack-cardputer-adv` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `m5stack-core-s3` | `m5stack-core-s3` | ✅ Build and hardware validated | GitHub Actions IDF 6.0.1 full build; maintainer hardware validation | Complete for the IDF 6 migration scope |
| `esp32s3` | `m5stack-stick-s3` | `m5stack-stick-s3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `m5stack-stopwatch` | `m5stack-stopwatch` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `magiclick-2p4` | `magiclick-2p4` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `magiclick-2p5` | `magiclick-2p5` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `minsi-k08-dual` | `minsi-k08-dual` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `mixgo-nova` | `mixgo-nova` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `movecall-cuican-esp32s3` | `movecall-cuican-esp32s3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `movecall-moji-esp32s3` | `movecall-moji-esp32s3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `nulllab-ai-vox-v3` | `nulllab-ai-vox-v3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `otto-robot` | `otto-robot` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `quandong-s3-dev` | `quandong-s3-dev` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `rymcu/bigsmart` | `rymcu-bigsmart` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `sensecap-watcher` | `sensecap-watcher` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `sp-esp32-s3-1.28-box` | `sp-esp32-s3-1.28-box` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `sp-esp32-s3-1.54-muma` | `sp-esp32-s3-1.54-muma` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `taiji-pi-s3` | `taiji-pi-s3`<br>`taiji-pi-s3-pdm` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-audio-board` | `waveshare-esp32-s3-audio-board` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-cam` | `waveshare-esp32-s3-cam-2`<br>`waveshare-esp32-s3-cam-2.8`<br>`waveshare-esp32-s3-cam-3.5`<br>`waveshare-esp32-s3-cam-1.83` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-epaper-1.54` | `waveshare-esp32-s3-epaper-1.54-v2`<br>`waveshare-esp32-s3-epaper-1.54-v1` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-epaper-3.97` | `waveshare-esp32-s3-epaper-3.97` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-lcd-0.85` | `waveshare-esp32-s3-lcd-0.85` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-rgb-matrix` | `waveshare-esp32-s3-rgb-matrix` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-rlcd-4.2` | `waveshare-esp32-s3-rlcd-4.2` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-amoled-1.32` | `waveshare-esp32-s3-touch-amoled-1.32` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-amoled-1.43c` | `waveshare-esp32-s3-touch-amoled-1.43c` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-amoled-1.75` | `waveshare-esp32-s3-touch-amoled-1.75`<br>`waveshare-esp32-s3-touch-amoled-1.75c` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-amoled-1.8` | `waveshare-esp32-s3-touch-amoled-1.8` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-amoled-1.8-v2` | `waveshare-esp32-s3-touch-amoled-1.8-v2` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-amoled-2.06` | `waveshare-esp32-s3-touch-amoled-2.06` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-amoled-2.16` | `waveshare-esp32-s3-touch-amoled-2.16` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-1.46` | `waveshare-esp32-s3-touch-lcd-1.46` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-1.54` | `waveshare-esp32-s3-touch-lcd-1.54` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-1.83` | `waveshare-esp32-s3-touch-lcd-1.83` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-1.85` | `waveshare-esp32-s3-touch-lcd-1.85` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-1.85b` | `waveshare-esp32-s3-touch-lcd-1.85b` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-1.85c` | `waveshare-esp32-s3-touch-lcd-1.85c` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-3.49` | `waveshare-esp32-s3-touch-lcd-3.49` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-3.5` | `waveshare-esp32-s3-touch-lcd-3.5` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-3.5b` | `waveshare-esp32-s3-touch-lcd-3.5b` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-4.3c` | `waveshare-esp32-s3-touch-lcd-4.3c` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-4b` | `waveshare-esp32-s3-touch-lcd-4b` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `waveshare/esp32-s3-touch-lcd-7c` | `waveshare-esp32-s3-touch-lcd-7c` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `xingzhi-abs-2.0` | `xingzhi-abs-2.0` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `xingzhi-cube-0.85tft-ml307` | `xingzhi-cube-0.85tft-ml307` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `xingzhi-cube-0.85tft-wifi` | `xingzhi-cube-0.85tft-wifi` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `xingzhi-cube-0.96oled-ml307` | `xingzhi-cube-0.96oled-ml307` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `xingzhi-cube-0.96oled-wifi` | `xingzhi-cube-0.96oled-wifi` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `xingzhi-cube-1.54tft-ml307` | `xingzhi-cube-1.54tft-ml307`<br>`xingzhi-cube-1.54tft-ml307-wechatui` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `xingzhi-cube-1.54tft-wifi` | `xingzhi-cube-1.54tft-wifi` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `xingzhi-metal-1.54-wifi` | `xingzhi-metal-1.54-wifi` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `yunliao-s3` | `yunliao-s3` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `zhengchen-1.54tft-ml307` | `zhengchen-1.54tft-ml307` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `zhengchen-1.54tft-wifi` | `zhengchen-1.54tft-wifi` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `zhengchen-cam` | `zhengchen-cam` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32s3` | `zhengchen-cam-ml307` | `zhengchen-cam-ml307` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32p4` | `espressif/esp32-p4-function-ev-board` | `espressif-esp32-p4-function-ev-board`<br>`espressif-esp32-p4x-function-ev-board` | ✅ Full builds passed | IDF 6.0.1 canonical release builds with ESP-SR 2.4.7 for Rev1 and Rev3 | Hardware smoke/peripheral regression pending |
| `esp32p4` | `m5stack-tab5` | `m5stack-tab5-p4x` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32p4` | `waveshare/esp32-p4-nano` | `waveshare-esp32-p4x-nano-10.1-a` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32p4` | `waveshare/esp32-p4-wifi6-touch-lcd` | `waveshare-esp32-p4x-wifi6-touch-lcd-4b`<br>`waveshare-esp32-p4x-wifi6-touch-lcd-4.3`<br>`waveshare-esp32-p4x-wifi6-touch-lcd-5`<br>`waveshare-esp32-p4x-wifi6-touch-lcd-7b`<br>`waveshare-esp32-p4x-wifi6-touch-lcd-3.4c`<br>`waveshare-esp32-p4x-wifi6-touch-lcd-4c`<br>`waveshare-esp32-p4x-wifi6-touch-lcd-7`<br>`waveshare-esp32-p4x-wifi6-touch-lcd-8`<br>`waveshare-esp32-p4x-wifi6-touch-lcd-10.1` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
| `esp32p4` | `waveshare/esp32-p4-wifi6-touch-lcd-3.5` | `waveshare-esp32-p4x-wifi6-touch-lcd-3.5` | ✅ Full build passed | GitHub Actions IDF 6.0.1 full build | Hardware smoke/peripheral regression pending |
## Next Steps and Acceptance Criteria
1. Run the IDF 6 BluFi CI job, then hardware-test at least one P4 v1.3 device on the legacy SDK path. When IDF 6-compatible releases of `touch_slider_sensor` and `touch_button_sensor` become available, re-enable and hardware-test the ESP Vocat PCB capacitive slider/button path.
2. For every green variant, complete a minimal hardware smoke test covering boot, networking, audio input/output, display/touch when present, camera when present, and 4G/Ethernet when present.
3. Perform a physical negative test for the `xmini-c3`/`xmini-c3-v3` firmware guard. CI proves that both images compile; it does not prove that a wrong image is safely rejected. Acceptance requires flashing each wrong image to a sacrificial or recoverable board and verifying that startup stops before any board-specific power or peripheral initialization can cause damage.
4. Keep third-party experiments under the ignored `components/` directory out of the migration branch. The reproducible baseline must use the published `78/esp_lcd_nv3023 1.0.1` and `wvirgil123/sscma_client 1.0.3` packages. `espfriends/servo_dog_ctrl 0.2.0` is an upstream registry dependency and needs no local override.
## Reproduction
```bash
source ~/.espressif/v6.0.1/esp-idf/export.sh
python scripts/build.py <board> --name <variant>
```
This document is a migration-status snapshot. Build compatibility and hardware compatibility are tracked separately. Add "hardware validated" to a board only after validation on physical hardware is complete.
+185
View File
@@ -0,0 +1,185 @@
# Dynamic Text Glyph Push Extension
This document defines version 1 of the `glyph_push` protocol extension. The extension lets a server
send bitmap glyphs that are missing from a device's installed text font. It applies equally to the
WebSocket and MQTT/UDP transports because capability advertisement and incoming JSON handling are
implemented in the shared protocol layer.
The extension supplements text rendering only. It does not change the text, TTS audio, or STT
semantics of the containing message.
## 1. Capability advertisement
The device advertises support in its client `hello` message:
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
}
}
```
`features.glyph_push` indicates support for this extension. A server must treat a missing or false
value as unsupported. The `v` field in each pushed payload carries the extension version.
The `text_font` object describes the exact font data installed on the device:
| Field | Type | Meaning |
|---|---|---|
| `bundle` | string | Explicit font bundle identifier. It changes when glyph metrics, rendering behavior, character sets, or wire compatibility change. |
| `charset` | string | Installed character set. Version 1 devices report `basic` or `common`. |
| `size` | number | Text font pixel profile used by the firmware. |
| `bpp` | number | Bits per pixel of the text font bitmap, currently `1` or `4`. |
`basic` is the font linked into the firmware. The standard XiaoZhi assets report `common` after
loading their common font from the assets partition. The server must use the values from each
device's hello message rather than inferring them from the board model.
An OTA assets package may replace the text font with a different size, bpp, character set, or font
family. The firmware still loads any structurally valid CBIN font. When the package also provides
complete `text_font_meta` fields, the device advertises those active runtime values and validates
glyph pushes against them. A legacy or custom package without compatible glyph metadata continues
to use its custom font, emoji, colors, and background, but advertises `glyph_push: false` and omits
`text_font`. This prevents incompatible fallback glyphs without restricting theme customization.
## 2. Server glyph payload
The server may attach a `glyph_push` object to either of these server-to-device messages:
- a TTS message with `"type": "tts"` and `"state": "sentence_start"`;
- an STT message with `"type": "stt"`.
Example:
```json
{
"type": "tts",
"state": "sentence_start",
"text": "𠮷野家",
"glyph_push": {
"v": 1,
"bundle": "noto-v1",
"size": 20,
"bpp": 4,
"glyphs": [
{
"codepoint": 134071,
"adv_w": 320,
"box_w": 20,
"box_h": 20,
"ofs_x": 0,
"ofs_y": 0,
"bitmap": "<base64-encoded bitmap>"
}
]
}
}
```
The payload header must match the device capability exactly:
| Field | Requirement |
|---|---|
| `v` | Must be `1`. |
| `bundle` | Must equal `text_font.bundle`. |
| `size` | Must equal `text_font.size`. |
| `bpp` | Must equal `text_font.bpp`. |
| `glyphs` | The partial glyph batch being pushed, containing at most 64 entries. |
Each item uses the LVGL native bitmap-font metrics:
| Field | Meaning |
|---|---|
| `codepoint` | Unicode code point from `1` through `0x10FFFF`. |
| `adv_w` | Horizontal advance in LVGL fixed-point units with four fractional bits (one pixel is 16 units). |
| `box_w`, `box_h` | Bitmap dimensions in pixels. Each dimension must be from 0 through 64. |
| `ofs_x`, `ofs_y` | Signed 16-bit glyph offsets relative to the text baseline and cursor position. |
| `bitmap` | Base64 encoding of the uncompressed LVGL plain bitmap. |
The decoded bitmap length must be exactly:
```text
ceil(box_w * box_h * bpp / 8)
```
The bitmap must use the same plain, zero-stride layout as the matching Noto full-bundle CBIN font.
Servers should extract and forward the bitmap and metrics directly from that CBIN profile instead of
rasterizing an unrelated font at request time.
The sum of decoded bitmap lengths in one payload must not exceed 64 KiB. If any header, glyph, or
bitmap is invalid, the device rejects the entire glyph payload but still displays the message text
using its installed fonts. A PSRAM device may also use fallback glyphs cached by earlier messages.
## 3. Server selection algorithm
For every connection that advertises `glyph_push: true`, the server should:
1. Resolve the full font bundle identified by `text_font.bundle`.
2. Select the CBIN profile matching `text_font.size` and `text_font.bpp`.
3. Decode the message text into Unicode code points.
4. Remove control characters, duplicates, and code points already present in
`text_font.charset`.
5. Extract the remaining glyphs from the full bundle.
6. Apply the per-message limits and attach one `glyph_push` object to the text message.
7. Omit `glyph_push` when no missing glyph is available.
The installed text font is searched before the dynamic fallback font. Pushed glyphs therefore fill
missing code points; they do not override glyphs in `basic` or `common`.
The full font bundle may be shared by all device connections in a server process. Per-connection
work is limited to using the capability tuple `(bundle, charset, size, bpp)` to select which glyphs
are missing and which profile to read.
## 4. Device cache behavior
All glyphs in one message are inserted first, followed by a single fallback-font rebuild. The device
never rebuilds the font once per glyph.
On a device with initialized PSRAM:
- bitmap, cmap, descriptor, and cache-entry storage is allocated in PSRAM;
- glyphs are retained across messages;
- the cache holds at most 256 glyphs and 64 KiB of decoded bitmap data;
- the least recently inserted or updated entries are evicted when a limit is exceeded.
On a device without PSRAM:
- storage uses internal RAM;
- only the current message's glyph batch is retained;
- the next text message replaces or clears the previous batch.
This distinction does not affect the protocol. A server can send the glyphs needed by each message
without knowing whether the device has PSRAM.
## 5. Compatibility and versioning
The server must not send glyphs when any of these conditions is true:
- `features.glyph_push` is absent or not supported by the server;
- the server does not have the advertised bundle;
- no full-font profile matches the advertised size and bpp;
- the glyph data cannot satisfy the version 1 validation rules.
Fallback is automatic: messages without `glyph_push`, and messages whose glyph payload is rejected,
are still processed normally.
When the font generator changes metrics, bitmap layout, source fonts, character sets, or rendering
behavior, publish a new explicit bundle identifier. Do not serve glyphs from one bundle under
another bundle's identifier even if their size and bpp happen to match.
## 6. Security requirements
Glyph payloads are untrusted network input. Implementations must validate the complete payload before
mutating a live font, bound both item count and decoded size, verify base64 decoded length, and reject
invalid code points or metrics. Servers should also bound their own per-message work and avoid
sending glyphs already covered by the advertised charset.
+174
View File
@@ -0,0 +1,174 @@
# 动态文字 Glyph Push 扩展
本文档定义 `glyph_push` 协议扩展的版本 1。服务器可以通过该扩展向设备下发本地文字
字库中缺失的位图 glyph。能力声明和 JSON 消息处理位于公共协议层,因此 WebSocket 与
MQTT/UDP 使用完全相同的扩展格式。
该扩展只补充文字渲染能力,不改变消息中的文本、TTS 音频或 STT 语义。
## 1. 能力声明
设备在客户端 `hello` 消息中声明能力:
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
}
}
```
`features.glyph_push` 表示设备支持该扩展。字段缺失或为 false 时,服务器必须视为设备不支持。
每次推送 payload 中的 `v` 字段负责表达扩展版本。
`text_font` 描述设备实际安装的文字字库:
| 字段 | 类型 | 含义 |
|---|---|---|
| `bundle` | string | 显式的字体 bundle 标识。glyph 度量、渲染方式、字符集或线格式变化时需要更换。 |
| `charset` | string | 已安装字符集。版本 1 设备报告 `basic``common`。 |
| `size` | number | 固件使用的文字字体像素规格。 |
| `bpp` | number | 字体位图的每像素位数,目前为 `1``4`。 |
`basic` 是链接进固件的字库。标准小智 assets 从分区加载 common 字库后会报告 `common`
服务器必须使用每个连接在 hello 中报告的实际值,不能根据板型推测。
OTA assets 可以把文字字体替换为不同字号、bpp、字符集或字体家族的 CBIN 字体。固件仍会
加载任何结构有效的 CBIN 字体;如果 assets 同时提供完整的 `text_font_meta`,设备会声明当前
实际使用的运行时字体参数,并据此校验 glyph push。旧版或自定义 assets 缺少兼容的 glyph
metadata 时,自定义字体、表情、颜色和背景仍然正常使用,但设备会声明 `glyph_push: false`
并省略 `text_font`,从而只禁止不兼容的 fallback glyph,不限制主题定制能力。
## 2. 服务器下发格式
服务器可以在以下消息中附加 `glyph_push` 对象:
- `"type": "tts"``"state": "sentence_start"` 的 TTS 消息;
- `"type": "stt"` 的 STT 消息。
示例:
```json
{
"type": "tts",
"state": "sentence_start",
"text": "𠮷野家",
"glyph_push": {
"v": 1,
"bundle": "noto-v1",
"size": 20,
"bpp": 4,
"glyphs": [
{
"codepoint": 134071,
"adv_w": 320,
"box_w": 20,
"box_h": 20,
"ofs_x": 0,
"ofs_y": 0,
"bitmap": "<base64 编码的位图>"
}
]
}
}
```
payload 头必须与设备能力完全匹配:
| 字段 | 要求 |
|---|---|
| `v` | 必须为 `1`。 |
| `bundle` | 必须等于 `text_font.bundle`。 |
| `size` | 必须等于 `text_font.size`。 |
| `bpp` | 必须等于 `text_font.bpp`。 |
| `glyphs` | 本次增量推送的 glyph 数组,最多 64 项。 |
每个 item 使用 LVGL 原生位图字体度量:
| 字段 | 含义 |
|---|---|
| `codepoint` | `1``0x10FFFF` 的 Unicode code point。 |
| `adv_w` | 带 4 位小数的 LVGL 定点水平 advance16 个单位等于 1 像素。 |
| `box_w``box_h` | 位图宽高,单项范围为 0 到 64 像素。 |
| `ofs_x``ofs_y` | 相对于文字基线和光标位置的有符号 16 位偏移。 |
| `bitmap` | 未压缩 LVGL plain 位图的 Base64 编码。 |
解码后的位图长度必须严格等于:
```text
ceil(box_w * box_h * bpp / 8)
```
位图必须与对应 Noto full bundle CBIN 字体使用相同的 plain、零 stride 布局。服务器应直接
提取相同 profile 的 CBIN 位图和度量,不应临时使用其他字体重新光栅化。
单条 payload 的解码位图总长度不得超过 64 KiB。任意头字段、glyph 或位图无效时,设备会
拒绝整批 glyph,但仍使用本地字体显示消息文本;有 PSRAM 的设备还可能使用此前缓存的
fallback glyph。
## 3. 服务器选择流程
对于声明 `glyph_push: true` 的每个连接,服务器应:
1. 根据 `text_font.bundle` 找到对应 full 字体 bundle。
2. 根据 `text_font.size``text_font.bpp` 选择 CBIN profile。
3. 将消息文本解码为 Unicode code point。
4. 去掉控制字符、重复字符及 `text_font.charset` 已包含的字符。
5. 从 full bundle 提取剩余 glyph。
6. 执行单消息限制,并将一个 `glyph_push` 对象附加到文字消息。
7. 没有可用的缺失 glyph 时省略 `glyph_push`
设备先查询本地文字字体,再查询动态 fallback。因此 glyph push 只补充缺字,不会覆盖
`basic``common` 中已有的 glyph。
同一服务器进程中的所有设备连接可以共享 full 字体 bundle 服务。每个连接只需根据
`(bundle, charset, size, bpp)` 能力组合判断缺字并选择 profile。
## 4. 设备缓存行为
同一消息中的全部 glyph 会先加入缓存,随后只执行一次 fallback 字体 rebuild,不会每加入
一个 glyph 就 rebuild 一次。
设备有已初始化的 PSRAM 时:
- bitmap、cmap、descriptor 和缓存条目存放在 PSRAM
- glyph 跨消息保留;
- 缓存上限为 256 个 glyph 和 64 KiB 解码位图;
- 超限时淘汰最早插入或更新的条目。
设备没有 PSRAM 时:
- 数据使用内部 RAM
- 只保留当前消息的 glyph 批次;
- 下一条文字消息会替换或清空上一批 glyph。
该差异不改变协议。服务器无需知道设备是否有 PSRAM,可以为每条消息发送它所需的 glyph。
## 5. 兼容与版本管理
以下任一条件成立时,服务器不得发送 glyph:
- `features.glyph_push` 缺失、为 false,或服务器不支持该扩展;
- 服务器没有设备声明的 bundle;
- 找不到匹配 size 和 bpp 的 full 字体 profile
- glyph 数据不满足版本 1 的校验要求。
兼容降级是自动的:没有 `glyph_push`,或 glyph payload 被拒绝时,设备仍会正常处理消息。
字体生成器的度量、位图布局、源字体、字符集或渲染方式变化时,应发布新的显式 bundle
标识。即使 size 和 bpp 相同,也不能用一个 bundle 的标识下发另一个 bundle 的 glyph。
## 6. 安全要求
glyph payload 是不可信网络输入。实现必须先完整校验 payload,再修改正在使用的字体;限制
item 数量和解码总长度;验证 Base64 解码长度;拒绝非法 code point 或度量。服务器也应限制
单消息处理量,避免重复发送设备声明字符集已经包含的 glyph。
+10 -1
View File
@@ -80,7 +80,14 @@ The device connects to the broker using:
"transport": "udp",
"features": {
"mcp": true,
"aec": true
"aec": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
},
"audio_params": {
"format": "opus",
@@ -92,6 +99,8 @@ The device connects to the broker using:
```
`features.mcp` is always set; `features.aec` is set when `CONFIG_USE_SERVER_AEC` is enabled.
`features.glyph_push` and `text_font` advertise the shared dynamic text-glyph extension described in
[Dynamic Text Glyph Push Extension](glyph-push.md).
#### 3.2.2 Server -> Device
+12 -2
View File
@@ -78,7 +78,14 @@ sequenceDiagram
"version": 3,
"transport": "udp",
"features": {
"mcp": true
"mcp": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
},
"audio_params": {
"format": "opus",
@@ -89,6 +96,9 @@ sequenceDiagram
}
```
`features.glyph_push``text_font` 声明 WebSocket 与 MQTT/UDP 共用的动态文字扩展,详见
[动态文字 Glyph Push 扩展](glyph-push_zh.md)。
#### 3.2.2 服务器响应 Hello
```json
@@ -390,4 +400,4 @@ MQTT + UDP 混合协议通过以下设计实现高效的音视频通信:
- **自动恢复**:支持连接断开后的自动重连
- **性能优化**:UDP 传输保证音频数据的实时性
该协议适用于对实时性要求较高的语音交互场景,但需要在网络复杂度和传输性能之间做出权衡。
该协议适用于对实时性要求较高的语音交互场景,但需要在网络复杂度和传输性能之间做出权衡。
+9 -1
View File
@@ -27,7 +27,14 @@ This document describes the WebSocket communication protocol between the device
"version": 1,
"features": {
"mcp": true,
"aec": true
"aec": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
},
"transport": "websocket",
"audio_params": {
@@ -39,6 +46,7 @@ This document describes the WebSocket communication protocol between the device
}
```
- `features` is optional and generated from compile-time configuration. For example, `"mcp": true` means the device supports MCP, and `"aec": true` is emitted when `CONFIG_USE_SERVER_AEC` is enabled.
- `"glyph_push": true` and `text_font` advertise the optional dynamic text-glyph extension. See [Dynamic Text Glyph Push Extension](glyph-push.md).
- `frame_duration` matches `OPUS_FRAME_DURATION_MS` (typically 60 ms).
4. **Server replies with "hello"**
+9 -1
View File
@@ -26,7 +26,14 @@
"type": "hello",
"version": 1,
"features": {
"mcp": true
"mcp": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
},
"transport": "websocket",
"audio_params": {
@@ -38,6 +45,7 @@
}
```
- 其中 `features` 字段为可选,内容根据设备编译配置自动生成。例如:`"mcp": true` 表示支持 MCP 协议。
- `"glyph_push": true` 和 `text_font` 声明可选的动态文字 glyph 扩展,详见[动态文字 Glyph Push 扩展](glyph-push_zh.md)。
- `frame_duration` 的值对应 `OPUS_FRAME_DURATION_MS`(例如 60ms)。
4. **服务器回复 "hello"**
+674 -665
View File
File diff suppressed because it is too large Load Diff
+395 -410
View File
@@ -118,25 +118,41 @@ choice
endchoice
choice BOARD_TYPE
prompt "Board Type"
default BOARD_TYPE_BREAD_COMPACT_WIFI
prompt "Target Board"
default BOARD_TYPE_BREAD_COMPACT_ESP32 if IDF_TARGET_ESP32
default BOARD_TYPE_XMINI_C3_V3 if IDF_TARGET_ESP32C3
default BOARD_TYPE_ESP_SENSAIRSHUTTLE if IDF_TARGET_ESP32C5
default BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_AMOLED_2_06 if IDF_TARGET_ESP32C6
default BOARD_TYPE_BREAD_COMPACT_WIFI if IDF_TARGET_ESP32S3
default BOARD_TYPE_ESP32_P4_FUNCTION_EV_BOARD if IDF_TARGET_ESP32P4
default BOARD_TYPE_ESP32_S31_FUNCTION_COREBOARD_1 if IDF_TARGET_ESP32S31
help
Board type. 开发板类型
Select the physical board hardware implementation.
The OTA board type and firmware variant name are configured separately.
config BOARD_TYPE_BREAD_COMPACT_WIFI
bool "Bread Compact WiFi (面包板)"
config BOARD_TYPE_ATK_DNESP32S3
bool "Alientek ATK-DNESP32S3 Development Board (正点原子)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_WIFI_LCD
bool "Bread Compact WiFi + LCD (面包板)"
config BOARD_TYPE_ATK_DNESP32S3_BOX
bool "Alientek ATK-DNESP32S3-BOX (正点原子)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_WIFI_CAM
bool "Bread Compact WiFi + LCD + Camera (面包板)"
config BOARD_TYPE_ATK_DNESP32S3_BOX0
bool "Alientek ATK-DNESP32S3-BOX0 (正点原子)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_ML307
bool "Bread Compact ML307/EC801E (面包板 4G)"
config BOARD_TYPE_ATK_DNESP32S3_BOX2_4G
bool "Alientek ATK-DNESP32S3-BOX2 4G (正点原子)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_NT26
bool "Bread Compact NT26 (面包板 4G)"
config BOARD_TYPE_ATK_DNESP32S3_BOX2_WIFI
bool "Alientek ATK-DNESP32S3-BOX2 Wi-Fi (正点原子)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX3
bool "Alientek ATK-DNESP32S3-BOX3 (正点原子)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3M_4G
bool "Alientek ATK-DNESP32S3M 4G (正点原子)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3M_WIFI
bool "Alientek ATK-DNESP32S3M Wi-Fi (正点原子)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_ESP32
bool "Bread Compact ESP32 DevKit (面包板)"
@@ -144,50 +160,92 @@ choice BOARD_TYPE
config BOARD_TYPE_BREAD_COMPACT_ESP32_LCD
bool "Bread Compact ESP32 DevKit + LCD (面包板)"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_XMINI_C3_V3
bool "Xmini C3 V3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_XMINI_C3_4G
bool "Xmini C3 4G"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_XMINI_C3
bool "Xmini C3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_ESP_KORVO2_V3
bool "Espressif Korvo2 V3"
config BOARD_TYPE_BREAD_COMPACT_ML307
bool "Bread Compact ML307/EC801E 4G (面包板)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_KORVO2_V3_RNDIS
bool "Espressif Korvo2 V3 RNDIS"
config BOARD_TYPE_BREAD_COMPACT_NT26
bool "Bread Compact NT26 4G (面包板)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_WIFI
bool "Bread Compact Wi-Fi (面包板)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_WIFI_LCD
bool "Bread Compact Wi-Fi + LCD (面包板)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_WIFI_CAM
bool "Bread Compact Wi-Fi + LCD + Camera (面包板)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DF_S3_AI_CAM
bool "DFRobot ESP32-S3 AI Camera Module"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DF_K10
bool "DFRobot UNIHIKER K10 (行空板 K10)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DOIT_S3_AIBOX
bool "DOIT S3 AIBox (四博智联 AI 陪伴盒子)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DU_CHATX
bool "Du ChatX Wi-Fi (嘟嘟开发板)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ELECTRON_BOT
bool "ElectronBot"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_P4_FUNCTION_EV_BOARD
bool "Espressif ESP32-P4/P4X-Function-EV-Board"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_ESP32_S3_BOX
bool "Espressif ESP32-S3-BOX"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_S3_BOX_3
bool "Espressif ESP32-S3-BOX-3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_S3_BOX_LITE
bool "Espressif ESP32-S3-BOX-Lite"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_S3_KORVO_2_V3
bool "Espressif ESP32-S3-Korvo-2 V3.0"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_S3_KORVO_2_V3_RNDIS
bool "Espressif ESP32-S3-Korvo-2 V3.0 RNDIS"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_S3_LCD_EV_BOARD
bool "Espressif ESP32-S3-LCD-EV-Board"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_S3_LCD_EV_BOARD_2
bool "Espressif ESP32-S3-LCD-EV-Board-2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_S31_FUNCTION_COREBOARD_1
bool "Espressif ESP32-S31-Function-CoreBoard-1"
depends on IDF_TARGET_ESP32S31
config BOARD_TYPE_ESP_HI
bool "Espressif ESP-HI"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_ESP_SENSAIRSHUTTLE
bool "Espressif ESP-SensairShuttle"
depends on IDF_TARGET_ESP32C5
config BOARD_TYPE_ESP_VOCAT
bool "Espressif ESP-VoCat"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_SPARKBOT
bool "Espressif SparkBot"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_SENSAIRSHUTTLE
bool "Espressif ESP-SensairShuttle"
config BOARD_TYPE_ESP_SPOT_C5
bool "Espressif Spot-C5"
depends on IDF_TARGET_ESP32C5
config BOARD_TYPE_ESP_SPOT_S3
bool "Espressif Spot-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_SPOT_C5
bool "Espressif Spot-C5"
depends on IDF_TARGET_ESP32C5
config BOARD_TYPE_ESP_HI
bool "Espressif ESP-HI"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_ESP_BOX_3
bool "Espressif ESP-BOX-3"
config BOARD_TYPE_Freenove_ESP32S3_DISPLAY_2_8_LCD
bool "Freenove ESP32-S3 Display 2.8-inch LCD"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_BOX
bool "Espressif ESP-BOX"
config BOARD_TYPE_GENJUTECH_S3_1_54TFT
bool "GenjuTech ESP32-S3 1.54-inch TFT (亘具科技)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_BOX_LITE
bool "Espressif ESP-BOX-Lite"
config BOARD_TYPE_HU_087
bool "HU-087"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_P4_FUNCTION_EV_BOARD
bool "Espressif ESP-P4-Function-EV-Board"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_ESP_VOCAT
bool "Espressif ESP-VoCat"
config BOARD_TYPE_JIUCHUAN
bool "Jiuchuan ESP32-S3 (九川智能)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_KEVIN_BOX_2
bool "Kevin Box 2"
@@ -202,65 +260,53 @@ choice BOARD_TYPE
bool "Kevin SP V4"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_KEVIN_YUYING_313LCD
bool "鱼鹰科技 3.13LCD"
bool "Kevin Yuying 3.13-inch LCD (鱼鹰科技)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_CGC
bool "CGC"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_CGC_144
bool "CGC 144"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_LICHUANG_DEV_S3
bool "立创·实战派 ESP32-S3"
config BOARD_TYPE_LABPLUS_LEDONG_V2
bool "Labplus Ledong V2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_QUANDONG_S3_DEV
bool "全动 ESP32-S3 开发板"
config BOARD_TYPE_LABPLUS_MPYTHON_V3
bool "Labplus mPython V3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LICHUANG_DEV_C3
bool "立创·实战派 ESP32-C3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_RYMCU_BIGSMART
bool "RYMCU BigSmart"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_EDA_TV_PRO
bool "EDA课程案例 EDA-TV-Pro"
config BOARD_TYPE_LCDWIKI_ES3C35P
bool "LCDWiki ES3C35P 3.5-inch ESP32-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_EDA_ROBOT_PRO
bool "EDA课程案例 EDA-Robot-Pro"
bool "LCEDA Course Example EDA-Robot-Pro (EDA 课程案例)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_EDA_SUPER_BEAR
bool "EDA课程案例 EDA-Super-Bear"
bool "LCEDA Course Example EDA-Super-Bear (EDA 课程案例)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DF_K10
bool "DFRobot 行空板 k10"
config BOARD_TYPE_EDA_TV_PRO
bool "LCEDA Course Example EDA-TV-Pro (EDA 课程案例)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DF_S3_AI_CAM
bool "DFRobot ESP32-S3 AI智能摄像头模块"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MAGICLICK_S3_2P4
bool "神奇按钮 Magiclick_2.4"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MAGICLICK_S3_2P5
bool "神奇按钮 Magiclick_2.5"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MAGICLICK_C3
bool "神奇按钮 Magiclick_C3"
config BOARD_TYPE_LICHUANG_DEV_C3
bool "LCKFB SZPI ESP32-C3 (立创·实战派)"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_MAGICLICK_C3_V2
bool "神奇按钮 Magiclick_C3_v2"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_M5STACK_STOPWATCH
bool "M5Stack StopWatch"
config BOARD_TYPE_LICHUANG_DEV_S3
bool "LCKFB SZPI ESP32-S3 (立创·实战派)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_CORE_S3
bool "M5Stack CoreS3"
config BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_0_V1_1
bool "LILYGO T-CameraPlus-S3 V1.0/V1.1"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_STICK_S3
bool "M5Stack StickS3"
config BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_2
bool "LILYGO T-CameraPlus-S3 V1.2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_CORE_TAB5
bool "M5Stack Tab5"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_LILYGO_T_CIRCLE_S3
bool "LILYGO T-Circle-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA
bool "LILYGO T-Display-S3-Pro MVSR LoRa"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA_NO_BATTERY
bool "LILYGO T-Display-S3-Pro MVSR LoRa (No Battery)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_ATOM_ECHOS3R
bool "M5Stack AtomEchoS3R"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_ATOM_MATRIX_ECHO_BASE
bool "M5Stack AtomMatrix + Echo Base"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_M5STACK_ATOM_S3_ECHO_BASE
bool "M5Stack AtomS3 + Echo Base"
depends on IDF_TARGET_ESP32S3
@@ -273,47 +319,191 @@ choice BOARD_TYPE
config BOARD_TYPE_M5STACK_ATOM_S3R_CAM_M12_ECHO_BASE
bool "M5Stack AtomS3R CAM/M12 + Echo Base"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_ATOM_ECHOS3R
bool "M5Stack AtomEchoS3R"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_CARDPUTER_ADV
bool "M5Stack Cardputer Adv"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_ATOM_MATRIX_ECHO_BASE
bool "M5Stack AtomMatrix + Echo Base"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_WAVESHARE_ESP32_TOUCH_LCD_3_5
bool "Waveshare ESP32-Touch-LCD-3.5"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_WAVESHARE_ESP32_S3_CAM_XXXX
bool "Waveshare ESP32-S3-CAM"
config BOARD_TYPE_M5STACK_CORE_S3
bool "M5Stack CoreS3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_AUDIO_BOARD
bool "Waveshare ESP32-S3-Audio-Board"
config BOARD_TYPE_M5STACK_STICK_S3
bool "M5Stack StickS3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_8
bool "Waveshare ESP32-S3-Touch-AMOLED-1.8"
config BOARD_TYPE_M5STACK_STOPWATCH
bool "M5Stack Stopwatch"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_8_v2
bool "Waveshare ESP32-S3-Touch-AMOLED-1.8-v2"
config BOARD_TYPE_M5STACK_CORE_TAB5
bool "M5Stack Tab5"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_MAGICLICK_S3_2P4
bool "Magiclick 2.4 (神奇按钮)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_43C
bool "Waveshare ESP32-S3-Touch-AMOLED-1.43C"
config BOARD_TYPE_MAGICLICK_S3_2P5
bool "Magiclick 2.5 (神奇按钮)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_2_06
bool "Waveshare ESP32-S3-Touch-AMOLED-2.06"
config BOARD_TYPE_MAGICLICK_C3
bool "Magiclick C3 (神奇按钮)"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_MAGICLICK_C3_V2
bool "Magiclick C3 V2 (神奇按钮)"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_MINSI_K08_DUAL
bool "Minsi K08 Dual Network (敏思科技)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MIXGO_NOVA
bool "MixGo Nova (元控·青春)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MOVECALL_CUICAN_ESP32S3
bool "Movecall CuiCan (璀璨 AI 吊坠)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MOVECALL_MOJI_ESP32S3
bool "Movecall Moji"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MOVECALL_MOJI2_ESP32C5
bool "Movecall Moji 2.0"
depends on IDF_TARGET_ESP32C5
config BOARD_TYPE_XINGZHI_ABS_2_0
bool "Nologo Xingzhi ABS 2.0 (无名科技星智)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_85TFT_ML307
bool "Nologo Xingzhi Cube 0.85-inch TFT ML307 (无名科技星智)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_85TFT_WIFI
bool "Nologo Xingzhi Cube 0.85-inch TFT Wi-Fi (无名科技星智)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_96OLED_ML307
bool "Nologo Xingzhi Cube 0.96-inch OLED ML307 (无名科技星智)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_96OLED_WIFI
bool "Nologo Xingzhi Cube 0.96-inch OLED Wi-Fi (无名科技星智)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_1_54TFT_ML307
bool "Nologo Xingzhi Cube 1.54-inch TFT ML307 (无名科技星智)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_1_54TFT_WIFI
bool "Nologo Xingzhi Cube 1.54-inch TFT Wi-Fi (无名科技星智)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_METAL_1_54_WIFI
bool "Nologo Xingzhi Metal 1.54-inch Wi-Fi (无名科技星智)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_AI_VOX3
bool "NullLab AI Vox V3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_OTTO_ROBOT
bool "Otto Robot"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_QUANDONG_S3_DEV
bool "Quandong ESP32-S3 Development Board (全动)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_RYMCU_BIGSMART
bool "RYMCU BigSmart"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SEEED_STUDIO_SENSECAP_WATCHER
bool "Seeed Studio SenseCAP Watcher"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SPOTPEAR_ESP32_S3_1_28_BOX
bool "Spotpear ESP32-S3 1.28 Box"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SPOTPEAR_ESP32_S3_1_54_MUMA
bool "Spotpear ESP32-S3 1.54 MUMA"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SURFER_C3_1_14TFT
bool "Surfer C3 1.14-inch TFT"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_TAIJI_PI_S3
bool "Taiji Pi ESP32-S3 (太极小派)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_C5_TOUCH_LCD_1_69
bool "Waveshare ESP32-C5-Touch-LCD-1.69"
depends on IDF_TARGET_ESP32C5
config BOARD_TYPE_WAVESHARE_ESP32_C6_LCD_0_85
bool "Waveshare ESP32-C6-LCD-0.85"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_LCD_1_69
bool "Waveshare ESP32-C6-LCD-1.69"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_AMOLED_1_32
bool "Waveshare ESP32-C6-Touch-AMOLED-1.32"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_AMOLED_1_43
bool "Waveshare ESP32-C6-Touch-AMOLED-1.43"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_AMOLED_1_8
bool "Waveshare ESP32-C6-Touch-AMOLED-1.8"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_AMOLED_2_06
bool "Waveshare ESP32-C6-Touch-AMOLED-2.06"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_AMOLED_2_16
bool "Waveshare ESP32-C6-Touch-AMOLED-2.16"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_2_16
bool "Waveshare ESP32-S3-Touch-AMOLED-2.16"
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_LCD_1_54
bool "Waveshare ESP32-C6-Touch-LCD-1.54"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_LCD_1_83
bool "Waveshare ESP32-C6-Touch-LCD-1.83"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_3_4C
bool "Waveshare ESP32-P4 Wi-Fi 6 Touch LCD 3.4C"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_3_5
bool "Waveshare ESP32-P4 Wi-Fi 6 Touch LCD 3.5"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_4B
bool "Waveshare ESP32-P4 Wi-Fi 6 Touch LCD 4B"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_4C
bool "Waveshare ESP32-P4 Wi-Fi 6 Touch LCD 4C"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_4_3
bool "Waveshare ESP32-P4 Wi-Fi 6 Touch LCD 4.3"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_5
bool "Waveshare ESP32-P4 Wi-Fi 6 Touch LCD 5"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_7
bool "Waveshare ESP32-P4 Wi-Fi 6 Touch LCD 7"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_7B
bool "Waveshare ESP32-P4 Wi-Fi 6 Touch LCD 7B"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_8
bool "Waveshare ESP32-P4 Wi-Fi 6 Touch LCD 8"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_10_1
bool "Waveshare ESP32-P4 Wi-Fi 6 Touch LCD 10.1"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_NANO
bool "Waveshare ESP32-P4-Nano"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_S3_AUDIO_BOARD
bool "Waveshare ESP32-S3 Audio Board"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_CAM_XXXX
bool "Waveshare ESP32-S3-CAM"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_ePaper_1_54_v1
bool "Waveshare ESP32-S3-e-Paper-1.54 V1"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_ePaper_1_54_v2
bool "Waveshare ESP32-S3-e-Paper-1.54 V2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_ePaper_3_97
bool "Waveshare ESP32-S3-e-Paper-3.97"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_LCD_0_85
bool "Waveshare ESP32-S3-LCD-0.85"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_RGB_MATRIX
bool "Waveshare ESP32-S3-RGB-Matrix"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_RLCD_4_2
bool "Waveshare ESP32-S3-RLCD-4.2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_32
bool "Waveshare ESP32-S3-Touch-AMOLED-1.32"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_43C
bool "Waveshare ESP32-S3-Touch-AMOLED-1.43C"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_75
bool "Waveshare ESP32-S3-Touch-AMOLED-1.75"
@@ -321,11 +511,47 @@ choice BOARD_TYPE
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_75C
bool "Waveshare ESP32-S3-Touch-AMOLED-1.75C"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_8
bool "Waveshare ESP32-S3-Touch-AMOLED-1.8"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_8_v2
bool "Waveshare ESP32-S3-Touch-AMOLED-1.8 V2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_2_06
bool "Waveshare ESP32-S3-Touch-AMOLED-2.06"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_2_16
bool "Waveshare ESP32-S3-Touch-AMOLED-2.16"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_46
bool "Waveshare ESP32-S3-Touch-LCD-1.46"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_54
bool "Waveshare ESP32-S3-Touch-LCD-1.54"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_83
bool "Waveshare ESP32-S3-Touch-LCD-1.83"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_RGB_MATRIX
bool "Waveshare ESP32-S3-RGB-Matrix"
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_85
bool "Waveshare ESP32-S3-Touch-LCD-1.85"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_85B
bool "Waveshare ESP32-S3-Touch-LCD-1.85B"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_85C
bool "Waveshare ESP32-S3-Touch-LCD-1.85C"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_3_49
bool "Waveshare ESP32-S3-Touch-LCD-3.49"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_3_49_V2
bool "Waveshare ESP32-S3-Touch-LCD-3.49 V2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_3_5
bool "Waveshare ESP32-S3-Touch-LCD-3.5"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_3_5B
bool "Waveshare ESP32-S3-Touch-LCD-3.5B"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_4B
bool "Waveshare ESP32-S3-Touch-LCD-4B"
@@ -336,257 +562,41 @@ choice BOARD_TYPE
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_7C
bool "Waveshare ESP32-S3-Touch-LCD-7C"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_85C
bool "Waveshare ESP32-S3-Touch-LCD-1.85C"
config BOARD_TYPE_WAVESHARE_ESP32_TOUCH_LCD_3_5
bool "Waveshare ESP32-Touch-LCD-3.5"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_WDMOMO_CGC
bool "WDMomo ESP32-CGC"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_WDMOMO_CGC_144
bool "WDMomo ESP32-CGC-144"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_XMINI_C3
bool "Xmini C3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_XMINI_C3_4G
bool "Xmini C3 4G"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_XMINI_C3_V3
bool "Xmini C3 V3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_XORIGIN_AIPI_LITE
bool "XOrigin AiPi-Lite"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_85
bool "Waveshare ESP32-S3-Touch-LCD-1.85"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_46
bool "Waveshare ESP32-S3-Touch-LCD-1.46"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_C6_LCD_1_69
bool "Waveshare ESP32-C6-LCD-1.69"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_LCD_1_83
bool "Waveshare ESP32-C6-Touch-LCD-1.83"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_AMOLED_1_43
bool "Waveshare ESP32-C6-Touch-AMOLOED-1.43"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_AMOLED_1_32
bool "Waveshare ESP32-C6-Touch-AMOLOED-1.32"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_AMOLED_1_8
bool "Waveshare ESP32-C6-Touch-AMOLED-1.8"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_TOUCH_LCD_1_54
bool "Waveshare ESP32-C6-Touch-LCD-1.54"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_C6_LCD_0_85
bool "Waveshare ESP32-C6-LCD-0.85"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_32
bool "Waveshare ESP32-S3-Touch-AMOLOED-1.32"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_3_49
bool "Waveshare ESP32-S3-Touch-LCD-3.49"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_3_5
bool "Waveshare ESP32-S3-Touch-LCD-3.5"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_ePaper_1_54_v1
bool "Waveshare ESP32-S3-ePaper-1.54_v1"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_ePaper_1_54_v2
bool "Waveshare ESP32-S3-ePaper-1.54_v2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_ePaper_3_97
bool "Waveshare ESP32-S3-ePaper-3.97"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_RLCD_4_2
bool "Waveshare ESP32-S3-RLCD-4.2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_3_5B
bool "Waveshare ESP32-S3-Touch-LCD-3.5B"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_54
bool "Waveshare ESP32-S3-Touch-LCD-1.54"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_S3_LCD_0_85
bool "Waveshare ESP32-S3-LCD-0.85"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_ESP32_P4_NANO
bool "Waveshare ESP32-P4-NANO"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_4B
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-4B"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_4_3
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-4.3"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_5
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-5"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_7B
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-7B"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_3_4C
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_3_5
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-3.5"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_4C
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-4C"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_7
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-7"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_8
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-8"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_10_1
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-10.1"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_TUDOUZI
bool "土豆子"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_CIRCLE_S3
bool "LILYGO T-Circle-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_0_V1_1
bool "LILYGO T-CameraPlus-S3_V1_0_V1_1"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_2
bool "LILYGO T-CameraPlus-S3_V1_2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA
bool "LILYGO T-Display-S3-Pro-MVSRLora"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA_NO_BATTERY
bool "LILYGO T-Display-S3-Pro-MVSRLora_No_Battery"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_DISPLAY_P4
bool "LILYGO T-Display-P4"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_MOVECALL_MOJI_ESP32S3
bool "Movecall Moji 小智AI衍生版"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MOVECALL_MOJI2_ESP32C5
bool "Movecall Moji2.0 小智AI衍生版"
depends on IDF_TARGET_ESP32C5
config BOARD_TYPE_MOVECALL_CUICAN_ESP32S3
bool "Movecall CuiCan 璀璨·AI吊坠"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3
bool "正点原子DNESP32S3开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX
bool "正点原子DNESP32S3-BOX"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX0
bool "正点原子DNESP32S3-BOX0"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX2_WIFI
bool "正点原子DNESP32S3-BOX2-WIFI"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX2_4G
bool "正点原子DNESP32S3-BOX2-4G"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3M_WIFI
bool "正点原子DNESP32S3M-WIFI"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3M_4G
bool "正点原子DNESP32S3M-4G"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX3
bool "正点原子DNESP32S3-BOX3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DU_CHATX
bool "嘟嘟开发板CHATX(wifi)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_TAIJI_PI_S3
bool "太极小派esp32s3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_85TFT_WIFI
bool "无名科技星智0.85(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_85TFT_ML307
bool "无名科技星智0.85(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_96OLED_WIFI
bool "无名科技星智0.96(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_96OLED_ML307
bool "无名科技星智0.96(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_1_54TFT_WIFI
bool "无名科技星智1.54(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_1_54TFT_ML307
bool "无名科技星智1.54(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_METAL_1_54_WIFI
bool "无名科技星智1.54 METAL(wifi)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_ABS_2_0
bool "无名科技星智ABS 2.0"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SEEED_STUDIO_SENSECAP_WATCHER
bool "Seeed Studio SenseCAP Watcher"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DOIT_S3_AIBOX
bool "四博智联AI陪伴盒子"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MIXGO_NOVA
bool "元控·青春"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_GENJUTECH_S3_1_54TFT
bool "亘具科技1.54(s3)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_S3_LCD_EV_Board
bool "乐鑫ESP S3 LCD EV Board开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_S3_LCD_EV_Board_2
bool "乐鑫ESP S3 LCD EV Board 2开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ZHENGCHEN_1_54TFT_WIFI
bool "征辰科技1.54(WIFI)"
config BOARD_TYPE_YUNLIAO_S3
bool "Yunliao S3 (小智云聊)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ZHENGCHEN_1_54TFT_ML307
bool "征辰科技1.54(ML307)"
bool "Zhengchen 1.54-inch TFT ML307 (征辰科技)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ZHENGCHEN_CAM
bool "征辰科技AI Camera"
config BOARD_TYPE_ZHENGCHEN_1_54TFT_WIFI
bool "Zhengchen 1.54-inch TFT Wi-Fi (征辰科技)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ZHENGCHEN_CAM_ML307
bool "征辰科技AI Camera(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MINSI_K08_DUAL
bool "敏思科技K08(DUAL)"
bool "Zhengchen AI Camera ML307 (征辰科技)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SPOTPEAR_ESP32_S3_1_54_MUMA
bool "Spotpear ESP32-S3-1.54-MUMA"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SPOTPEAR_ESP32_S3_1_28_BOX
bool "Spotpear ESP32-S3-1.28-BOX"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_OTTO_ROBOT
bool "ottoRobot"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ELECTRON_BOT
bool "electronBot"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_JIUCHUAN
bool "九川智能"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LABPLUS_MPYTHON_V3
bool "labplus mpython_v3 board"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LABPLUS_LEDONG_V2
bool "labplus ledong_v2 board"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SURFER_C3_1_14TFT
bool "Surfer-C3-1.14TFT"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_YUNLIAO_S3
bool "小智云聊-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WIRELESS_TAG_WTP4C5MP07S
bool "Wireless-Tag WTP4C5MP07S"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_AIPI_LITE
bool "AIPI-Lite"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_HU_087
bool "HU-087"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_Freenove_ESP32S3_DISPLAY_2_8_LCD
bool "Freenove ESP32S3 Display 2.8 LCD"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_AI_VOX3
bool "NULLLAB-AI-VOX3"
config BOARD_TYPE_ZHENGCHEN_CAM
bool "Zhengchen AI Camera Wi-Fi (征辰科技)"
depends on IDF_TARGET_ESP32S3
endchoice
@@ -655,34 +665,14 @@ menu "Ethernet Configuration"
endmenu
choice
depends on BOARD_TYPE_LILYGO_T_DISPLAY_P4
prompt "Select the screen type"
default SCREEN_TYPE_HI8561
config SCREEN_TYPE_HI8561
bool "HI8561"
config SCREEN_TYPE_RM69A10
bool "RM69A10"
endchoice
choice
depends on BOARD_TYPE_LILYGO_T_DISPLAY_P4
prompt "Select the color format of the screen"
default SCREEN_PIXEL_FORMAT_RGB565
config SCREEN_PIXEL_FORMAT_RGB565
bool "RGB565"
config SCREEN_PIXEL_FORMAT_RGB888
bool "RGB888"
endchoice
choice ESP_S3_LCD_EV_Board_Version_TYPE
depends on BOARD_TYPE_ESP_S3_LCD_EV_Board
choice ESP32_S3_LCD_EV_BOARD_VERSION
depends on BOARD_TYPE_ESP32_S3_LCD_EV_BOARD
prompt "EV_BOARD Type"
default ESP_S3_LCD_EV_Board_1p4
config ESP_S3_LCD_EV_Board_1p4
bool "乐鑫ESP32_S3_LCD_EV_Board-MB_V1.4"
config ESP_S3_LCD_EV_Board_1p5
bool "乐鑫ESP32_S3_LCD_EV_Board-MB_V1.5"
default ESP32_S3_LCD_EV_BOARD_V1_4
config ESP32_S3_LCD_EV_BOARD_V1_4
bool "Espressif ESP32-S3-LCD-EV-Board-MB v1.4"
config ESP32_S3_LCD_EV_BOARD_V1_5
bool "Espressif ESP32-S3-LCD-EV-Board-MB v1.5"
endchoice
choice DISPLAY_OLED_TYPE
@@ -700,7 +690,7 @@ choice DISPLAY_OLED_TYPE
endchoice
choice DISPLAY_LCD_TYPE
depends on BOARD_TYPE_BREAD_COMPACT_WIFI_LCD || BOARD_TYPE_BREAD_COMPACT_ESP32_LCD || BOARD_TYPE_CGC || BOARD_TYPE_BREAD_COMPACT_WIFI_CAM
depends on BOARD_TYPE_BREAD_COMPACT_WIFI_LCD || BOARD_TYPE_BREAD_COMPACT_ESP32_LCD || BOARD_TYPE_WDMOMO_CGC || BOARD_TYPE_BREAD_COMPACT_WIFI_CAM
prompt "LCD Type"
default LCD_ST7789_240X320
help
@@ -740,7 +730,7 @@ choice DISPLAY_LCD_TYPE
endchoice
choice DISPLAY_ESP32S3_KORVO2_V3
depends on BOARD_TYPE_ESP_KORVO2_V3 || BOARD_TYPE_ESP_KORVO2_V3_RNDIS
depends on BOARD_TYPE_ESP32_S3_KORVO_2_V3 || BOARD_TYPE_ESP32_S3_KORVO_2_V3_RNDIS
prompt "ESP32S3_KORVO2_V3 LCD Type"
default ESP32S3_KORVO2_V3_LCD_ST7789
help
@@ -809,7 +799,7 @@ choice DISPLAY_STYLE
config USE_EMOTE_MESSAGE_STYLE
bool "Emote animation style"
depends on BOARD_TYPE_ESP_BOX || BOARD_TYPE_ESP_BOX_3 \
depends on BOARD_TYPE_ESP32_S3_BOX || BOARD_TYPE_ESP32_S3_BOX_3 \
|| BOARD_TYPE_ESP_VOCAT || BOARD_TYPE_LICHUANG_DEV_S3 || BOARD_TYPE_RYMCU_BIGSMART \
|| BOARD_TYPE_ESP_SENSAIRSHUTTLE
endchoice
@@ -827,7 +817,7 @@ config USE_MULTILINE_CHAT_MESSAGE
choice WAKE_WORD_TYPE
prompt "Wake Word Implementation Type"
default USE_AFE_WAKE_WORD if (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM
default USE_AFE_WAKE_WORD if (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4 || IDF_TARGET_ESP32S31) && SPIRAM
default WAKE_WORD_DISABLED
help
Choose the type of wake word implementation to use
@@ -845,15 +835,15 @@ choice WAKE_WORD_TYPE
config USE_AFE_WAKE_WORD
bool "Wakenet model with AFE"
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4 || IDF_TARGET_ESP32S31) && SPIRAM
help
Support AEC if available, requires ESP32 S3 and PSRAM
Support AEC if available, requires ESP32-S3, ESP32-P4, or ESP32-S31 and PSRAM
config USE_CUSTOM_WAKE_WORD
bool "Multinet model (Custom Wake Word)"
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4 || IDF_TARGET_ESP32S31) && SPIRAM
help
Requires ESP32 S3 and PSRAM
Requires ESP32-S3, ESP32-P4, or ESP32-S31 and PSRAM
endchoice
@@ -899,22 +889,22 @@ config WAKE_WORD_DETECTION_IN_LISTENING
config USE_AUDIO_PROCESSOR
bool "Enable AFE Audio Processing"
default y
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4 || IDF_TARGET_ESP32S31) && SPIRAM
help
Enable the shared AFE uplink path with AEC and VAD. Noise suppression
is not enabled because this project does not provide an NSNet model.
Requires ESP32-S3 or ESP32-P4 with PSRAM.
Requires ESP32-S3, ESP32-P4, or ESP32-S31 with PSRAM.
config USE_DEVICE_AEC
bool "Enable Device-Side AEC"
default n
depends on USE_AUDIO_PROCESSOR && (BOARD_TYPE_ESP_BOX_3 || BOARD_TYPE_ESP_BOX || BOARD_TYPE_ESP_BOX_LITE \
|| BOARD_TYPE_LICHUANG_DEV_S3 || BOARD_TYPE_RYMCU_BIGSMART || BOARD_TYPE_ESP_KORVO2_V3 || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_75|| BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_75C || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_83\
depends on USE_AUDIO_PROCESSOR && (BOARD_TYPE_ESP32_S3_BOX_3 || BOARD_TYPE_ESP32_S3_BOX || BOARD_TYPE_ESP32_S3_BOX_LITE \
|| BOARD_TYPE_LICHUANG_DEV_S3 || BOARD_TYPE_RYMCU_BIGSMART || BOARD_TYPE_ESP32_S3_KORVO_2_V3 || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_75|| BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_75C || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_83\
|| BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_2_06 || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_4B || BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_4B || BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_4_3 || BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_5 \
|| BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_7B \
|| BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_3_4C || BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_4C || BOARD_TYPE_ESP_S3_LCD_EV_Board_2 || BOARD_TYPE_YUNLIAO_S3 \
|| BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_3_4C || BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_4C || BOARD_TYPE_ESP32_S3_LCD_EV_BOARD_2 || BOARD_TYPE_YUNLIAO_S3 \
|| BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_7 || BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_8 || BOARD_TYPE_WAVESHARE_ESP32_P4_WIFI6_TOUCH_LCD_10_1 \
|| BOARD_TYPE_ESP_VOCAT || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_3_49 || BOARD_TYPE_WAVESHARE_ESP32_S3_RLCD_4_2 || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_85B || BOARD_TYPE_ZHENGCHEN_CAM || BOARD_TYPE_ZHENGCHEN_CAM_ML307 \
|| BOARD_TYPE_ESP_VOCAT || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_3_49 || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_3_49_V2 || BOARD_TYPE_WAVESHARE_ESP32_S3_RLCD_4_2 || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_85B || BOARD_TYPE_ZHENGCHEN_CAM || BOARD_TYPE_ZHENGCHEN_CAM_ML307 \
|| BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_4_3C || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_LCD_1_54 || BOARD_TYPE_WAVESHARE_ESP32_S3_LCD_0_85 || BOARD_TYPE_AI_VOX3 || BOARD_TYPE_WAVESHARE_ESP32_S3_TOUCH_AMOLED_1_43C || BOARD_TYPE_ATK_DNESP32S3_BOX3)
help
To work properly, device-side AEC requires a clean output reference path from the speaker signal and physical acoustic isolation between the microphone and speaker.
@@ -940,19 +930,14 @@ menu "WiFi Configuration Method"
default y
help
Use WiFi Hotspot to transmit WiFi configuration data
config USE_ACOUSTIC_WIFI_PROVISIONING
bool "Acoustic"
help
Use audio signal to transmit WiFi configuration data
config USE_ESP_BLUFI_WIFI_PROVISIONING
bool "Esp Blufi"
help
Use esp blufi protocol to transmit WiFi configuration data
bool "ESP-BluFi"
select BT_ENABLED
select BT_BLE_42_FEATURES_SUPPORTED
select BT_BLE_BLUFI_ENABLE
select MBEDTLS_DHM_C
help
Use the PSA Crypto based BluFi security protocol adopted by ESP-IDF 6.
The provisioning client must support ffdhe3072, SHA-256, and AES-CTR.
endmenu
config AUDIO_DEBUG_UDP_SERVER
+263 -203
View File
@@ -1,25 +1,24 @@
#include "application.h"
#include "assets.h"
#include "assets/lang_config.h"
#include "audio_codec.h"
#include "board.h"
#include "display.h"
#include "system_info.h"
#include "audio_codec.h"
#include "mqtt_protocol.h"
#include "websocket_protocol.h"
#include "assets/lang_config.h"
#include "mcp_server.h"
#include "assets.h"
#include "mqtt_protocol.h"
#include "settings.h"
#include "system_info.h"
#include "text_glyph_payload.h"
#include "websocket_protocol.h"
#include <cstring>
#include <esp_log.h>
#include <cJSON.h>
#include <driver/gpio.h>
#include <esp_log.h>
#include <arpa/inet.h>
#include <font_awesome.h>
#include <cJSON.h>
#include <cstring>
#define TAG "Application"
Application::Application() {
event_group_ = xEventGroupCreate();
@@ -33,16 +32,16 @@ Application::Application() {
aec_mode_ = kAecOff;
#endif
esp_timer_create_args_t clock_timer_args = {
.callback = [](void* arg) {
Application* app = (Application*)arg;
xEventGroupSetBits(app->event_group_, MAIN_EVENT_CLOCK_TICK);
},
.arg = this,
.dispatch_method = ESP_TIMER_TASK,
.name = "clock_timer",
.skip_unhandled_events = true
};
esp_timer_create_args_t clock_timer_args = {.callback =
[](void* arg) {
Application* app = (Application*)arg;
xEventGroupSetBits(app->event_group_,
MAIN_EVENT_CLOCK_TICK);
},
.arg = this,
.dispatch_method = ESP_TIMER_TASK,
.name = "clock_timer",
.skip_unhandled_events = true};
esp_timer_create(&clock_timer_args, &clock_timer_handle_);
}
@@ -54,9 +53,7 @@ Application::~Application() {
vEventGroupDelete(event_group_);
}
bool Application::SetDeviceState(DeviceState state) {
return state_machine_.TransitionTo(state);
}
bool Application::SetDeviceState(DeviceState state) { return state_machine_.TransitionTo(state); }
void Application::Initialize() {
auto& board = Board::GetInstance();
@@ -83,6 +80,9 @@ void Application::Initialize() {
callbacks.on_vad_change = [this](bool speaking) {
xEventGroupSetBits(event_group_, MAIN_EVENT_VAD_CHANGE);
};
callbacks.on_playback_drained = [this]() {
xEventGroupSetBits(event_group_, MAIN_EVENT_PLAYBACK_DRAINED);
};
audio_service_.SetCallbacks(callbacks);
// Add state change listeners
@@ -101,7 +101,7 @@ void Application::Initialize() {
// Set network event callback for UI updates and network state handling
board.SetNetworkEventCallback([this](NetworkEvent event, const std::string& data) {
auto display = Board::GetInstance().GetDisplay();
switch (event) {
case NetworkEvent::Scanning:
display->ShowNotification(Lang::Strings::SCANNING_WIFI, 30000);
@@ -141,13 +141,16 @@ void Application::Initialize() {
display->SetStatus(Lang::Strings::DETECTING_MODULE);
break;
case NetworkEvent::ModemErrorNoSim:
Alert(Lang::Strings::ERROR, Lang::Strings::PIN_ERROR, "triangle_exclamation", Lang::Sounds::OGG_ERR_PIN);
Alert(Lang::Strings::ERROR, Lang::Strings::PIN_ERROR, "warning",
Lang::Sounds::OGG_ERR_PIN);
break;
case NetworkEvent::ModemErrorRegDenied:
Alert(Lang::Strings::ERROR, Lang::Strings::REG_ERROR, "triangle_exclamation", Lang::Sounds::OGG_ERR_REG);
Alert(Lang::Strings::ERROR, Lang::Strings::REG_ERROR, "warning",
Lang::Sounds::OGG_ERR_REG);
break;
case NetworkEvent::ModemErrorInitFailed:
Alert(Lang::Strings::ERROR, Lang::Strings::MODEM_INIT_ERROR, "triangle_exclamation", Lang::Sounds::OGG_EXCLAMATION);
Alert(Lang::Strings::ERROR, Lang::Strings::MODEM_INIT_ERROR, "warning",
Lang::Sounds::OGG_EXCLAMATION);
break;
case NetworkEvent::ModemErrorTimeout:
display->SetStatus(Lang::Strings::REGISTERING_NETWORK);
@@ -166,27 +169,20 @@ void Application::Run() {
// Set the priority of the main task to 10
vTaskPrioritySet(nullptr, 10);
const EventBits_t ALL_EVENTS =
MAIN_EVENT_SCHEDULE |
MAIN_EVENT_SEND_AUDIO |
MAIN_EVENT_WAKE_WORD_DETECTED |
MAIN_EVENT_VAD_CHANGE |
MAIN_EVENT_CLOCK_TICK |
MAIN_EVENT_ERROR |
MAIN_EVENT_NETWORK_CONNECTED |
MAIN_EVENT_NETWORK_DISCONNECTED |
MAIN_EVENT_TOGGLE_CHAT |
MAIN_EVENT_START_LISTENING |
MAIN_EVENT_STOP_LISTENING |
MAIN_EVENT_ACTIVATION_DONE |
MAIN_EVENT_STATE_CHANGED;
const EventBits_t ALL_EVENTS =
MAIN_EVENT_SCHEDULE | MAIN_EVENT_SEND_AUDIO | MAIN_EVENT_WAKE_WORD_DETECTED |
MAIN_EVENT_VAD_CHANGE | MAIN_EVENT_CLOCK_TICK | MAIN_EVENT_ERROR |
MAIN_EVENT_NETWORK_CONNECTED | MAIN_EVENT_NETWORK_DISCONNECTED | MAIN_EVENT_TOGGLE_CHAT |
MAIN_EVENT_START_LISTENING | MAIN_EVENT_STOP_LISTENING | MAIN_EVENT_ACTIVATION_DONE |
MAIN_EVENT_STATE_CHANGED | MAIN_EVENT_PLAYBACK_DRAINED;
while (true) {
auto bits = xEventGroupWaitBits(event_group_, ALL_EVENTS, pdTRUE, pdFALSE, portMAX_DELAY);
if (bits & MAIN_EVENT_ERROR) {
SetDeviceState(kDeviceStateIdle);
Alert(Lang::Strings::ERROR, last_error_message_.c_str(), "circle_xmark", Lang::Sounds::OGG_EXCLAMATION);
Alert(Lang::Strings::ERROR, last_error_message_.c_str(), "cancel",
Lang::Sounds::OGG_EXCLAMATION);
}
if (bits & MAIN_EVENT_NETWORK_CONNECTED) {
@@ -205,6 +201,16 @@ void Application::Run() {
HandleStateChangedEvent();
}
if (bits & MAIN_EVENT_PLAYBACK_DRAINED) {
// Deferred listening start (auto mode): the playback queue has
// drained, so it is now safe to enable voice processing.
if (pending_listening_start_ && GetDeviceState() == kDeviceStateListening &&
audio_service_.IsPlaybackIdle()) {
pending_listening_start_ = false;
StartListeningAudio();
}
}
if (bits & MAIN_EVENT_TOGGLE_CHAT) {
HandleToggleChatEvent();
}
@@ -220,6 +226,12 @@ void Application::Run() {
if (bits & MAIN_EVENT_SEND_AUDIO) {
while (auto packet = audio_service_.PopPacketFromSendQueue()) {
if (protocol_ && !protocol_->SendAudio(std::move(packet))) {
// Drop the remaining packets. Leaving them in the queue would
// stall the Opus codec task (it waits for queue space), which in
// turn deadlocks the whole audio input pipeline, as no new
// MAIN_EVENT_SEND_AUDIO event would ever be triggered again.
while (audio_service_.PopPacketFromSendQueue())
;
break;
}
}
@@ -249,10 +261,12 @@ void Application::Run() {
clock_ticks_++;
auto display = Board::GetInstance().GetDisplay();
display->UpdateStatusBar();
// Print debug info every 10 seconds
if (clock_ticks_ % 10 == 0) {
SystemInfo::PrintHeapStats();
// SystemInfo::PrintTaskList();
// SystemInfo::PrintTaskCpuUsage(pdMS_TO_TICKS(1000));
}
}
}
@@ -270,12 +284,14 @@ void Application::HandleNetworkConnectedEvent() {
return;
}
xTaskCreate([](void* arg) {
Application* app = static_cast<Application*>(arg);
app->ActivationTask();
app->activation_task_handle_ = nullptr;
vTaskDelete(NULL);
}, "activation", 4096 * 2, this, 2, &activation_task_handle_);
xTaskCreate(
[](void* arg) {
Application* app = static_cast<Application*>(arg);
app->ActivationTask();
app->activation_task_handle_ = nullptr;
vTaskDelete(NULL);
},
"activation", 4096 * 2, this, 2, &activation_task_handle_);
}
// Update the status bar immediately to show the network state
@@ -286,7 +302,8 @@ void Application::HandleNetworkConnectedEvent() {
void Application::HandleNetworkDisconnectedEvent() {
// Close current conversation when network disconnected
auto state = GetDeviceState();
if (state == kDeviceStateConnecting || state == kDeviceStateListening || state == kDeviceStateSpeaking) {
if (state == kDeviceStateConnecting || state == kDeviceStateListening ||
state == kDeviceStateSpeaking) {
ESP_LOGI(TAG, "Closing audio channel due to network disconnection");
protocol_->CloseAudioChannel();
}
@@ -352,7 +369,7 @@ void Application::CheckAssetsVersion() {
ESP_LOGW(TAG, "Assets partition is disabled for board %s", BOARD_NAME);
return;
}
Settings settings("assets", true);
// Check if there is a new assets need to be downloaded
std::string download_url = settings.GetString("download_url");
@@ -362,27 +379,29 @@ void Application::CheckAssetsVersion() {
char message[256];
snprintf(message, sizeof(message), Lang::Strings::FOUND_NEW_ASSETS, download_url.c_str());
Alert(Lang::Strings::LOADING_ASSETS, message, "cloud_arrow_down", Lang::Sounds::OGG_UPGRADE);
Alert(Lang::Strings::LOADING_ASSETS, message, "cloud_download", Lang::Sounds::OGG_UPGRADE);
// Wait for the audio service to be idle for 3 seconds
vTaskDelay(pdMS_TO_TICKS(3000));
SetDeviceState(kDeviceStateUpgrading);
board.SetPowerSaveLevel(PowerSaveLevel::PERFORMANCE);
display->SetChatMessage("system", Lang::Strings::PLEASE_WAIT);
bool success = assets.Download(download_url, [this, display](int progress, size_t speed) -> void {
char buffer[32];
snprintf(buffer, sizeof(buffer), "%d%% %uKB/s", progress, speed / 1024);
Schedule([display, message = std::string(buffer)]() {
display->SetChatMessage("system", message.c_str());
bool success =
assets.Download(download_url, [this, display](int progress, size_t speed) -> void {
char buffer[32];
snprintf(buffer, sizeof(buffer), "%d%% %uKB/s", progress, speed / 1024);
Schedule([display, message = std::string(buffer)]() {
display->SetChatMessage("system", message.c_str());
});
});
});
board.SetPowerSaveLevel(PowerSaveLevel::LOW_POWER);
vTaskDelay(pdMS_TO_TICKS(1000));
if (!success) {
Alert(Lang::Strings::ERROR, Lang::Strings::DOWNLOAD_ASSETS_FAILED, "circle_xmark", Lang::Sounds::OGG_EXCLAMATION);
Alert(Lang::Strings::ERROR, Lang::Strings::DOWNLOAD_ASSETS_FAILED, "cancel",
Lang::Sounds::OGG_EXCLAMATION);
vTaskDelay(pdMS_TO_TICKS(2000));
SetDeviceState(kDeviceStateActivating);
return;
@@ -392,13 +411,13 @@ void Application::CheckAssetsVersion() {
// Apply assets
assets.Apply();
display->SetChatMessage("system", "");
display->SetEmotion("microchip_ai");
display->SetEmotion("robot_2");
}
void Application::CheckNewVersion() {
const int MAX_RETRY = 10;
int retry_count = 0;
int retry_delay = 10; // Initial retry delay in seconds
int retry_delay = 10; // Initial retry delay in seconds
auto& board = Board::GetInstance();
while (true) {
@@ -414,27 +433,30 @@ void Application::CheckNewVersion() {
}
char error_message[128];
snprintf(error_message, sizeof(error_message), "code=%d, url=%s", err, ota_->GetCheckVersionUrl().c_str());
snprintf(error_message, sizeof(error_message), "code=%d, url=%s", err,
ota_->GetCheckVersionUrl().c_str());
char buffer[256];
snprintf(buffer, sizeof(buffer), Lang::Strings::CHECK_NEW_VERSION_FAILED, retry_delay, error_message);
Alert(Lang::Strings::ERROR, buffer, "cloud_slash", Lang::Sounds::OGG_EXCLAMATION);
snprintf(buffer, sizeof(buffer), Lang::Strings::CHECK_NEW_VERSION_FAILED, retry_delay,
error_message);
Alert(Lang::Strings::ERROR, buffer, "cloud_off", Lang::Sounds::OGG_EXCLAMATION);
ESP_LOGW(TAG, "Check new version failed, retry in %d seconds (%d/%d)", retry_delay, retry_count, MAX_RETRY);
ESP_LOGW(TAG, "Check new version failed, retry in %d seconds (%d/%d)", retry_delay,
retry_count, MAX_RETRY);
for (int i = 0; i < retry_delay; i++) {
vTaskDelay(pdMS_TO_TICKS(1000));
if (GetDeviceState() == kDeviceStateIdle) {
break;
}
}
retry_delay *= 2; // Double the retry delay
retry_delay *= 2; // Double the retry delay
continue;
}
retry_count = 0;
retry_delay = 10; // Reset retry delay
retry_delay = 10; // Reset retry delay
if (ota_->HasNewVersion()) {
if (UpgradeFirmware(ota_->GetFirmwareUrl(), ota_->GetFirmwareVersion())) {
return; // This line will never be reached after reboot
return; // This line will never be reached after reboot
}
// If upgrade failed, continue to normal operation
}
@@ -486,29 +508,29 @@ void Application::InitializeProtocol() {
protocol_ = std::make_unique<MqttProtocol>();
}
protocol_->OnConnected([this]() {
DismissAlert();
});
protocol_->OnConnected([this]() { DismissAlert(); });
protocol_->OnNetworkError([this](const std::string& message) {
last_error_message_ = message;
xEventGroupSetBits(event_group_, MAIN_EVENT_ERROR);
});
protocol_->OnIncomingAudio([this](std::unique_ptr<AudioStreamPacket> packet) {
if (GetDeviceState() == kDeviceStateSpeaking) {
audio_service_.PushPacketToDecodeQueue(std::move(packet));
}
});
protocol_->OnAudioChannelOpened([this, codec, &board]() {
board.SetPowerSaveLevel(PowerSaveLevel::PERFORMANCE);
if (protocol_->server_sample_rate() != codec->output_sample_rate()) {
ESP_LOGW(TAG, "Server sample rate %d does not match device output sample rate %d, resampling may cause distortion",
protocol_->server_sample_rate(), codec->output_sample_rate());
ESP_LOGW(TAG,
"Server sample rate %d does not match device output sample rate %d, "
"resampling may cause distortion",
protocol_->server_sample_rate(), codec->output_sample_rate());
}
});
protocol_->OnAudioChannelClosed([this, &board]() {
board.SetPowerSaveLevel(PowerSaveLevel::LOW_POWER);
Schedule([this]() {
@@ -517,12 +539,19 @@ void Application::InitializeProtocol() {
SetDeviceState(kDeviceStateIdle);
});
});
protocol_->OnIncomingJson([this, display](const cJSON* root) {
// Parse JSON data
auto type = cJSON_GetObjectItem(root, "type");
if (!cJSON_IsString(type)) {
ESP_LOGW(TAG, "Incoming JSON message has no type");
return;
}
if (strcmp(type->valuestring, "tts") == 0) {
auto state = cJSON_GetObjectItem(root, "state");
if (!cJSON_IsString(state)) {
return;
}
if (strcmp(state->valuestring, "start") == 0) {
Schedule([this]() {
aborted_ = false;
@@ -541,8 +570,15 @@ void Application::InitializeProtocol() {
} else if (strcmp(state->valuestring, "sentence_start") == 0) {
auto text = cJSON_GetObjectItem(root, "text");
if (cJSON_IsString(text)) {
std::vector<TextGlyph> glyphs;
uint8_t bpp = 0;
if (!TextGlyphPayload::Parse(root, glyphs, bpp)) {
glyphs.clear();
}
ESP_LOGI(TAG, "<< %s", text->valuestring);
Schedule([display, message = std::string(text->valuestring)]() {
Schedule([display, message = std::string(text->valuestring),
glyphs = std::move(glyphs), bpp]() {
display->AddTextGlyphs(glyphs, bpp);
display->SetChatMessage("assistant", message.c_str());
});
}
@@ -550,8 +586,15 @@ void Application::InitializeProtocol() {
} else if (strcmp(type->valuestring, "stt") == 0) {
auto text = cJSON_GetObjectItem(root, "text");
if (cJSON_IsString(text)) {
std::vector<TextGlyph> glyphs;
uint8_t bpp = 0;
if (!TextGlyphPayload::Parse(root, glyphs, bpp)) {
glyphs.clear();
}
ESP_LOGI(TAG, ">> %s", text->valuestring);
Schedule([display, message = std::string(text->valuestring)]() {
Schedule([display, message = std::string(text->valuestring),
glyphs = std::move(glyphs), bpp]() {
display->AddTextGlyphs(glyphs, bpp);
display->SetChatMessage("user", message.c_str());
});
}
@@ -573,9 +616,7 @@ void Application::InitializeProtocol() {
ESP_LOGI(TAG, "System command: %s", command->valuestring);
if (strcmp(command->valuestring, "reboot") == 0) {
// Do a reboot if user requests a OTA update
Schedule([this]() {
Reboot();
});
Schedule([this]() { Reboot(); });
} else {
ESP_LOGW(TAG, "Unknown system command: %s", command->valuestring);
}
@@ -585,7 +626,8 @@ void Application::InitializeProtocol() {
auto message = cJSON_GetObjectItem(root, "message");
auto emotion = cJSON_GetObjectItem(root, "emotion");
if (cJSON_IsString(status) && cJSON_IsString(message) && cJSON_IsString(emotion)) {
Alert(status->valuestring, message->valuestring, emotion->valuestring, Lang::Sounds::OGG_VIBRATION);
Alert(status->valuestring, message->valuestring, emotion->valuestring,
Lang::Sounds::OGG_VIBRATION);
} else {
ESP_LOGW(TAG, "Alert command requires status, message and emotion");
}
@@ -594,9 +636,10 @@ void Application::InitializeProtocol() {
auto payload = cJSON_GetObjectItem(root, "payload");
ESP_LOGI(TAG, "Received custom message: %s", cJSON_PrintUnformatted(root));
if (cJSON_IsObject(payload)) {
Schedule([this, display, payload_str = std::string(cJSON_PrintUnformatted(payload))]() {
display->SetChatMessage("system", payload_str.c_str());
});
Schedule(
[this, display, payload_str = std::string(cJSON_PrintUnformatted(payload))]() {
display->SetChatMessage("system", payload_str.c_str());
});
} else {
ESP_LOGW(TAG, "Invalid custom message format: missing payload");
}
@@ -605,7 +648,7 @@ void Application::InitializeProtocol() {
ESP_LOGW(TAG, "Unknown message type: %s", type->valuestring);
}
});
protocol_->Start();
}
@@ -614,32 +657,27 @@ void Application::ShowActivationCode(const std::string& code, const std::string&
char digit;
const std::string_view& sound;
};
static const std::array<digit_sound, 10> digit_sounds{{
digit_sound{'0', Lang::Sounds::OGG_0},
digit_sound{'1', Lang::Sounds::OGG_1},
digit_sound{'2', Lang::Sounds::OGG_2},
digit_sound{'3', Lang::Sounds::OGG_3},
digit_sound{'4', Lang::Sounds::OGG_4},
digit_sound{'5', Lang::Sounds::OGG_5},
digit_sound{'6', Lang::Sounds::OGG_6},
digit_sound{'7', Lang::Sounds::OGG_7},
digit_sound{'8', Lang::Sounds::OGG_8},
digit_sound{'9', Lang::Sounds::OGG_9}
}};
static const std::array<digit_sound, 10> digit_sounds{
{digit_sound{'0', Lang::Sounds::OGG_0}, digit_sound{'1', Lang::Sounds::OGG_1},
digit_sound{'2', Lang::Sounds::OGG_2}, digit_sound{'3', Lang::Sounds::OGG_3},
digit_sound{'4', Lang::Sounds::OGG_4}, digit_sound{'5', Lang::Sounds::OGG_5},
digit_sound{'6', Lang::Sounds::OGG_6}, digit_sound{'7', Lang::Sounds::OGG_7},
digit_sound{'8', Lang::Sounds::OGG_8}, digit_sound{'9', Lang::Sounds::OGG_9}}};
// This sentence uses 9KB of SRAM, so we need to wait for it to finish
Alert(Lang::Strings::ACTIVATION, message.c_str(), "link", Lang::Sounds::OGG_ACTIVATION);
for (const auto& digit : code) {
auto it = std::find_if(digit_sounds.begin(), digit_sounds.end(),
[digit](const digit_sound& ds) { return ds.digit == digit; });
[digit](const digit_sound& ds) { return ds.digit == digit; });
if (it != digit_sounds.end()) {
audio_service_.PlaySound(it->sound);
}
}
}
void Application::Alert(const char* status, const char* message, const char* emotion, const std::string_view& sound) {
void Application::Alert(const char* status, const char* message, const char* emotion,
const std::string_view& sound) {
ESP_LOGW(TAG, "Alert [%s] %s: %s", emotion, status, message);
auto display = Board::GetInstance().GetDisplay();
display->SetStatus(status);
@@ -659,21 +697,15 @@ void Application::DismissAlert() {
}
}
void Application::ToggleChatState() {
xEventGroupSetBits(event_group_, MAIN_EVENT_TOGGLE_CHAT);
}
void Application::ToggleChatState() { xEventGroupSetBits(event_group_, MAIN_EVENT_TOGGLE_CHAT); }
void Application::StartListening() {
xEventGroupSetBits(event_group_, MAIN_EVENT_START_LISTENING);
}
void Application::StartListening() { xEventGroupSetBits(event_group_, MAIN_EVENT_START_LISTENING); }
void Application::StopListening() {
xEventGroupSetBits(event_group_, MAIN_EVENT_STOP_LISTENING);
}
void Application::StopListening() { xEventGroupSetBits(event_group_, MAIN_EVENT_STOP_LISTENING); }
void Application::HandleToggleChatEvent() {
auto state = GetDeviceState();
if (state == kDeviceStateActivating) {
SetDeviceState(kDeviceStateIdle);
return;
@@ -697,9 +729,7 @@ void Application::HandleToggleChatEvent() {
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
// Schedule to let the state change be processed first (UI update)
Schedule([this, mode]() {
ContinueOpenAudioChannel(mode);
});
Schedule([this, mode]() { ContinueOpenAudioChannel(mode); });
return;
}
SetListeningMode(mode);
@@ -722,6 +752,9 @@ void Application::ContinueOpenAudioChannel(ListeningMode mode) {
if (!protocol_->IsAudioChannelOpened()) {
if (!protocol_->OpenAudioChannel()) {
// Return to idle so the device is not stuck in the connecting
// state (not every failure path reports a network error)
SetDeviceState(kDeviceStateIdle);
return;
}
}
@@ -731,7 +764,7 @@ void Application::ContinueOpenAudioChannel(ListeningMode mode) {
void Application::HandleStartListeningEvent() {
auto state = GetDeviceState();
if (state == kDeviceStateActivating) {
SetDeviceState(kDeviceStateIdle);
return;
@@ -745,14 +778,12 @@ void Application::HandleStartListeningEvent() {
ESP_LOGE(TAG, "Protocol not initialized");
return;
}
if (state == kDeviceStateIdle) {
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
// Schedule to let the state change be processed first (UI update)
Schedule([this]() {
ContinueOpenAudioChannel(kListeningModeManualStop);
});
Schedule([this]() { ContinueOpenAudioChannel(kListeningModeManualStop); });
return;
}
SetListeningMode(kListeningModeManualStop);
@@ -764,7 +795,7 @@ void Application::HandleStartListeningEvent() {
void Application::HandleStopListeningEvent() {
auto state = GetDeviceState();
if (state == kDeviceStateAudioTesting) {
audio_service_.EnableAudioTesting(false);
SetDeviceState(kDeviceStateWifiConfiguring);
@@ -787,24 +818,12 @@ void Application::HandleWakeWordDetectedEvent() {
ESP_LOGI(TAG, "Wake word detected: %s (state: %d)", wake_word.c_str(), (int)state);
if (state == kDeviceStateIdle) {
audio_service_.EncodeWakeWord();
auto wake_word = audio_service_.GetLastWakeWord();
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
// Schedule to let the state change be processed first (UI update),
// then continue with OpenAudioChannel which may block for ~1 second
Schedule([this, wake_word]() {
ContinueWakeWordInvoke(wake_word);
});
return;
}
// Channel already opened, continue directly
ContinueWakeWordInvoke(wake_word);
BeginWakeWordInvoke(wake_word);
} else if (state == kDeviceStateSpeaking || state == kDeviceStateListening) {
AbortSpeaking(kAbortReasonWakeWordDetected);
// Clear send queue to avoid sending residues to server
while (audio_service_.PopPacketFromSendQueue());
while (audio_service_.PopPacketFromSendQueue())
;
if (state == kDeviceStateListening) {
protocol_->SendStartListening(GetDefaultListeningMode());
@@ -823,6 +842,30 @@ void Application::HandleWakeWordDetectedEvent() {
}
}
void Application::BeginWakeWordInvoke(const std::string& wake_word) {
// Must run in the main task with the device in idle state
audio_service_.EncodeWakeWord();
// Always pass through the connecting state, even if the audio channel is
// already opened. ContinueWakeWordInvoke() rejects any other state, so
// skipping this transition would silently drop the wake word invocation.
if (!SetDeviceState(kDeviceStateConnecting)) {
// Wake word detection was stopped by the detection itself; restore it
// so the device does not become unresponsive to wake words.
audio_service_.EnableWakeWordDetection(true);
return;
}
if (!protocol_->IsAudioChannelOpened()) {
// Schedule to let the state change be processed first (UI update),
// then continue with OpenAudioChannel which may block for ~1 second
Schedule([this, wake_word]() { ContinueWakeWordInvoke(wake_word); });
return;
}
// Channel already opened, continue directly
ContinueWakeWordInvoke(wake_word);
}
void Application::ContinueWakeWordInvoke(const std::string& wake_word) {
// Check state again in case it was changed during scheduling
if (GetDeviceState() != kDeviceStateConnecting) {
@@ -835,7 +878,10 @@ void Application::ContinueWakeWordInvoke(const std::string& wake_word) {
if (!protocol_->IsAudioChannelOpened()) {
if (!protocol_->OpenAudioChannel()) {
audio_service_.EnableWakeWordDetection(true);
// Return to idle so the device is not stuck in the connecting
// state (not every failure path reports a network error), and
// wake word detection is re-enabled by the idle state handler.
SetDeviceState(kDeviceStateIdle);
return;
}
}
@@ -860,18 +906,21 @@ void Application::ContinueWakeWordInvoke(const std::string& wake_word) {
void Application::HandleStateChangedEvent() {
DeviceState new_state = state_machine_.GetState();
clock_ticks_ = 0;
// Any state change invalidates a pending deferred listening start;
// the Listening case below re-arms it when needed.
pending_listening_start_ = false;
auto& board = Board::GetInstance();
auto display = board.GetDisplay();
auto led = board.GetLed();
led->OnStateChanged();
switch (new_state) {
case kDeviceStateUnknown:
case kDeviceStateIdle:
display->SetStatus(Lang::Strings::STANDBY);
display->ClearChatMessages(); // Clear messages first
display->SetEmotion("neutral"); // Then set emotion (wechat mode checks child count)
display->ClearChatMessages(); // Clear messages first
display->SetEmotion("neutral"); // Then set emotion (wechat mode checks child count)
audio_service_.EnableVoiceProcessing(false);
audio_service_.EnableWakeWordDetection(true);
break;
@@ -886,29 +935,17 @@ void Application::HandleStateChangedEvent() {
// Make sure the audio processor is running
if (play_popup_on_listening_ || !audio_service_.IsAudioProcessorRunning()) {
// For auto mode, wait for playback queue to be empty before enabling voice processing
// This prevents audio truncation when STOP arrives late due to network jitter
if (listening_mode_ == kListeningModeAutoStop) {
audio_service_.WaitForPlaybackQueueEmpty();
// For auto mode, wait for the playback queue to drain before enabling
// voice processing. This prevents audio truncation when STOP arrives
// late due to network jitter. Instead of blocking the main loop here,
// defer the start until MAIN_EVENT_PLAYBACK_DRAINED arrives.
if (listening_mode_ == kListeningModeAutoStop && !audio_service_.IsPlaybackIdle()) {
pending_listening_start_ = true;
} else {
StartListeningAudio();
}
// Send the start listening command
protocol_->SendStartListening(listening_mode_);
audio_service_.EnableVoiceProcessing(true);
}
#ifdef CONFIG_WAKE_WORD_DETECTION_IN_LISTENING
// Enable wake word detection in listening mode (configured via Kconfig)
audio_service_.EnableWakeWordDetection(audio_service_.IsAfeWakeWord());
#else
// Disable wake word detection in listening mode
audio_service_.EnableWakeWordDetection(false);
#endif
// Play popup sound after ResetDecoder (in EnableVoiceProcessing) has been called
if (play_popup_on_listening_) {
play_popup_on_listening_ = false;
audio_service_.PlaySound(Lang::Sounds::OGG_POPUP);
} else {
ConfigureWakeWordForListening();
}
break;
case kDeviceStateSpeaking:
@@ -931,6 +968,36 @@ void Application::HandleStateChangedEvent() {
}
}
void Application::StartListeningAudio() {
// Runs in the main loop, either directly from HandleStateChangedEvent or
// deferred via MAIN_EVENT_PLAYBACK_DRAINED once the playback queue drains.
if (GetDeviceState() != kDeviceStateListening) {
return;
}
// Send the start listening command
protocol_->SendStartListening(listening_mode_);
audio_service_.EnableVoiceProcessing(true);
ConfigureWakeWordForListening();
// Play popup sound after ResetDecoder (in EnableVoiceProcessing) has been called
if (play_popup_on_listening_) {
play_popup_on_listening_ = false;
audio_service_.PlaySound(Lang::Sounds::OGG_POPUP);
}
}
void Application::ConfigureWakeWordForListening() {
#ifdef CONFIG_WAKE_WORD_DETECTION_IN_LISTENING
// Enable wake word detection in listening mode (configured via Kconfig)
audio_service_.EnableWakeWordDetection(audio_service_.IsAfeWakeWord());
#else
// Disable wake word detection in listening mode
audio_service_.EnableWakeWordDetection(false);
#endif
}
void Application::Schedule(std::function<void()>&& callback) {
{
std::lock_guard<std::mutex> lock(mutex_);
@@ -983,7 +1050,8 @@ bool Application::UpgradeFirmware(const std::string& url, const std::string& ver
}
ESP_LOGI(TAG, "Starting firmware upgrade from URL: %s", upgrade_url.c_str());
Alert(Lang::Strings::OTA_UPGRADE, Lang::Strings::UPGRADING, "download", Lang::Sounds::OGG_UPGRADE);
Alert(Lang::Strings::OTA_UPGRADE, Lang::Strings::UPGRADING, "download",
Lang::Sounds::OGG_UPGRADE);
vTaskDelay(pdMS_TO_TICKS(3000));
SetDeviceState(kDeviceStateUpgrading);
@@ -1005,17 +1073,19 @@ bool Application::UpgradeFirmware(const std::string& url, const std::string& ver
if (!upgrade_success) {
// Upgrade failed, restart audio service and continue running
ESP_LOGE(TAG, "Firmware upgrade failed, restarting audio service and continuing operation...");
audio_service_.Start(); // Restart audio service
board.SetPowerSaveLevel(PowerSaveLevel::LOW_POWER); // Restore power save level
Alert(Lang::Strings::ERROR, Lang::Strings::UPGRADE_FAILED, "circle_xmark", Lang::Sounds::OGG_EXCLAMATION);
ESP_LOGE(TAG,
"Firmware upgrade failed, restarting audio service and continuing operation...");
audio_service_.Start(); // Restart audio service
board.SetPowerSaveLevel(PowerSaveLevel::LOW_POWER); // Restore power save level
Alert(Lang::Strings::ERROR, Lang::Strings::UPGRADE_FAILED, "cancel",
Lang::Sounds::OGG_EXCLAMATION);
vTaskDelay(pdMS_TO_TICKS(3000));
return false;
} else {
// Upgrade success, reboot immediately
ESP_LOGI(TAG, "Firmware upgrade successful, rebooting...");
display->SetChatMessage("system", "Upgrade successful, rebooting...");
vTaskDelay(pdMS_TO_TICKS(1000)); // Brief pause to show message
vTaskDelay(pdMS_TO_TICKS(1000)); // Brief pause to show message
Reboot();
return true;
}
@@ -1027,25 +1097,18 @@ void Application::WakeWordInvoke(const std::string& wake_word) {
}
auto state = GetDeviceState();
if (state == kDeviceStateIdle) {
audio_service_.EncodeWakeWord();
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
// Schedule to let the state change be processed first (UI update)
Schedule([this, wake_word]() {
ContinueWakeWordInvoke(wake_word);
});
return;
}
// Channel already opened, continue directly
ContinueWakeWordInvoke(wake_word);
} else if (state == kDeviceStateSpeaking) {
Schedule([this]() {
AbortSpeaking(kAbortReasonNone);
if (state == kDeviceStateIdle) {
// May be called from outside the main task (e.g. board button
// callbacks), so schedule the invocation instead of running it here
Schedule([this, wake_word]() {
if (GetDeviceState() == kDeviceStateIdle) {
BeginWakeWordInvoke(wake_word);
}
});
} else if (state == kDeviceStateListening) {
} else if (state == kDeviceStateSpeaking) {
Schedule([this]() { AbortSpeaking(kAbortReasonNone); });
} else if (state == kDeviceStateListening) {
Schedule([this]() {
if (protocol_) {
protocol_->CloseAudioChannel();
@@ -1077,7 +1140,7 @@ void Application::RegisterMcpBroadcastCallback(std::function<void(const std::str
void Application::SendMcpMessage(const std::string& payload) {
// Always schedule to run in main task for thread safety
Schedule([this, payload](){
Schedule([this, payload]() {
if (protocol_) {
protocol_->SendMcpMessage(payload);
}
@@ -1093,18 +1156,18 @@ void Application::SetAecMode(AecMode mode) {
auto& board = Board::GetInstance();
auto display = board.GetDisplay();
switch (aec_mode_) {
case kAecOff:
audio_service_.EnableDeviceAec(false);
display->ShowNotification(Lang::Strings::RTC_MODE_OFF);
break;
case kAecOnServerSide:
audio_service_.EnableDeviceAec(false);
display->ShowNotification(Lang::Strings::RTC_MODE_ON);
break;
case kAecOnDeviceSide:
audio_service_.EnableDeviceAec(true);
display->ShowNotification(Lang::Strings::RTC_MODE_ON);
break;
case kAecOff:
audio_service_.EnableDeviceAec(false);
display->ShowNotification(Lang::Strings::RTC_MODE_OFF);
break;
case kAecOnServerSide:
audio_service_.EnableDeviceAec(false);
display->ShowNotification(Lang::Strings::RTC_MODE_ON);
break;
case kAecOnDeviceSide:
audio_service_.EnableDeviceAec(true);
display->ShowNotification(Lang::Strings::RTC_MODE_ON);
break;
}
// If the AEC mode is changed, close the audio channel
@@ -1114,9 +1177,7 @@ void Application::SetAecMode(AecMode mode) {
});
}
void Application::PlaySound(const std::string_view& sound) {
audio_service_.PlaySound(sound);
}
void Application::PlaySound(const std::string_view& sound) { audio_service_.PlaySound(sound); }
void Application::ResetProtocol() {
Schedule([this]() {
@@ -1128,4 +1189,3 @@ void Application::ResetProtocol() {
protocol_.reset();
});
}
+5
View File
@@ -32,6 +32,7 @@
#define MAIN_EVENT_START_LISTENING (1 << 10)
#define MAIN_EVENT_STOP_LISTENING (1 << 11)
#define MAIN_EVENT_STATE_CHANGED (1 << 12)
#define MAIN_EVENT_PLAYBACK_DRAINED (1 << 13)
enum AecMode {
@@ -144,6 +145,7 @@ private:
bool aborted_ = false;
bool assets_version_checked_ = false;
bool play_popup_on_listening_ = false; // Flag to play popup sound after state changes to listening
bool pending_listening_start_ = false; // Waiting for playback to drain before starting listening (auto mode)
int clock_ticks_ = 0;
TaskHandle_t activation_task_handle_ = nullptr;
@@ -158,7 +160,10 @@ private:
void HandleActivationDoneEvent();
void HandleWakeWordDetectedEvent();
void ContinueOpenAudioChannel(ListeningMode mode);
void BeginWakeWordInvoke(const std::string& wake_word);
void ContinueWakeWordInvoke(const std::string& wake_word);
void StartListeningAudio();
void ConfigureWakeWordForListening();
// Activation task (runs in background)
void ActivationTask();
+139 -80
View File
@@ -1,33 +1,37 @@
#include "assets.h"
#include "application.h"
#include "board.h"
#include "display.h"
#include "application.h"
#include "lvgl_theme.h"
#include "emote_display.h"
#include "expression_emote.h"
#include "lvgl_theme.h"
#if HAVE_LVGL
#include "display/lcd_display.h"
#include <spi_flash_mmap.h>
#include "display/lcd_display.h"
#include "display/lvgl_display/lvgl_display.h"
#endif
#include <esp_heap_caps.h>
#include <esp_log.h>
#include <esp_timer.h>
#include <esp_heap_caps.h>
#include <cbin_font.h>
#include <noto_font_bundle.h>
#include <cstring>
#define TAG "Assets"
#define PARTITION_LABEL "assets"
struct mmap_assets_table {
char asset_name[32]; /*!< Name of the asset */
uint32_t asset_size; /*!< Size of the asset */
uint32_t asset_offset; /*!< Offset of the asset */
uint16_t asset_width; /*!< Width of the asset */
uint16_t asset_height; /*!< Height of the asset */
char asset_name[32]; /*!< Name of the asset */
uint32_t asset_size; /*!< Size of the asset */
uint32_t asset_offset; /*!< Offset of the asset */
uint16_t asset_width; /*!< Width of the asset */
uint16_t asset_height; /*!< Height of the asset */
};
Assets::Assets() {
UseBuiltInTextFontCapability();
#if HAVE_LVGL
strategy_ = std::make_unique<Assets::LvglStrategy>();
#else
@@ -37,12 +41,11 @@ Assets::Assets() {
InitializePartition();
}
Assets::~Assets() {
UnApplyPartition();
}
Assets::~Assets() { UnApplyPartition(); }
bool Assets::FindPartition(Assets* assets) {
assets->partition_ = esp_partition_find_first(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY, PARTITION_LABEL);
assets->partition_ = esp_partition_find_first(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY,
PARTITION_LABEL);
if (assets->partition_ == nullptr) {
ESP_LOGI(TAG, "No assets partition found");
return false;
@@ -59,11 +62,24 @@ bool Assets::InitializePartition() {
}
void Assets::UnApplyPartition() {
UseBuiltInTextFontCapability();
if (strategy_) {
strategy_->UnApplyPartition(this);
}
}
void Assets::UseBuiltInTextFontCapability() {
text_font_capability_ = {
.glyph_push = true,
.bundle = NOTO_FONT_BUNDLE_ID,
.charset = "basic",
.size = TEXT_FONT_SIZE,
.bpp = TEXT_FONT_BPP,
};
}
void Assets::DisableTextFontGlyphPush() { text_font_capability_ = {}; }
bool Assets::GetAssetData(const std::string& name, void*& ptr, size_t& size) {
return strategy_ ? strategy_->GetAssetData(this, name, ptr, size) : false;
}
@@ -140,11 +156,14 @@ bool Assets::LvglStrategy::InitializePartition(Assets* assets) {
ESP_LOGI(TAG, "The storage free size is %ld KB", storage_size / 1024);
ESP_LOGI(TAG, "The partition size is %ld KB", assets->partition_->size / 1024);
if (storage_size < assets->partition_->size) {
ESP_LOGE(TAG, "The free size %ld KB is less than assets partition required %ld KB", storage_size / 1024, assets->partition_->size / 1024);
ESP_LOGE(TAG, "The free size %ld KB is less than assets partition required %ld KB",
storage_size / 1024, assets->partition_->size / 1024);
return false;
}
esp_err_t err = esp_partition_mmap(assets->partition_, 0, assets->partition_->size, ESP_PARTITION_MMAP_DATA, (const void**)&mmap_root_, &mmap_handle_);
esp_err_t err =
esp_partition_mmap(assets->partition_, 0, assets->partition_->size, ESP_PARTITION_MMAP_DATA,
(const void**)&mmap_root_, &mmap_handle_);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to mmap assets partition: %s", esp_err_to_name(err));
return false;
@@ -157,7 +176,8 @@ bool Assets::LvglStrategy::InitializePartition(Assets* assets) {
uint32_t stored_len = *(uint32_t*)(mmap_root_ + 8);
if (stored_len > assets->partition_->size - 12) {
ESP_LOGD(TAG, "The stored_len (0x%lx) is greater than the partition size (0x%lx) - 12", stored_len, assets->partition_->size);
ESP_LOGD(TAG, "The stored_len (0x%lx) is greater than the partition size (0x%lx) - 12",
stored_len, assets->partition_->size);
return false;
}
@@ -167,7 +187,8 @@ bool Assets::LvglStrategy::InitializePartition(Assets* assets) {
ESP_LOGI(TAG, "The checksum calculation time is %d ms", int((end_time - start_time) / 1000));
if (calculated_checksum != stored_chksum) {
ESP_LOGE(TAG, "The calculated checksum (0x%lx) does not match the stored checksum (0x%lx)", calculated_checksum, stored_chksum);
ESP_LOGE(TAG, "The calculated checksum (0x%lx) does not match the stored checksum (0x%lx)",
calculated_checksum, stored_chksum);
return false;
}
@@ -175,10 +196,9 @@ bool Assets::LvglStrategy::InitializePartition(Assets* assets) {
for (uint32_t i = 0; i < stored_files; i++) {
auto item = (const mmap_assets_table*)(mmap_root_ + 12 + i * sizeof(mmap_assets_table));
auto asset = Asset{
.size = static_cast<size_t>(item->asset_size),
.offset = static_cast<size_t>(12 + sizeof(mmap_assets_table) * stored_files + item->asset_offset)
};
auto asset = Asset{.size = static_cast<size_t>(item->asset_size),
.offset = static_cast<size_t>(
12 + sizeof(mmap_assets_table) * stored_files + item->asset_offset)};
assets_[item->asset_name] = asset;
}
return checksum_valid_;
@@ -192,17 +212,19 @@ void Assets::LvglStrategy::UnApplyPartition(Assets* assets) {
}
checksum_valid_ = false;
assets_.clear();
(void)assets; // Unused parameter
(void)assets; // Unused parameter
}
bool Assets::LvglStrategy::GetAssetData(Assets* assets, const std::string& name, void*& ptr, size_t& size) {
bool Assets::LvglStrategy::GetAssetData(Assets* assets, const std::string& name, void*& ptr,
size_t& size) {
auto asset = assets_.find(name);
if (asset == assets_.end()) {
return false;
}
auto data = (const char*)(mmap_root_ + asset->second.offset);
if (data[0] != 'Z' || data[1] != 'Z') {
ESP_LOGE(TAG, "The asset %s is not valid with magic %02x%02x", name.c_str(), data[0], data[1]);
ESP_LOGE(TAG, "The asset %s is not valid with magic %02x%02x", name.c_str(), data[0],
data[1]);
return false;
}
@@ -228,7 +250,8 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
cJSON* version = cJSON_GetObjectItem(root, "version");
if (cJSON_IsNumber(version)) {
if (version->valuedouble > 1) {
ESP_LOGE(TAG, "The assets version %d is not supported, please upgrade the firmware", version->valueint);
ESP_LOGE(TAG, "The assets version %d is not supported, please upgrade the firmware",
version->valueint);
return false;
}
}
@@ -244,15 +267,39 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
std::string fonts_text_file = font->valuestring;
if (assets->GetAssetData(fonts_text_file, ptr, size)) {
auto text_font = std::make_shared<LvglCBinFont>(ptr);
if (text_font->font() == nullptr) {
ESP_LOGE(TAG, "Failed to load fonts.bin");
return false;
}
if (light_theme != nullptr) {
light_theme->set_text_font(text_font);
}
if (dark_theme != nullptr) {
dark_theme->set_text_font(text_font);
auto display = dynamic_cast<LvglDisplay*>(Board::GetInstance().GetDisplay());
if (text_font->font() == nullptr || display == nullptr ||
!display->SetTextFont(text_font)) {
ESP_LOGW(TAG, "Ignoring invalid text font asset %s", fonts_text_file.c_str());
} else {
assets->DisableTextFontGlyphPush();
cJSON* metadata = cJSON_GetObjectItem(root, "text_font_meta");
cJSON* charset = cJSON_GetObjectItem(metadata, "charset");
cJSON* font_size = cJSON_GetObjectItem(metadata, "size");
cJSON* font_bpp = cJSON_GetObjectItem(metadata, "bpp");
cJSON* bundle = cJSON_GetObjectItem(metadata, "bundle");
bool supports_glyph_push =
cJSON_IsString(charset) &&
(std::strcmp(charset->valuestring, "basic") == 0 ||
std::strcmp(charset->valuestring, "common") == 0) &&
cJSON_IsNumber(font_size) && font_size->valueint > 0 &&
font_size->valuedouble == font_size->valueint && cJSON_IsNumber(font_bpp) &&
font_bpp->valuedouble == font_bpp->valueint &&
(font_bpp->valueint == 1 || font_bpp->valueint == 4) &&
font_bpp->valueint == text_font->bpp() && cJSON_IsString(bundle) &&
bundle->valuestring[0] != '\0' && std::strlen(bundle->valuestring) <= 64;
if (supports_glyph_push) {
assets->text_font_capability_ = {
.glyph_push = true,
.bundle = bundle->valuestring,
.charset = charset->valuestring,
.size = font_size->valueint,
.bpp = font_bpp->valueint,
};
} else {
ESP_LOGW(TAG, "Loaded custom text font without compatible glyph push metadata");
}
}
} else {
ESP_LOGE(TAG, "The font file %s is not found", fonts_text_file.c_str());
@@ -269,12 +316,14 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
cJSON* name = cJSON_GetObjectItem(emoji, "name");
cJSON* file = cJSON_GetObjectItem(emoji, "file");
cJSON* eaf = cJSON_GetObjectItem(emoji, "eaf");
if (cJSON_IsString(name) && cJSON_IsString(file) && (NULL== eaf)) {
if (cJSON_IsString(name) && cJSON_IsString(file) && (NULL == eaf)) {
if (!assets->GetAssetData(file->valuestring, ptr, size)) {
ESP_LOGE(TAG, "Emoji %s image file %s is not found", name->valuestring, file->valuestring);
ESP_LOGE(TAG, "Emoji %s image file %s is not found", name->valuestring,
file->valuestring);
continue;
}
custom_emoji_collection->AddEmoji(name->valuestring, new LvglRawImage(ptr, size));
custom_emoji_collection->AddEmoji(name->valuestring,
new LvglRawImage(ptr, size));
}
}
}
@@ -284,6 +333,7 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
if (dark_theme != nullptr) {
dark_theme->set_emoji_collection(custom_emoji_collection);
}
Board::GetInstance().GetDisplay()->SetEmojiCollection(custom_emoji_collection);
}
cJSON* skin = cJSON_GetObjectItem(root, "skin");
@@ -297,12 +347,15 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
light_theme->set_text_color(LvglTheme::ParseColor(text_color->valuestring));
}
if (cJSON_IsString(background_color)) {
light_theme->set_background_color(LvglTheme::ParseColor(background_color->valuestring));
light_theme->set_chat_background_color(LvglTheme::ParseColor(background_color->valuestring));
light_theme->set_background_color(
LvglTheme::ParseColor(background_color->valuestring));
light_theme->set_chat_background_color(
LvglTheme::ParseColor(background_color->valuestring));
}
if (cJSON_IsString(background_image)) {
if (!assets->GetAssetData(background_image->valuestring, ptr, size)) {
ESP_LOGE(TAG, "The background image file %s is not found", background_image->valuestring);
ESP_LOGE(TAG, "The background image file %s is not found",
background_image->valuestring);
return false;
}
auto background_image = std::make_shared<LvglCBinImage>(ptr);
@@ -318,12 +371,15 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
dark_theme->set_text_color(LvglTheme::ParseColor(text_color->valuestring));
}
if (cJSON_IsString(background_color)) {
dark_theme->set_background_color(LvglTheme::ParseColor(background_color->valuestring));
dark_theme->set_chat_background_color(LvglTheme::ParseColor(background_color->valuestring));
dark_theme->set_background_color(
LvglTheme::ParseColor(background_color->valuestring));
dark_theme->set_chat_background_color(
LvglTheme::ParseColor(background_color->valuestring));
}
if (cJSON_IsString(background_image)) {
if (!assets->GetAssetData(background_image->valuestring, ptr, size)) {
ESP_LOGE(TAG, "The background image file %s is not found", background_image->valuestring);
ESP_LOGE(TAG, "The background image file %s is not found",
background_image->valuestring);
return false;
}
auto background_image = std::make_shared<LvglCBinImage>(ptr);
@@ -352,11 +408,11 @@ bool Assets::LvglStrategy::Apply(Assets* assets, bool refresh_display_theme) {
}
}
}
cJSON_Delete(root);
return true;
}
#endif // HAVE_LVGL
#endif // HAVE_LVGL
bool Assets::EmoteStrategy::InitializePartition(Assets* assets) {
assets->partition_valid_ = false;
@@ -371,12 +427,14 @@ bool Assets::EmoteStrategy::InitializePartition(Assets* assets) {
if (emote_display && emote_display->GetEmoteHandle() != nullptr) {
const emote_data_t data = {
.type = EMOTE_SOURCE_PARTITION,
.source = {
.partition_label = PARTITION_LABEL,
},
.flags = {
.mmap_enable = true, //must be true here!!!
},
.source =
{
.partition_label = PARTITION_LABEL,
},
.flags =
{
.mmap_enable = true, // must be true here!!!
},
};
ret = emote_mount_assets(emote_display->GetEmoteHandle(), &data);
} else {
@@ -392,16 +450,18 @@ void Assets::EmoteStrategy::UnApplyPartition(Assets* assets) {
if (emote_display && emote_display->GetEmoteHandle() != nullptr) {
emote_unmount_assets(emote_display->GetEmoteHandle());
}
(void)assets; // Unused parameter
(void)assets; // Unused parameter
}
bool Assets::EmoteStrategy::GetAssetData(Assets* assets, const std::string& name, void*& ptr, size_t& size) {
bool Assets::EmoteStrategy::GetAssetData(Assets* assets, const std::string& name, void*& ptr,
size_t& size) {
auto display = Board::GetInstance().GetDisplay();
auto* emote_display = dynamic_cast<emote::EmoteDisplay*>(display);
if (emote_display && emote_display->GetEmoteHandle() != nullptr) {
const uint8_t* data = nullptr;
size_t data_size = 0;
if (ESP_OK == emote_get_asset_data_by_name(emote_display->GetEmoteHandle(), name.c_str(), &data, &data_size)) {
if (ESP_OK == emote_get_asset_data_by_name(emote_display->GetEmoteHandle(), name.c_str(),
&data, &data_size)) {
ptr = const_cast<void*>(static_cast<const void*>(data));
size = data_size;
return true;
@@ -409,7 +469,7 @@ bool Assets::EmoteStrategy::GetAssetData(Assets* assets, const std::string& name
ESP_LOGE(TAG, "Failed to get asset data by name: %s", name.c_str());
return false;
}
(void)assets; // Unused parameter
(void)assets; // Unused parameter
return false;
}
@@ -425,7 +485,8 @@ bool Assets::EmoteStrategy::Apply(Assets* assets, bool refresh_display_theme) {
return true;
}
bool Assets::Download(std::string url, std::function<void(int progress, size_t speed)> progress_callback) {
bool Assets::Download(std::string url,
std::function<void(int progress, size_t speed)> progress_callback) {
ESP_LOGI(TAG, "Downloading new version of assets from %s", url.c_str());
auto network = Board::GetInstance().GetNetwork();
@@ -449,25 +510,24 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
}
if (content_length > partition_->size) {
ESP_LOGE(TAG, "Assets file size (%u) is larger than partition size (%lu)",
content_length, partition_->size);
ESP_LOGE(TAG, "Assets file size (%u) is larger than partition size (%lu)", content_length,
partition_->size);
return false;
}
constexpr size_t HEADER_SIZE = 12;
if (content_length < HEADER_SIZE) {
ESP_LOGE(TAG, "Content length (%u) is smaller than header size (%u)",
content_length, HEADER_SIZE);
ESP_LOGE(TAG, "Content length (%u) is smaller than header size (%u)", content_length,
HEADER_SIZE);
return false;
}
const size_t SECTOR_SIZE = esp_partition_get_main_flash_sector_size();
using BufferPtr = std::unique_ptr<char, decltype(&heap_caps_free)>;
BufferPtr buffer(
static_cast<char *>(heap_caps_malloc(SECTOR_SIZE, MALLOC_CAP_INTERNAL)),
&heap_caps_free);
BufferPtr buffer(static_cast<char*>(heap_caps_malloc(SECTOR_SIZE, MALLOC_CAP_INTERNAL)),
&heap_caps_free);
if (!buffer) {
ESP_LOGE(TAG, "Failed to allocate buffer");
@@ -482,8 +542,7 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
ESP_LOGI(TAG,
"Sector size: %u, content length: %u, "
"sectors to erase: %u, total erase size: %u",
SECTOR_SIZE, content_length,
sectors_to_erase, total_erase_size);
SECTOR_SIZE, content_length, sectors_to_erase, total_erase_size);
size_t total_written = 0;
size_t recent_written = 0;
@@ -529,17 +588,17 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
size_t sector_start = current_sector * SECTOR_SIZE;
size_t sector_end = sector_start + SECTOR_SIZE;
if (sector_end > partition_->size) {
ESP_LOGE(TAG, "Sector end (%u) exceeds partition size (%lu)",
sector_end, partition_->size);
ESP_LOGE(TAG, "Sector end (%u) exceeds partition size (%lu)", sector_end,
partition_->size);
erase_failed = true;
break;
}
ESP_LOGD(TAG, "Erasing sector %u (offset: %u, size: %u)",
current_sector, sector_start, SECTOR_SIZE);
ESP_LOGD(TAG, "Erasing sector %u (offset: %u, size: %u)", current_sector,
sector_start, SECTOR_SIZE);
esp_err_t err = esp_partition_erase_range(partition_, sector_start, SECTOR_SIZE);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to erase sector %u at offset %u: %s",
current_sector, sector_start, esp_err_to_name(err));
ESP_LOGE(TAG, "Failed to erase sector %u at offset %u: %s", current_sector,
sector_start, esp_err_to_name(err));
erase_failed = true;
break;
}
@@ -563,15 +622,13 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
}
// Calculate progress
if (esp_timer_get_time() - last_calc_time >= 1000000 || (header_collected + total_written) == content_length) {
if (esp_timer_get_time() - last_calc_time >= 1000000 ||
(header_collected + total_written) == content_length) {
size_t progress = (header_collected + total_written) * 100 / content_length;
size_t speed = recent_written;
ESP_LOGI(TAG, "Progress: %u%% (%u/%u), Speed: %u B/s, Sectors erased: %u",
progress,
(unsigned int)(header_collected + total_written),
(unsigned int)content_length,
(unsigned int)speed,
(unsigned int)current_sector);
ESP_LOGI(TAG, "Progress: %u%% (%u/%u), Speed: %u B/s, Sectors erased: %u", progress,
(unsigned int)(header_collected + total_written), (unsigned int)content_length,
(unsigned int)speed, (unsigned int)current_sector);
if (progress_callback) {
progress_callback(progress, speed);
@@ -602,7 +659,9 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
return false;
}
ESP_LOGI(TAG, "Header written, assets download completed, total written: %u bytes, total sectors erased: %u",
ESP_LOGI(TAG,
"Header written, assets download completed, total written: %u bytes, total sectors "
"erased: %u",
(unsigned int)(header_collected + total_written), (unsigned int)current_sector);
// Re-initialize the assets partition
@@ -612,4 +671,4 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
}
return true;
}
}
+27 -10
View File
@@ -1,12 +1,12 @@
#ifndef ASSETS_H
#define ASSETS_H
#include <string>
#include <functional>
#include <memory>
#include <string>
#include <cJSON.h>
#include <esp_partition.h>
#include <cJSON.h>
#include <model_path.h>
#include <map>
#include <string>
@@ -20,6 +20,14 @@ struct Asset {
size_t offset;
};
struct TextFontCapability {
bool glyph_push = false;
std::string bundle;
std::string charset;
int size = 0;
int bpp = 0;
};
class Assets {
public:
static Assets& GetInstance() {
@@ -28,12 +36,14 @@ public:
}
~Assets();
bool Download(std::string url, std::function<void(int progress, size_t speed)> progress_callback);
bool Download(std::string url,
std::function<void(int progress, size_t speed)> progress_callback);
bool Apply(bool refresh_display_theme = true);
bool GetAssetData(const std::string& name, void*& ptr, size_t& size);
inline bool partition_valid() const { return partition_valid_; }
inline std::string default_assets_url() const { return default_assets_url_; }
inline TextFontCapability text_font_capability() const { return text_font_capability_; }
private:
Assets();
@@ -44,22 +54,27 @@ private:
void UnApplyPartition();
static bool FindPartition(Assets* assets);
static bool LoadSrmodelsFromIndex(Assets* assets, cJSON* root = nullptr);
void UseBuiltInTextFontCapability();
void DisableTextFontGlyphPush();
class AssetStrategy {
public:
virtual ~AssetStrategy() = default;
virtual bool Apply(Assets* assets, bool refresh_display_theme = true) = 0;
virtual bool InitializePartition(Assets* assets) = 0;
virtual void UnApplyPartition(Assets* assets) = 0;
virtual bool GetAssetData(Assets* assets, const std::string& name, void*& ptr, size_t& size) = 0;
virtual bool GetAssetData(Assets* assets, const std::string& name, void*& ptr,
size_t& size) = 0;
};
class LvglStrategy : public AssetStrategy {
public:
bool Apply(Assets* assets, bool refresh_display_theme = true) override;
bool InitializePartition(Assets* assets) override;
void UnApplyPartition(Assets* assets) override;
bool GetAssetData(Assets* assets, const std::string& name, void*& ptr, size_t& size) override;
bool GetAssetData(Assets* assets, const std::string& name, void*& ptr,
size_t& size) override;
private:
static uint32_t CalculateChecksum(const char* data, uint32_t length);
std::map<std::string, Asset> assets_;
@@ -67,15 +82,16 @@ private:
const char* mmap_root_ = nullptr;
bool checksum_valid_ = false;
};
class EmoteStrategy : public AssetStrategy {
public:
bool Apply(Assets* assets, bool refresh_display_theme = true) override;
bool InitializePartition(Assets* assets) override;
void UnApplyPartition(Assets* assets) override;
bool GetAssetData(Assets* assets, const std::string& name, void*& ptr, size_t& size) override;
bool GetAssetData(Assets* assets, const std::string& name, void*& ptr,
size_t& size) override;
};
// Strategy instance
std::unique_ptr<AssetStrategy> strategy_;
@@ -83,6 +99,7 @@ protected:
const esp_partition_t* partition_ = nullptr;
bool partition_valid_ = false;
std::string default_assets_url_;
TextFontCapability text_font_capability_;
srmodel_list_t* models_list_ = nullptr;
};
+2 -2
View File
@@ -11,7 +11,7 @@ engine.
| Target | Engine | Wake word | Uplink processing |
| --- | --- | --- | --- |
| ESP32-S3 / ESP32-P4 | `AfeAudioEngine` | WakeNet inside AFE, or MultiNet fed from AFE output | FD AEC + VAD when audio processing is enabled |
| ESP32-S3 / ESP32-P4 / ESP32-S31 | `AfeAudioEngine` | WakeNet inside AFE, or MultiNet fed from AFE output | FD AEC + VAD when audio processing is enabled |
| ESP32 / ESP32-C3 / ESP32-C5 / ESP32-C6 | `LiteAudioEngine` | Standalone WakeNet when configured | Raw mono PCM |
`AfeAudioEngine` owns a single FD AFE instance. WakeNet and voice uplink share
@@ -66,7 +66,7 @@ flowchart LR
- `AudioInputTask` reads codec input and feeds the selected engine.
- `AudioOutputTask` drains decoded PCM to the codec output.
- `OpusCodecTask` encodes uplink PCM and decodes downlink packets.
- `AfeAudioEngine` has its own AFE fetch task on S3/P4.
- `AfeAudioEngine` has its own AFE fetch task on S3/P4/S31.
The audio power timer still enables and disables codec ADC/DAC channels based on
activity; the engine refactor does not change that policy.
+10
View File
@@ -4,6 +4,7 @@
#include <freertos/FreeRTOS.h>
#include <freertos/event_groups.h>
#include <driver/i2s_std.h>
#include <esp_idf_version.h>
#include <vector>
#include <string>
@@ -14,6 +15,15 @@
#define AUDIO_CODEC_DMA_DESC_NUM 6
#define AUDIO_CODEC_DMA_FRAME_NUM 240
// ESP-IDF 6 removed i2s_port_t and changed i2s_chan_config_t::id to an integer.
// Keep numeric I2S controller IDs usable on targets where IDF 5 does not expose
// every value through the target-specific i2s_port_t enum (for example ESP32-C3).
#if ESP_IDF_VERSION >= ESP_IDF_VERSION_VAL(6, 0, 0)
#define XIAOZHI_I2S_PORT(port) (port)
#else
#define XIAOZHI_I2S_PORT(port) static_cast<i2s_port_t>(port)
#endif
class AudioCodec {
public:
AudioCodec();
+172 -59
View File
@@ -22,7 +22,7 @@
.self_delimited = false, \
}
#if CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32P4
#if CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32P4 || CONFIG_IDF_TARGET_ESP32S31
#include "engines/afe_audio_engine.h"
#else
#include "engines/lite_audio_engine.h"
@@ -85,7 +85,7 @@ void AudioService::Initialize(AudioCodec* codec) {
}
}
#if CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32P4
#if CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32P4 || CONFIG_IDF_TARGET_ESP32S31
audio_engine_ = std::make_unique<AfeAudioEngine>();
#else
audio_engine_ = std::make_unique<LiteAudioEngine>();
@@ -120,8 +120,9 @@ void AudioService::Initialize(AudioCodec* codec) {
}
void AudioService::Start() {
service_stopped_ = false;
xEventGroupClearBits(event_group_, AS_EVENT_AUDIO_TESTING_RUNNING | AS_EVENT_WAKE_WORD_RUNNING | AS_EVENT_AUDIO_PROCESSOR_RUNNING);
service_stopped_.store(false);
xEventGroupClearBits(event_group_, AS_EVENT_AUDIO_TESTING_RUNNING | AS_EVENT_WAKE_WORD_RUNNING |
AS_EVENT_AUDIO_PROCESSOR_RUNNING | AS_EVENT_AUDIO_INPUT_STOP_REQUEST);
esp_timer_start_periodic(audio_power_timer_, 1000000);
@@ -165,17 +166,25 @@ void AudioService::Start() {
void AudioService::Stop() {
esp_timer_stop(audio_power_timer_);
service_stopped_ = true;
service_stopped_.store(true);
xEventGroupSetBits(event_group_, AS_EVENT_AUDIO_TESTING_RUNNING |
AS_EVENT_WAKE_WORD_RUNNING |
AS_EVENT_AUDIO_PROCESSOR_RUNNING);
std::lock_guard<std::mutex> lock(audio_queue_mutex_);
audio_encode_queue_.clear();
audio_decode_queue_.clear();
audio_playback_queue_.clear();
audio_testing_queue_.clear();
audio_queue_cv_.notify_all();
bool notify_drained = false;
{
std::lock_guard<std::mutex> lock(audio_queue_mutex_);
++playback_generation_;
audio_encode_queue_.clear();
audio_decode_queue_.clear();
audio_playback_queue_.clear();
audio_testing_queue_.clear();
notify_drained = MarkPlaybackDrainedLocked();
audio_queue_cv_.notify_all();
}
if (notify_drained && callbacks_.on_playback_drained) {
callbacks_.on_playback_drained();
}
}
bool AudioService::ReadAudioData(std::vector<int16_t>& data, int sample_rate, int samples) {
@@ -225,16 +234,39 @@ bool AudioService::ReadAudioData(std::vector<int16_t>& data, int sample_rate, in
}
void AudioService::AudioInputTask() {
constexpr EventBits_t kAudioInputActiveBits = AS_EVENT_AUDIO_TESTING_RUNNING |
AS_EVENT_WAKE_WORD_RUNNING | AS_EVENT_AUDIO_PROCESSOR_RUNNING;
while (true) {
EventBits_t bits = xEventGroupWaitBits(event_group_, AS_EVENT_AUDIO_TESTING_RUNNING |
AS_EVENT_WAKE_WORD_RUNNING | AS_EVENT_AUDIO_PROCESSOR_RUNNING,
EventBits_t bits = xEventGroupWaitBits(event_group_, kAudioInputActiveBits |
AS_EVENT_AUDIO_INPUT_STOP_REQUEST,
pdFALSE, pdFALSE, portMAX_DELAY);
if (service_stopped_) {
if (service_stopped_.load()) {
// ADC continuous mode keeps its hardware mutex from start until stop,
// so the input task that started it must also stop it before exiting.
if (codec_->input_enabled()) {
codec_->EnableInput(false);
}
break;
}
if (audio_input_need_warmup_) {
audio_input_need_warmup_ = false;
if (bits & AS_EVENT_AUDIO_INPUT_STOP_REQUEST) {
xEventGroupClearBits(event_group_, AS_EVENT_AUDIO_INPUT_STOP_REQUEST);
// Recheck the active state in this task. Audio capture may have been
// enabled after the timer posted the stop request.
bits = xEventGroupGetBits(event_group_);
if ((bits & kAudioInputActiveBits) == 0) {
if (codec_->input_enabled()) {
codec_->EnableInput(false);
}
// Do not process the stale active bits returned by waitBits().
continue;
}
}
if (audio_input_need_warmup_.exchange(false)) {
vTaskDelay(pdMS_TO_TICKS(120));
continue;
}
@@ -282,13 +314,16 @@ void AudioService::AudioInputTask() {
void AudioService::AudioOutputTask() {
while (true) {
std::unique_lock<std::mutex> lock(audio_queue_mutex_);
audio_queue_cv_.wait(lock, [this]() { return !audio_playback_queue_.empty() || service_stopped_; });
if (service_stopped_) {
audio_queue_cv_.wait(lock, [this]() {
return !audio_playback_queue_.empty() || service_stopped_.load();
});
if (service_stopped_.load()) {
break;
}
auto task = std::move(audio_playback_queue_.front());
audio_playback_queue_.pop_front();
output_in_flight_ = true;
audio_queue_cv_.notify_all();
lock.unlock();
@@ -304,13 +339,22 @@ void AudioService::AudioOutputTask() {
last_output_time_ = std::chrono::steady_clock::now();
debug_statistics_.playback_count++;
bool notify_drained = false;
lock.lock();
#if CONFIG_USE_SERVER_AEC
/* Record the timestamp for server AEC */
if (task->timestamp > 0) {
lock.lock();
timestamp_queue_.push_back(task->timestamp);
}
#endif
output_in_flight_ = false;
notify_drained = MarkPlaybackDrainedLocked();
audio_queue_cv_.notify_all();
lock.unlock();
if (notify_drained && callbacks_.on_playback_drained) {
callbacks_.on_playback_drained();
}
}
ESP_LOGW(TAG, "Audio output task stopped");
@@ -320,11 +364,11 @@ void AudioService::OpusCodecTask() {
while (true) {
std::unique_lock<std::mutex> lock(audio_queue_mutex_);
audio_queue_cv_.wait(lock, [this]() {
return service_stopped_ ||
(!audio_encode_queue_.empty() && audio_send_queue_.size() < MAX_SEND_PACKETS_IN_QUEUE) ||
return service_stopped_.load() ||
!audio_encode_queue_.empty() ||
(!audio_decode_queue_.empty() && audio_playback_queue_.size() < MAX_PLAYBACK_TASKS_IN_QUEUE);
});
if (service_stopped_) {
if (service_stopped_.load()) {
break;
}
@@ -332,6 +376,8 @@ void AudioService::OpusCodecTask() {
if (!audio_decode_queue_.empty() && audio_playback_queue_.size() < MAX_PLAYBACK_TASKS_IN_QUEUE) {
auto packet = std::move(audio_decode_queue_.front());
audio_decode_queue_.pop_front();
decode_in_flight_ = true;
const uint32_t generation = playback_generation_;
audio_queue_cv_.notify_all();
lock.unlock();
@@ -340,6 +386,7 @@ void AudioService::OpusCodecTask() {
task->timestamp = packet->timestamp;
SetDecodeSampleRate(packet->sample_rate, packet->frame_duration);
bool decoded = false;
if (opus_decoder_ != nullptr) {
task->pcm.resize(decoder_frame_size_);
esp_audio_dec_in_raw_t raw = {
@@ -369,22 +416,30 @@ void AudioService::OpusCodecTask() {
resampled.resize(actual_output);
task->pcm = std::move(resampled);
}
lock.lock();
audio_playback_queue_.push_back(std::move(task));
audio_queue_cv_.notify_all();
debug_statistics_.decode_count++;
decoded = true;
} else {
ESP_LOGE(TAG, "Failed to decode audio after resize, error code: %d", ret);
lock.lock();
}
} else {
ESP_LOGE(TAG, "Audio decoder is not configured");
lock.lock();
}
lock.lock();
if (decoded && generation == playback_generation_ && !service_stopped_.load()) {
audio_playback_queue_.push_back(std::move(task));
}
decode_in_flight_ = false;
debug_statistics_.decode_count++;
const bool notify_drained = MarkPlaybackDrainedLocked();
audio_queue_cv_.notify_all();
lock.unlock();
if (notify_drained && callbacks_.on_playback_drained) {
callbacks_.on_playback_drained();
}
lock.lock();
}
/* Encode the audio to send queue */
if (!audio_encode_queue_.empty() && audio_send_queue_.size() < MAX_SEND_PACKETS_IN_QUEUE) {
if (!audio_encode_queue_.empty()) {
auto task = std::move(audio_encode_queue_.front());
audio_encode_queue_.pop_front();
audio_queue_cv_.notify_all();
@@ -413,6 +468,11 @@ void AudioService::OpusCodecTask() {
if (task->type == kAudioTaskTypeEncodeToSendQueue) {
{
std::lock_guard<std::mutex> lock2(audio_queue_mutex_);
/* Never let a full send queue stall encoding: stale realtime
* audio is useless to the server, so drop the oldest packet. */
if (audio_send_queue_.size() >= MAX_SEND_PACKETS_IN_QUEUE) {
audio_send_queue_.pop_front();
}
audio_send_queue_.push_back(std::move(packet));
}
if (callbacks_.on_send_queue_available) {
@@ -477,33 +537,62 @@ void AudioService::PushTaskToEncodeQueue(AudioTaskType type, std::vector<int16_t
auto task = std::make_unique<AudioTask>();
task->type = type;
task->pcm = std::move(pcm);
/* Push the task to the encode queue */
std::unique_lock<std::mutex> lock(audio_queue_mutex_);
/* If the task is to send queue, we need to set the timestamp */
if (type == kAudioTaskTypeEncodeToSendQueue && !timestamp_queue_.empty()) {
if (timestamp_queue_.size() <= MAX_TIMESTAMPS_IN_QUEUE) {
task->timestamp = timestamp_queue_.front();
} else {
ESP_LOGW(TAG, "Timestamp queue (%u) is full, dropping timestamp", timestamp_queue_.size());
uint32_t dropped_total = 0;
{
/* Push the task to the encode queue */
std::unique_lock<std::mutex> lock(audio_queue_mutex_);
/* If the task is to send queue, we need to set the timestamp */
if (type == kAudioTaskTypeEncodeToSendQueue && !timestamp_queue_.empty()) {
if (timestamp_queue_.size() <= MAX_TIMESTAMPS_IN_QUEUE) {
task->timestamp = timestamp_queue_.front();
} else {
ESP_LOGW(TAG, "Timestamp queue (%u) is full, dropping timestamp", timestamp_queue_.size());
}
timestamp_queue_.pop_front();
}
timestamp_queue_.pop_front();
/* Microphone audio is realtime, so drop the oldest frame instead of blocking.
* Blocking here would stall the audio engine task (AFE fetch) and deadlock the
* whole input pipeline when the send queue stops being drained (e.g. network
* congestion or a failed UDP send). */
if (audio_encode_queue_.size() >= MAX_ENCODE_TASKS_IN_QUEUE) {
audio_encode_queue_.pop_front();
dropped_total = ++debug_statistics_.encode_drop_count;
}
audio_encode_queue_.push_back(std::move(task));
audio_queue_cv_.notify_all();
}
audio_queue_cv_.wait(lock, [this]() { return audio_encode_queue_.size() < MAX_ENCODE_TASKS_IN_QUEUE; });
audio_encode_queue_.push_back(std::move(task));
audio_queue_cv_.notify_all();
/* Log outside the lock (UART writes are slow and would starve the codec task),
* at most once per second. */
if (dropped_total > 0) {
int64_t now = esp_timer_get_time();
if (now - last_encode_drop_log_time_ >= 1000000) {
last_encode_drop_log_time_ = now;
ESP_LOGW(TAG, "Encode queue is full, dropping oldest frame (dropped %lu so far)",
(unsigned long)dropped_total);
}
}
}
bool AudioService::PushPacketToDecodeQueue(std::unique_ptr<AudioStreamPacket> packet, bool wait) {
std::unique_lock<std::mutex> lock(audio_queue_mutex_);
if (audio_decode_queue_.size() >= MAX_DECODE_PACKETS_IN_QUEUE) {
if (wait) {
audio_queue_cv_.wait(lock, [this]() { return audio_decode_queue_.size() < MAX_DECODE_PACKETS_IN_QUEUE; });
audio_queue_cv_.wait(lock, [this]() {
return service_stopped_.load() ||
audio_decode_queue_.size() < MAX_DECODE_PACKETS_IN_QUEUE;
});
} else {
return false;
}
}
if (service_stopped_.load()) {
return false;
}
playback_drained_notified_ = false;
audio_decode_queue_.push_back(std::move(packet));
audio_queue_cv_.notify_all();
return true;
@@ -596,6 +685,9 @@ void AudioService::EnableAudioTesting(bool enable) {
/* Copy audio_testing_queue_ to audio_decode_queue_ */
std::lock_guard<std::mutex> lock(audio_queue_mutex_);
audio_decode_queue_ = std::move(audio_testing_queue_);
if (!audio_decode_queue_.empty()) {
playback_drained_notified_ = false;
}
audio_queue_cv_.notify_all();
}
}
@@ -640,28 +732,47 @@ void AudioService::PlaySound(const std::string_view& ogg) {
bool AudioService::IsIdle() {
std::lock_guard<std::mutex> lock(audio_queue_mutex_);
return audio_encode_queue_.empty() && audio_decode_queue_.empty() && audio_playback_queue_.empty() && audio_testing_queue_.empty();
return audio_encode_queue_.empty() && IsPlaybackDrainedLocked() && audio_testing_queue_.empty();
}
void AudioService::WaitForPlaybackQueueEmpty() {
std::unique_lock<std::mutex> lock(audio_queue_mutex_);
audio_queue_cv_.wait(lock, [this]() {
return service_stopped_ || (audio_decode_queue_.empty() && audio_playback_queue_.empty());
});
bool AudioService::IsPlaybackIdle() {
std::lock_guard<std::mutex> lock(audio_queue_mutex_);
return IsPlaybackDrainedLocked();
}
void AudioService::ResetDecoder() {
std::lock_guard<std::mutex> lock(audio_queue_mutex_);
std::unique_lock<std::mutex> decoder_lock(decoder_mutex_);
if (opus_decoder_ != nullptr) {
esp_opus_dec_reset(opus_decoder_);
bool notify_drained = false;
{
std::lock_guard<std::mutex> lock(audio_queue_mutex_);
++playback_generation_;
std::unique_lock<std::mutex> decoder_lock(decoder_mutex_);
if (opus_decoder_ != nullptr) {
esp_opus_dec_reset(opus_decoder_);
}
decoder_lock.unlock();
timestamp_queue_.clear();
audio_decode_queue_.clear();
audio_playback_queue_.clear();
audio_testing_queue_.clear();
notify_drained = MarkPlaybackDrainedLocked();
audio_queue_cv_.notify_all();
}
decoder_lock.unlock();
timestamp_queue_.clear();
audio_decode_queue_.clear();
audio_playback_queue_.clear();
audio_testing_queue_.clear();
audio_queue_cv_.notify_all();
if (notify_drained && callbacks_.on_playback_drained) {
callbacks_.on_playback_drained();
}
}
bool AudioService::IsPlaybackDrainedLocked() const {
return audio_decode_queue_.empty() && audio_playback_queue_.empty() &&
!decode_in_flight_ && !output_in_flight_;
}
bool AudioService::MarkPlaybackDrainedLocked() {
if (!IsPlaybackDrainedLocked() || playback_drained_notified_) {
return false;
}
playback_drained_notified_ = true;
return true;
}
void AudioService::CheckAndUpdateAudioPowerState() {
@@ -669,7 +780,9 @@ void AudioService::CheckAndUpdateAudioPowerState() {
auto input_elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(now - last_input_time_).count();
auto output_elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(now - last_output_time_).count();
if (input_elapsed > AUDIO_POWER_TIMEOUT_MS && codec_->input_enabled()) {
codec_->EnableInput(false);
// ADC continuous start/stop must run in the same task. Wake the audio
// input task instead of closing the codec from the esp_timer task.
xEventGroupSetBits(event_group_, AS_EVENT_AUDIO_INPUT_STOP_REQUEST);
}
if (output_elapsed > AUDIO_POWER_TIMEOUT_MS && codec_->output_enabled()) {
// Keep TX clock when duplex RX is active; otherwise RX may stall on some boards.
+16 -5
View File
@@ -2,6 +2,7 @@
#define AUDIO_SERVICE_H
#include <memory>
#include <atomic>
#include <deque>
#include <condition_variable>
#include <chrono>
@@ -49,7 +50,7 @@
#define AS_EVENT_AUDIO_TESTING_RUNNING (1 << 0)
#define AS_EVENT_WAKE_WORD_RUNNING (1 << 1)
#define AS_EVENT_AUDIO_PROCESSOR_RUNNING (1 << 2)
#define AS_EVENT_PLAYBACK_NOT_EMPTY (1 << 3)
#define AS_EVENT_AUDIO_INPUT_STOP_REQUEST (1 << 4)
#define AS_OPUS_GET_FRAME_DRU_ENUM(duration_ms) \
((duration_ms) == 5 ? ESP_OPUS_ENC_FRAME_DURATION_5_MS : \
@@ -79,6 +80,8 @@ struct AudioServiceCallbacks {
std::function<void(const std::string&)> on_wake_word_detected;
std::function<void(bool)> on_vad_change;
std::function<void(void)> on_audio_testing_queue_full;
// Fired when the decode/playback queues and their in-flight work are drained.
std::function<void(void)> on_playback_drained;
};
@@ -91,7 +94,7 @@ enum AudioTaskType {
struct AudioTask {
AudioTaskType type;
std::vector<int16_t> pcm;
uint32_t timestamp;
uint32_t timestamp = 0;
};
struct DebugStatistics {
@@ -99,6 +102,7 @@ struct DebugStatistics {
uint32_t decode_count = 0;
uint32_t encode_count = 0;
uint32_t playback_count = 0;
uint32_t encode_drop_count = 0;
};
class AudioService {
@@ -114,7 +118,7 @@ public:
const std::string& GetLastWakeWord() const;
bool IsVoiceDetected() const { return voice_detected_; }
bool IsIdle();
void WaitForPlaybackQueueEmpty();
bool IsPlaybackIdle();
bool IsWakeWordRunning() const { return xEventGroupGetBits(event_group_) & AS_EVENT_WAKE_WORD_RUNNING; }
bool IsAudioProcessorRunning() const { return xEventGroupGetBits(event_group_) & AS_EVENT_AUDIO_PROCESSOR_RUNNING; }
bool IsAfeWakeWord();
@@ -154,6 +158,7 @@ private:
int decoder_duration_ms_ = OPUS_FRAME_DURATION_MS;
int decoder_frame_size_ = 0;
DebugStatistics debug_statistics_;
int64_t last_encode_drop_log_time_ = 0;
srmodel_list_t* models_list_ = nullptr;
EventGroupHandle_t event_group_;
@@ -169,6 +174,10 @@ private:
std::deque<std::unique_ptr<AudioStreamPacket>> audio_testing_queue_;
std::deque<std::unique_ptr<AudioTask>> audio_encode_queue_;
std::deque<std::unique_ptr<AudioTask>> audio_playback_queue_;
bool decode_in_flight_ = false;
bool output_in_flight_ = false;
bool playback_drained_notified_ = true;
uint32_t playback_generation_ = 0;
// For server AEC
std::deque<uint32_t> timestamp_queue_;
@@ -179,8 +188,8 @@ private:
#else
bool device_aec_enabled_ = false;
#endif
bool service_stopped_ = true;
bool audio_input_need_warmup_ = false;
std::atomic<bool> service_stopped_{true};
std::atomic<bool> audio_input_need_warmup_{false};
esp_timer_handle_t audio_power_timer_ = nullptr;
std::chrono::steady_clock::time_point last_input_time_;
@@ -193,6 +202,8 @@ private:
bool InitializeAudioEngine();
void SetDecodeSampleRate(int sample_rate, int frame_duration);
void CheckAndUpdateAudioPowerState();
bool IsPlaybackDrainedLocked() const;
bool MarkPlaybackDrainedLocked();
};
#endif
+20 -8
View File
@@ -7,14 +7,18 @@
#define TAG "BoxAudioCodec"
BoxAudioCodec::BoxAudioCodec(void* i2c_master_handle, int input_sample_rate, int output_sample_rate,
gpio_num_t mclk, gpio_num_t bclk, gpio_num_t ws, gpio_num_t dout, gpio_num_t din,
gpio_num_t pa_pin, uint8_t es8311_addr, uint8_t es7210_addr, bool input_reference) {
duplex_ = true; // 是否双工
input_reference_ = input_reference; // 是否使用参考输入,实现回声消除
input_channels_ = input_reference_ ? 2 : 1; // 输入通道数
gpio_num_t mclk, gpio_num_t bclk, gpio_num_t ws, gpio_num_t dout,
gpio_num_t din, gpio_num_t pa_pin, uint8_t es8311_addr,
uint8_t es7210_addr, bool input_reference, float input_gain,
int reference_gain_channel, float reference_gain) {
duplex_ = true; // 是否双工
input_reference_ = input_reference; // 是否使用参考输入,实现回声消除
input_channels_ = input_reference_ ? 2 : 1; // 输入通道数
input_sample_rate_ = input_sample_rate;
output_sample_rate_ = output_sample_rate;
input_gain_ = 30;
input_gain_ = input_gain;
reference_gain_channel_ = reference_gain_channel;
reference_gain_ = reference_gain;
CreateDuplexChannels(mclk, bclk, ws, dout, din);
@@ -203,7 +207,15 @@ void BoxAudioCodec::EnableInput(bool enable) {
fs.channel_mask |= ESP_CODEC_DEV_MAKE_CHANNEL_MASK(1);
}
ESP_ERROR_CHECK(esp_codec_dev_open(input_dev_, &fs));
ESP_ERROR_CHECK(esp_codec_dev_set_in_channel_gain(input_dev_, ESP_CODEC_DEV_MAKE_CHANNEL_MASK(0), input_gain_));
ESP_ERROR_CHECK(esp_codec_dev_set_in_channel_gain(
input_dev_, ESP_CODEC_DEV_MAKE_CHANNEL_MASK(0), input_gain_));
if (input_reference_ && reference_gain_channel_ >= 0) {
// ES7210 gain masks use physical MIC numbering, which differs
// from the TDM slot order (MIC1, MIC3, MIC2, MIC4).
ESP_ERROR_CHECK(esp_codec_dev_set_in_channel_gain(
input_dev_, ESP_CODEC_DEV_MAKE_CHANNEL_MASK(reference_gain_channel_),
reference_gain_));
}
} else {
ESP_ERROR_CHECK(esp_codec_dev_close(input_dev_));
}
@@ -244,4 +256,4 @@ int BoxAudioCodec::Write(const int16_t* data, int samples) {
ESP_ERROR_CHECK_WITHOUT_ABORT(esp_codec_dev_write(output_dev_, (void*)data, samples * sizeof(int16_t)));
}
return samples;
}
}
+8 -3
View File
@@ -20,16 +20,21 @@ private:
esp_codec_dev_handle_t output_dev_ = nullptr;
esp_codec_dev_handle_t input_dev_ = nullptr;
std::mutex data_if_mutex_;
int reference_gain_channel_ = -1;
float reference_gain_ = 0.0f;
void CreateDuplexChannels(gpio_num_t mclk, gpio_num_t bclk, gpio_num_t ws, gpio_num_t dout, gpio_num_t din);
void CreateDuplexChannels(gpio_num_t mclk, gpio_num_t bclk, gpio_num_t ws, gpio_num_t dout,
gpio_num_t din);
virtual int Read(int16_t* dest, int samples) override;
virtual int Write(const int16_t* data, int samples) override;
public:
BoxAudioCodec(void* i2c_master_handle, int input_sample_rate, int output_sample_rate,
gpio_num_t mclk, gpio_num_t bclk, gpio_num_t ws, gpio_num_t dout, gpio_num_t din,
gpio_num_t pa_pin, uint8_t es8311_addr, uint8_t es7210_addr, bool input_reference);
gpio_num_t mclk, gpio_num_t bclk, gpio_num_t ws, gpio_num_t dout, gpio_num_t din,
gpio_num_t pa_pin, uint8_t es8311_addr, uint8_t es7210_addr, bool input_reference,
float input_gain = 30.0f, int reference_gain_channel = -1,
float reference_gain = 0.0f);
virtual ~BoxAudioCodec();
virtual void SetOutputVolume(int volume) override;
+17 -2
View File
@@ -1,6 +1,8 @@
#include "es8311_audio_codec.h"
#include <esp_log.h>
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#define TAG "Es8311AudioCodec"
@@ -36,6 +38,7 @@ Es8311AudioCodec::Es8311AudioCodec(void* i2c_master_handle, i2c_port_t i2c_port,
};
ctrl_if_ = audio_codec_new_i2c_ctrl(&i2c_cfg);
assert(ctrl_if_ != NULL);
ResetCodec();
gpio_if_ = audio_codec_new_gpio();
assert(gpio_if_ != NULL);
@@ -58,6 +61,17 @@ Es8311AudioCodec::Es8311AudioCodec(void* i2c_master_handle, i2c_port_t i2c_port,
}
}
void Es8311AudioCodec::ResetCodec() {
// Hold the ES8311 digital blocks in reset for several milliseconds, as
// recommended by the initialization guide. Normal codec initialization
// releases the reset and starts the state machine.
uint8_t reset_value = 0x1F;
ESP_ERROR_CHECK(static_cast<esp_err_t>(
ctrl_if_->write_reg(ctrl_if_, 0x00, 1, &reset_value, 1)));
vTaskDelay(pdMS_TO_TICKS(5));
ESP_LOGI(TAG, "ES8311 software reset complete");
}
Es8311AudioCodec::~Es8311AudioCodec() {
esp_codec_dev_delete(dev_);
@@ -88,7 +102,8 @@ void Es8311AudioCodec::UpdateDeviceState() {
ESP_ERROR_CHECK(esp_codec_dev_set_in_gain(dev_, input_gain_));
ESP_ERROR_CHECK(esp_codec_dev_set_out_vol(dev_, output_volume_));
} else if (!input_enabled_ && !output_enabled_ && dev_ != nullptr) {
esp_codec_dev_close(dev_);
ESP_ERROR_CHECK(esp_codec_dev_close(dev_));
esp_codec_dev_delete(dev_);
dev_ = nullptr;
}
if (pa_pin_ != GPIO_NUM_NC) {
@@ -199,4 +214,4 @@ int Es8311AudioCodec::Write(const int16_t* data, int samples) {
ESP_ERROR_CHECK_WITHOUT_ABORT(esp_codec_dev_write(dev_, (void*)data, samples * sizeof(int16_t)));
}
return samples;
}
}
+2 -1
View File
@@ -23,6 +23,7 @@ private:
std::mutex data_if_mutex_;
void CreateDuplexChannels(gpio_num_t mclk, gpio_num_t bclk, gpio_num_t ws, gpio_num_t dout, gpio_num_t din);
void ResetCodec();
void UpdateDeviceState();
virtual int Read(int16_t* dest, int samples) override;
@@ -39,4 +40,4 @@ public:
virtual void EnableOutput(bool enable) override;
};
#endif // _ES8311_AUDIO_CODEC_H
#endif // _ES8311_AUDIO_CODEC_H
+1 -1
View File
@@ -3,7 +3,7 @@
#include "audio_codec.h"
#include <driver/i2c.h>
#include <driver/i2c_master.h>
#include <driver/gpio.h>
#include <esp_codec_dev.h>
#include <esp_codec_dev_defaults.h>
+18 -9
View File
@@ -176,14 +176,17 @@ void Es8388AudioCodec::EnableOutput(bool enable) {
return;
}
if (enable) {
esp_codec_dev_sample_info_t fs = {
.bits_per_sample = 16,
.channel = 1,
.channel_mask = 0,
.sample_rate = (uint32_t)output_sample_rate_,
.mclk_multiple = 0,
};
ESP_ERROR_CHECK(esp_codec_dev_open(output_dev_, &fs));
if (!output_device_opened_) {
esp_codec_dev_sample_info_t fs = {
.bits_per_sample = 16,
.channel = 1,
.channel_mask = 0,
.sample_rate = (uint32_t)output_sample_rate_,
.mclk_multiple = 0,
};
ESP_ERROR_CHECK(esp_codec_dev_open(output_dev_, &fs));
output_device_opened_ = true;
}
ESP_ERROR_CHECK(esp_codec_dev_set_out_vol(output_dev_, output_volume_));
// Set analog output volume to 0dB, default is -45dB
@@ -193,11 +196,17 @@ void Es8388AudioCodec::EnableOutput(bool enable) {
ctrl_if_->write_reg(ctrl_if_, reg, 1, &reg_val, 1);
}
ESP_ERROR_CHECK(esp_codec_dev_set_out_mute(output_dev_, false));
if (pa_pin_ != GPIO_NUM_NC) {
gpio_set_level(pa_pin_, 1);
}
} else {
ESP_ERROR_CHECK(esp_codec_dev_close(output_dev_));
// Keep TX open because the paired RX channel may use it as the clock
// source. Reopening the shared duplex interface after WiFi provisioning
// can otherwise leave playback silent on ESP-IDF 6.
if (output_device_opened_) {
ESP_ERROR_CHECK(esp_codec_dev_set_out_mute(output_dev_, true));
}
if (pa_pin_ != GPIO_NUM_NC) {
gpio_set_level(pa_pin_, 0);
}
+1
View File
@@ -18,6 +18,7 @@ private:
esp_codec_dev_handle_t output_dev_ = nullptr;
esp_codec_dev_handle_t input_dev_ = nullptr;
bool output_device_opened_ = false;
gpio_num_t pa_pin_ = GPIO_NUM_NC;
std::mutex data_if_mutex_;
+22 -11
View File
@@ -169,21 +169,32 @@ void Es8389AudioCodec::EnableOutput(bool enable) {
return;
}
if (enable) {
// Play 16bit 1 channel
esp_codec_dev_sample_info_t fs = {
.bits_per_sample = 16,
.channel = 1,
.channel_mask = 0,
.sample_rate = (uint32_t)output_sample_rate_,
.mclk_multiple = 0,
};
ESP_ERROR_CHECK(esp_codec_dev_open(output_dev_, &fs));
if (!output_device_opened_) {
// Play 16bit 1 channel
esp_codec_dev_sample_info_t fs = {
.bits_per_sample = 16,
.channel = 1,
.channel_mask = 0,
.sample_rate = (uint32_t)output_sample_rate_,
.mclk_multiple = 0,
};
ESP_ERROR_CHECK(esp_codec_dev_open(output_dev_, &fs));
output_device_opened_ = true;
}
ESP_ERROR_CHECK(esp_codec_dev_set_out_vol(output_dev_, output_volume_));
ESP_ERROR_CHECK(esp_codec_dev_set_out_mute(output_dev_, false));
if (pa_pin_ != GPIO_NUM_NC) {
gpio_set_level(pa_pin_, 1);
}
} else {
ESP_ERROR_CHECK(esp_codec_dev_close(output_dev_));
// Keep the ES8389 TX data interface open. On ESP32-S3 the RX channel
// uses the paired TX channel as its clock source, and reopening TX via
// esp_codec_dev after it has been closed can leave the IDF 6 duplex
// channel silent even though esp_codec_dev_open reports success.
// Muting preserves the logical power state without reconfiguring I2S.
if (output_device_opened_) {
ESP_ERROR_CHECK(esp_codec_dev_set_out_mute(output_dev_, true));
}
if (pa_pin_ != GPIO_NUM_NC) {
gpio_set_level(pa_pin_, 0);
}
@@ -203,4 +214,4 @@ int Es8389AudioCodec::Write(const int16_t* data, int samples) {
ESP_ERROR_CHECK_WITHOUT_ABORT(esp_codec_dev_write(output_dev_, (void*)data, samples * sizeof(int16_t)));
}
return samples;
}
}
+2 -1
View File
@@ -3,7 +3,7 @@
#include "audio_codec.h"
#include <driver/i2c.h>
#include <driver/i2c_master.h>
#include <driver/gpio.h>
#include <esp_codec_dev.h>
#include <esp_codec_dev_defaults.h>
@@ -18,6 +18,7 @@ private:
esp_codec_dev_handle_t output_dev_ = nullptr;
esp_codec_dev_handle_t input_dev_ = nullptr;
bool output_device_opened_ = false;
gpio_num_t pa_pin_ = GPIO_NUM_NC;
std::mutex data_if_mutex_;
+7 -7
View File
@@ -22,7 +22,7 @@ NoAudioCodecDuplex::NoAudioCodecDuplex(int input_sample_rate, int output_sample_
output_sample_rate_ = output_sample_rate;
i2s_chan_config_t chan_cfg = {
.id = I2S_NUM_0,
.id = XIAOZHI_I2S_PORT(0),
.role = I2S_ROLE_MASTER,
.dma_desc_num = AUDIO_CODEC_DMA_DESC_NUM,
.dma_frame_num = AUDIO_CODEC_DMA_FRAME_NUM,
@@ -83,7 +83,7 @@ NoAudioCodecSimplex::NoAudioCodecSimplex(int input_sample_rate, int output_sampl
// Create a new channel for speaker
i2s_chan_config_t chan_cfg = {
.id = (i2s_port_t)0,
.id = XIAOZHI_I2S_PORT(0),
.role = I2S_ROLE_MASTER,
.dma_desc_num = AUDIO_CODEC_DMA_DESC_NUM,
.dma_frame_num = AUDIO_CODEC_DMA_FRAME_NUM,
@@ -134,7 +134,7 @@ NoAudioCodecSimplex::NoAudioCodecSimplex(int input_sample_rate, int output_sampl
ESP_ERROR_CHECK(i2s_channel_init_std_mode(tx_handle_, &std_cfg));
// Create a new channel for MIC
chan_cfg.id = (i2s_port_t)1;
chan_cfg.id = XIAOZHI_I2S_PORT(1);
ESP_ERROR_CHECK(i2s_new_channel(&chan_cfg, nullptr, &rx_handle_));
std_cfg.clk_cfg.sample_rate_hz = (uint32_t)input_sample_rate_;
std_cfg.gpio_cfg.bclk = mic_sck;
@@ -152,7 +152,7 @@ NoAudioCodecSimplex::NoAudioCodecSimplex(int input_sample_rate, int output_sampl
// Create a new channel for speaker
i2s_chan_config_t chan_cfg = {
.id = (i2s_port_t)0,
.id = XIAOZHI_I2S_PORT(0),
.role = I2S_ROLE_MASTER,
.dma_desc_num = AUDIO_CODEC_DMA_DESC_NUM,
.dma_frame_num = AUDIO_CODEC_DMA_FRAME_NUM,
@@ -203,7 +203,7 @@ NoAudioCodecSimplex::NoAudioCodecSimplex(int input_sample_rate, int output_sampl
ESP_ERROR_CHECK(i2s_channel_init_std_mode(tx_handle_, &std_cfg));
// Create a new channel for MIC
chan_cfg.id = (i2s_port_t)1;
chan_cfg.id = XIAOZHI_I2S_PORT(1);
ESP_ERROR_CHECK(i2s_new_channel(&chan_cfg, nullptr, &rx_handle_));
std_cfg.clk_cfg.sample_rate_hz = (uint32_t)input_sample_rate_;
std_cfg.slot_cfg.slot_mask = mic_slot_mask;
@@ -293,7 +293,7 @@ NoAudioCodecSimplexPdm::NoAudioCodecSimplexPdm(int input_sample_rate, int output
output_sample_rate_ = output_sample_rate;
// Create a new channel for speaker
i2s_chan_config_t tx_chan_cfg = I2S_CHANNEL_DEFAULT_CONFIG((i2s_port_t)1, I2S_ROLE_MASTER);
i2s_chan_config_t tx_chan_cfg = I2S_CHANNEL_DEFAULT_CONFIG(XIAOZHI_I2S_PORT(1), I2S_ROLE_MASTER);
tx_chan_cfg.dma_desc_num = AUDIO_CODEC_DMA_DESC_NUM;
tx_chan_cfg.dma_frame_num = AUDIO_CODEC_DMA_FRAME_NUM;
tx_chan_cfg.auto_clear_after_cb = true;
@@ -343,7 +343,7 @@ NoAudioCodecSimplexPdm::NoAudioCodecSimplexPdm(int input_sample_rate, int output
ESP_ERROR_CHECK(i2s_channel_init_std_mode(tx_handle_, &tx_std_cfg));
#if SOC_I2S_SUPPORTS_PDM_RX
// Create a new channel for MIC in PDM mode
i2s_chan_config_t rx_chan_cfg = I2S_CHANNEL_DEFAULT_CONFIG((i2s_port_t)0, I2S_ROLE_MASTER);
i2s_chan_config_t rx_chan_cfg = I2S_CHANNEL_DEFAULT_CONFIG(XIAOZHI_I2S_PORT(0), I2S_ROLE_MASTER);
ESP_ERROR_CHECK(i2s_new_channel(&rx_chan_cfg, NULL, &rx_handle_));
i2s_pdm_rx_config_t pdm_rx_cfg = {
.clk_cfg = I2S_PDM_RX_CLK_DEFAULT_CONFIG((uint32_t)input_sample_rate_),
+70 -20
View File
@@ -126,14 +126,14 @@ bool AfeAudioEngine::Initialize(AudioCodec* codec, int frame_duration_ms, srmode
? nullptr
: esp_srmodel_filter(models_, ESP_VADN_PREFIX, nullptr);
afe_config_t* afe_config = afe_config_init(
input_format.c_str(), models_, AFE_TYPE_FD, AFE_MODE_LOW_COST);
input_format.c_str(), models_, AFE_TYPE_VC, AFE_MODE_HIGH_PERF);
if (afe_config == nullptr) {
ESP_LOGE(TAG, "Failed to create AFE configuration");
return false;
}
afe_config->aec_init = codec_->input_reference();
afe_config->aec_mode = AEC_MODE_FD_LOW_COST;
afe_config->aec_mode = AEC_MODE_VOIP_HIGH_PERF;
afe_config->aec_nlp_level = AEC_NLP_LEVEL_VERYAGGR;
afe_config->ns_init = false;
afe_config->vad_init = kUseAfeForVoiceProcessing;
@@ -217,18 +217,16 @@ void AfeAudioEngine::EnableWakeWordDetection(bool enable) {
return;
}
// WakeNet enable/disable on the AFE instance is applied by ProcessingTask
// (see ApplyAfeControls), driven by the kWakeWordEnabled bit.
if (enable) {
if (wake_detector_ == WakeDetector::kWakeNet) {
afe_iface_->enable_wakenet(afe_data_);
} else {
if (wake_detector_ == WakeDetector::kMultiNet) {
custom_wake_word_->Start();
}
xEventGroupSetBits(event_group_, kWakeWordEnabled);
} else {
xEventGroupClearBits(event_group_, kWakeWordEnabled);
if (wake_detector_ == WakeDetector::kWakeNet) {
afe_iface_->disable_wakenet(afe_data_);
} else {
if (wake_detector_ == WakeDetector::kMultiNet) {
custom_wake_word_->Stop();
}
}
@@ -246,7 +244,7 @@ void AfeAudioEngine::EnableVoiceProcessing(bool enable) {
}
void AfeAudioEngine::EnableDeviceAec(bool enable) {
device_aec_enabled_ = enable;
device_aec_enabled_.store(enable);
if (enable && (codec_ == nullptr || !codec_->input_reference())) {
ESP_LOGW(TAG, "Device AEC requires a playback reference channel");
}
@@ -290,39 +288,84 @@ void AfeAudioEngine::UpdateActiveState() {
xEventGroupSetBits(event_group_, kAfeActive);
} else {
xEventGroupClearBits(event_group_, kAfeActive);
control_generation_.fetch_add(1);
std::lock_guard<std::mutex> lock(input_buffer_mutex_);
input_buffer_.clear();
if (afe_data_ != nullptr) {
afe_iface_->reset_buffer(afe_data_);
// Don't call reset_buffer() here: this runs in the main task while
// ProcessingTask may be inside fetch_with_delay() on the same AFE
// instance, and a concurrent reset corrupts the ring buffer state.
// Defer the reset to ProcessingTask, which owns the fetch side.
reset_pending_ = true;
}
}
if ((bits & kVoiceProcessingEnabled) == 0) {
output_buffer_.clear();
// output_buffer_ is owned by the task that produces output frames,
// so let that task clear it instead of racing with it here.
output_reset_pending_ = true;
}
UpdateAecState();
afe_control_dirty_ = true;
}
void AfeAudioEngine::UpdateAecState() {
if (afe_data_ == nullptr || codec_ == nullptr || !codec_->input_reference()) {
return;
}
afe_control_dirty_ = true;
}
void AfeAudioEngine::ApplyAfeControls() {
EventBits_t bits = xEventGroupGetBits(event_group_);
const bool enable = (bits & kWakeWordEnabled) ||
(device_aec_enabled_ && (bits & kVoiceProcessingEnabled));
if (enable) {
afe_iface_->enable_aec(afe_data_);
} else {
afe_iface_->disable_aec(afe_data_);
if (wake_detector_ == WakeDetector::kWakeNet) {
if (bits & kWakeWordEnabled) {
afe_iface_->enable_wakenet(afe_data_);
} else {
afe_iface_->disable_wakenet(afe_data_);
}
}
if (codec_->input_reference()) {
const bool enable_aec = (bits & kWakeWordEnabled) ||
(device_aec_enabled_.load() && (bits & kVoiceProcessingEnabled));
if (enable_aec) {
afe_iface_->enable_aec(afe_data_);
} else {
afe_iface_->disable_aec(afe_data_);
}
}
}
void AfeAudioEngine::ApplyPendingReset() {
if (!reset_pending_.exchange(false)) {
return;
}
// Discard audio recorded before (re)activation. Holding input_buffer_mutex_
// serializes the reset against Feed(); fetch/reset both run in this task.
std::lock_guard<std::mutex> lock(input_buffer_mutex_);
input_buffer_.clear();
afe_iface_->reset_buffer(afe_data_);
}
void AfeAudioEngine::ProcessingTask() {
while (true) {
xEventGroupWaitBits(event_group_, kAfeActive, pdFALSE, pdTRUE, portMAX_DELAY);
auto* result = afe_iface_->fetch_with_delay(afe_data_, portMAX_DELAY);
ApplyPendingReset();
if ((xEventGroupGetBits(event_group_) & kAfeActive) == 0) {
continue;
}
if (afe_control_dirty_.exchange(false)) {
// WakeNet/AEC toggles are not safe against a concurrent fetch,
// so they are applied here, in the task that owns the fetch side.
ApplyAfeControls();
}
const uint32_t generation = control_generation_.load();
auto* result = afe_iface_->fetch_with_delay(afe_data_, portMAX_DELAY);
if (generation != control_generation_.load() ||
(xEventGroupGetBits(event_group_) & kAfeActive) == 0) {
// A disable/re-enable may make an old blocked fetch return after the
// AFE is active again. Reset immediately and never process that frame.
ApplyPendingReset();
continue;
}
if (result == nullptr || result->ret_value == ESP_FAIL) {
if (result != nullptr) {
ESP_LOGW(TAG, "AFE fetch failed: %d", result->ret_value);
@@ -362,7 +405,8 @@ void AfeAudioEngine::HandleWakeWordResult(const afe_fetch_result_t* result) {
last_detected_wake_word_ = wake_words_[model_index];
xEventGroupClearBits(event_group_, kWakeWordEnabled);
afe_iface_->disable_wakenet(afe_data_);
// UpdateActiveState marks the AFE controls dirty; the next loop iteration
// of ProcessingTask disables WakeNet via ApplyAfeControls.
UpdateActiveState();
if (wake_word_detected_callback_) {
wake_word_detected_callback_(last_detected_wake_word_);
@@ -370,6 +414,9 @@ void AfeAudioEngine::HandleWakeWordResult(const afe_fetch_result_t* result) {
}
void AfeAudioEngine::HandleVoiceResult(const afe_fetch_result_t* result) {
if (output_reset_pending_.exchange(false)) {
output_buffer_.clear();
}
if (vad_state_change_callback_) {
if (result->vad_state == VAD_SPEECH && !is_speaking_) {
is_speaking_ = true;
@@ -402,6 +449,9 @@ void AfeAudioEngine::OutputRawAudio(const std::vector<int16_t>& data) {
if (!output_callback_ || codec_ == nullptr) {
return;
}
if (output_reset_pending_.exchange(false)) {
output_buffer_.clear();
}
const size_t channels = codec_->input_channels();
if (channels <= 1) {
output_buffer_.insert(output_buffer_.end(), data.begin(), data.end());
+13 -1
View File
@@ -1,6 +1,7 @@
#ifndef AFE_AUDIO_ENGINE_H
#define AFE_AUDIO_ENGINE_H
#include <atomic>
#include <condition_variable>
#include <deque>
#include <functional>
@@ -65,7 +66,16 @@ private:
TaskHandle_t processing_task_ = nullptr;
int frame_samples_ = 0;
bool is_speaking_ = false;
bool device_aec_enabled_ = false;
std::atomic<bool> device_aec_enabled_{false};
// Deferred AFE buffer reset, performed by ProcessingTask (see UpdateActiveState)
std::atomic<bool> reset_pending_{false};
// Deferred WakeNet/AEC toggles, applied by ProcessingTask (see ApplyAfeControls)
std::atomic<bool> afe_control_dirty_{false};
// Deferred output_buffer_ clear, performed by the output-producing task
std::atomic<bool> output_reset_pending_{false};
// Incremented whenever an active AFE session is invalidated. ProcessingTask
// uses it to reject a fetch result produced before a disable/re-enable cycle.
std::atomic<uint32_t> control_generation_{0};
WakeDetector wake_detector_ = WakeDetector::kNone;
std::unique_ptr<CustomWakeWord> custom_wake_word_;
@@ -90,6 +100,8 @@ private:
void ProcessingTask();
void UpdateActiveState();
void UpdateAecState();
void ApplyAfeControls();
void ApplyPendingReset();
void OutputRawAudio(const std::vector<int16_t>& data);
void HandleWakeWordResult(const afe_fetch_result_t* result);
void HandleVoiceResult(const afe_fetch_result_t* result);
+2
View File
@@ -2,6 +2,8 @@
#define CUSTOM_WAKE_WORD_H
#include <esp_attr.h>
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#include <esp_mn_iface.h>
#include <esp_mn_models.h>
#include <model_path.h>
-12
View File
@@ -1,12 +0,0 @@
{
"target": "esp32s3",
"builds": [
{
"name": "aipi-lite",
"sdkconfig_append": [
"CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y",
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/16m.csv\""
]
}
]
}
-17
View File
@@ -1,17 +0,0 @@
{
"target": "esp32s3",
"builds": [
{
"name": "aipi-lite_en",
"sdkconfig_append": [
"CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y",
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/16m.csv\"",
"CONFIG_LANGUAGE_EN_US=y",
"CONFIG_SR_WN_WN9_NIHAOXIAOZHI_TTS=n",
"CONFIG_SR_WN_WN9_JARVIS_TTS=y",
"CONFIG_SR_WN_WN9_SOPHIA_TTS=y"
]
}
]
}
@@ -192,8 +192,7 @@ private:
},
.bus_width = 8,
.max_transfer_bytes = DISPLAY_WIDTH * DISPLAY_HEIGHT * sizeof(uint16_t),
.psram_trans_align = 64,
.sram_trans_align = 4,
.dma_burst_size = 64,
};
ESP_ERROR_CHECK(esp_lcd_new_i80_bus(&bus_config, &i80_bus));
@@ -217,11 +216,10 @@ private:
};
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i80(i80_bus, &io_config, &panel_io));
esp_lcd_panel_dev_config_t panel_config = {
.reset_gpio_num = LCD_NUM_RST,
.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_RGB,
.bits_per_pixel = 16,
};
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_RGB;
panel_config.bits_per_pixel = 16;
panel_config.reset_gpio_num = LCD_NUM_RST;
ESP_ERROR_CHECK(esp_lcd_new_panel_st7789(panel_io, &panel_config, &panel));
esp_lcd_panel_reset(panel);
@@ -1,4 +1,6 @@
{
"manufacturer": "alientek",
"type": "atk-dnesp32s3-box",
"target": "esp32s3",
"builds": [
{
@@ -8,4 +10,4 @@
]
}
]
}
}
@@ -1,4 +1,6 @@
{
"manufacturer": "alientek",
"type": "atk-dnesp32s3-box0",
"target": "esp32s3",
"builds": [
{
@@ -6,4 +8,4 @@
"sdkconfig_append": []
}
]
}
}
@@ -347,8 +347,7 @@ private:
},
.bus_width = 8,
.max_transfer_bytes = DISPLAY_WIDTH * DISPLAY_HEIGHT * sizeof(uint16_t),
.psram_trans_align = 64,
.sram_trans_align = 4,
.dma_burst_size = 64,
};
ESP_ERROR_CHECK(esp_lcd_new_i80_bus(&bus_config, &i80_bus));
@@ -374,11 +373,10 @@ private:
};
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i80(i80_bus, &io_config, &panel_io));
esp_lcd_panel_dev_config_t panel_config = {
.reset_gpio_num = LCD_PIN_RST,
.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_RGB,
.bits_per_pixel = 16,
};
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_RGB;
panel_config.bits_per_pixel = 16;
panel_config.reset_gpio_num = LCD_PIN_RST;
ESP_ERROR_CHECK(esp_lcd_new_panel_st7789(panel_io, &panel_config, &panel));
esp_lcd_panel_reset(panel);
@@ -437,7 +435,8 @@ public:
AUDIO_I2S_GPIO_DOUT,
AUDIO_I2S_GPIO_DIN,
GPIO_NUM_NC,
AUDIO_CODEC_ES8389_ADDR
AUDIO_CODEC_ES8389_ADDR,
AUDIO_CODEC_USE_MCLK
);
return &audio_codec;
}
@@ -9,8 +9,8 @@ enum PowerSupply {
kDeviceBatterySupply,
};
#define AUDIO_INPUT_SAMPLE_RATE 16000
#define AUDIO_OUTPUT_SAMPLE_RATE 16000
#define AUDIO_INPUT_SAMPLE_RATE 24000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_38
#define AUDIO_I2S_GPIO_WS GPIO_NUM_42
@@ -21,6 +21,7 @@ enum PowerSupply {
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_48
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_47
#define AUDIO_CODEC_ES8389_ADDR ES8389_CODEC_DEFAULT_ADDR
#define AUDIO_CODEC_USE_MCLK true
#define SPISD_PIN_MOSI GPIO_NUM_16
#define SPISD_PIN_MISO GPIO_NUM_18
@@ -77,4 +78,3 @@ enum PowerSupply {
#define Module_4G_TX_PIN GPIO_NUM_43
#endif // _BOARD_CONFIG_H_
@@ -1,4 +1,6 @@
{
"manufacturer": "alientek",
"type": "atk-dnesp32s3-box2-4g",
"target": "esp32s3",
"builds": [
{
@@ -6,4 +8,4 @@
"sdkconfig_append": []
}
]
}
}
@@ -327,8 +327,7 @@ private:
},
.bus_width = 8,
.max_transfer_bytes = DISPLAY_WIDTH * DISPLAY_HEIGHT * sizeof(uint16_t),
.psram_trans_align = 64,
.sram_trans_align = 4,
.dma_burst_size = 64,
};
ESP_ERROR_CHECK(esp_lcd_new_i80_bus(&bus_config, &i80_bus));
@@ -354,11 +353,10 @@ private:
};
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i80(i80_bus, &io_config, &panel_io));
esp_lcd_panel_dev_config_t panel_config = {
.reset_gpio_num = LCD_PIN_RST,
.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_RGB,
.bits_per_pixel = 16,
};
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_RGB;
panel_config.bits_per_pixel = 16;
panel_config.reset_gpio_num = LCD_PIN_RST;
ESP_ERROR_CHECK(esp_lcd_new_panel_st7789(panel_io, &panel_config, &panel));
esp_lcd_panel_reset(panel);
@@ -417,7 +415,7 @@ public:
AUDIO_I2S_GPIO_DIN,
GPIO_NUM_NC,
AUDIO_CODEC_ES8389_ADDR,
false);
AUDIO_CODEC_USE_MCLK);
return &audio_codec;
}
@@ -8,8 +8,8 @@ enum PowerSupply {
kDeviceBatterySupply,
};
#define AUDIO_INPUT_SAMPLE_RATE 16000
#define AUDIO_OUTPUT_SAMPLE_RATE 16000
#define AUDIO_INPUT_SAMPLE_RATE 24000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_38
#define AUDIO_I2S_GPIO_WS GPIO_NUM_42
@@ -20,6 +20,7 @@ enum PowerSupply {
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_48
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_47
#define AUDIO_CODEC_ES8389_ADDR ES8389_CODEC_DEFAULT_ADDR
#define AUDIO_CODEC_USE_MCLK true
#define R_BUTTON_GPIO GPIO_NUM_0
@@ -68,4 +69,3 @@ enum PowerSupply {
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT false
#endif // _BOARD_CONFIG_H_
@@ -1,4 +1,6 @@
{
"manufacturer": "alientek",
"type": "atk-dnesp32s3-box2-wifi",
"target": "esp32s3",
"builds": [
{
@@ -6,4 +8,4 @@
"sdkconfig_append": []
}
]
}
}
@@ -30,8 +30,8 @@
#define TAG "atk_dnesp32s3_box3"
LV_FONT_DECLARE(font_puhui_20_4);
LV_FONT_DECLARE(font_awesome_20_4);
LV_FONT_DECLARE(font_noto_sans_basic_20_4);
LV_FONT_DECLARE(font_material_symbols_20_4);
class atk_dnesp32s3_box3 : public WifiBoard {
private:
@@ -1,4 +1,6 @@
{
"manufacturer": "alientek",
"type": "atk-dnesp32s3-box3",
"target": "esp32s3",
"builds": [
{
@@ -1,4 +1,6 @@
{
"manufacturer": "alientek",
"type": "atk-dnesp32s3",
"target": "esp32s3",
"builds": [
{
@@ -10,4 +12,4 @@
]
}
]
}
}
@@ -0,0 +1,6 @@
{
"manufacturer": "alientek",
"type": "atk-dnesp32s3m-4g",
"target": "esp32s3",
"builds": []
}
@@ -0,0 +1,6 @@
{
"manufacturer": "alientek",
"type": "atk-dnesp32s3m-wifi",
"target": "esp32s3",
"builds": []
}
@@ -1,4 +1,5 @@
{
"type": "bread-compact-esp32-lcd",
"target": "esp32",
"builds": [
{
@@ -1,4 +1,5 @@
{
"type": "bread-compact-esp32",
"target": "esp32",
"builds": [
{
@@ -47,24 +47,24 @@ private:
// SSD1306 config
esp_lcd_panel_io_i2c_config_t io_config = {
.dev_addr = 0x3C,
.on_color_trans_done = nullptr,
.user_ctx = nullptr,
.scl_speed_hz = 400 * 1000,
.control_phase_bytes = 1,
.dc_bit_offset = 6,
.lcd_cmd_bits = 8,
.lcd_param_bits = 8,
.on_color_trans_done = nullptr,
.user_ctx = nullptr,
.flags = {
.dc_low_on_data = 0,
.disable_control_phase = 0,
},
.scl_speed_hz = 400 * 1000,
};
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i2c_v2(display_i2c_bus_, &io_config, &panel_io_));
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i2c(display_i2c_bus_, &io_config, &panel_io_));
ESP_LOGI(TAG, "Install SSD1306 driver");
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.reset_gpio_num = -1;
panel_config.reset_gpio_num = GPIO_NUM_NC;
panel_config.bits_per_pixel = 1;
esp_lcd_panel_ssd1306_config_t ssd1306_config = {
@@ -48,24 +48,24 @@ private:
// SSD1306 config
esp_lcd_panel_io_i2c_config_t io_config = {
.dev_addr = 0x3C,
.on_color_trans_done = nullptr,
.user_ctx = nullptr,
.scl_speed_hz = 400 * 1000,
.control_phase_bytes = 1,
.dc_bit_offset = 6,
.lcd_cmd_bits = 8,
.lcd_param_bits = 8,
.on_color_trans_done = nullptr,
.user_ctx = nullptr,
.flags = {
.dc_low_on_data = 0,
.disable_control_phase = 0,
},
.scl_speed_hz = 400 * 1000,
};
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i2c_v2(display_i2c_bus_, &io_config, &panel_io_));
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i2c(display_i2c_bus_, &io_config, &panel_io_));
ESP_LOGI(TAG, "Install SSD1306 driver");
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.reset_gpio_num = -1;
panel_config.reset_gpio_num = GPIO_NUM_NC;
panel_config.bits_per_pixel = 1;
esp_lcd_panel_ssd1306_config_t ssd1306_config = {
@@ -1,4 +1,5 @@
{
"type": "bread-compact-ml307",
"target": "esp32s3",
"builds": [
{
@@ -47,24 +47,24 @@ private:
// SSD1306 config
esp_lcd_panel_io_i2c_config_t io_config = {
.dev_addr = 0x3C,
.on_color_trans_done = nullptr,
.user_ctx = nullptr,
.scl_speed_hz = 400 * 1000,
.control_phase_bytes = 1,
.dc_bit_offset = 6,
.lcd_cmd_bits = 8,
.lcd_param_bits = 8,
.on_color_trans_done = nullptr,
.user_ctx = nullptr,
.flags = {
.dc_low_on_data = 0,
.disable_control_phase = 0,
},
.scl_speed_hz = 400 * 1000,
};
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i2c_v2(display_i2c_bus_, &io_config, &panel_io_));
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i2c(display_i2c_bus_, &io_config, &panel_io_));
ESP_LOGI(TAG, "Install SSD1306 driver");
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.reset_gpio_num = -1;
panel_config.reset_gpio_num = GPIO_NUM_NC;
panel_config.bits_per_pixel = 1;
esp_lcd_panel_ssd1306_config_t ssd1306_config = {
@@ -1,4 +1,5 @@
{
"type": "bread-compact-nt26",
"target": "esp32s3",
"builds": [
{
@@ -52,24 +52,24 @@ private:
// SSD1306 config
esp_lcd_panel_io_i2c_config_t io_config = {
.dev_addr = 0x3C,
.on_color_trans_done = nullptr,
.user_ctx = nullptr,
.scl_speed_hz = 400 * 1000,
.control_phase_bytes = 1,
.dc_bit_offset = 6,
.lcd_cmd_bits = 8,
.lcd_param_bits = 8,
.on_color_trans_done = nullptr,
.user_ctx = nullptr,
.flags = {
.dc_low_on_data = 0,
.disable_control_phase = 0,
},
.scl_speed_hz = 400 * 1000,
};
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i2c_v2(display_i2c_bus_, &io_config, &panel_io_));
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i2c(display_i2c_bus_, &io_config, &panel_io_));
ESP_LOGI(TAG, "Install SSD1306 driver");
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.reset_gpio_num = -1;
panel_config.reset_gpio_num = GPIO_NUM_NC;
panel_config.bits_per_pixel = 1;
esp_lcd_panel_ssd1306_config_t ssd1306_config = {
@@ -1,4 +1,5 @@
{
"type": "bread-compact-wifi",
"target": "esp32s3",
"builds": [
{
-375
View File
@@ -1,375 +0,0 @@
#include "afsk_demod.h"
#include <cstring>
#include <algorithm>
#include "esp_log.h"
#include "display.h"
#include "ssid_manager.h"
#ifndef M_PI
#define M_PI 3.14159265358979323846
#endif
namespace audio_wifi_config
{
static const char *kLogTag = "AUDIO_WIFI_CONFIG";
void ReceiveWifiCredentialsFromAudio(Application *app,
WifiManager *wifi_manager,
Display *display,
size_t input_channels
)
{
const int kInputSampleRate = 16000; // Input sampling rate
const float kDownsampleStep = static_cast<float>(kInputSampleRate) / static_cast<float>(kAudioSampleRate); // Downsampling step
std::vector<int16_t> audio_data;
AudioSignalProcessor signal_processor(kAudioSampleRate, kMarkFrequency, kSpaceFrequency, kBitRate, kWindowSize);
AudioDataBuffer data_buffer;
while (true)
{
// 检查Application状态,只有在WiFi配置模式下才处理音频
if (app->GetDeviceState() != kDeviceStateWifiConfiguring) {
// 不在WiFi配置状态,休眠100ms后再检查
vTaskDelay(pdMS_TO_TICKS(100));
continue;
}
if (!app->GetAudioService().ReadAudioData(audio_data, 16000, 480)) { // 16kHz, 480 samples corresponds to 30ms data
// 读取音频失败,短暂延迟后重试
ESP_LOGI(kLogTag, "Failed to read audio data, retrying.");
vTaskDelay(pdMS_TO_TICKS(10));
continue;
}
if (input_channels == 2) { // 如果是双声道输入,转换为单声道
auto mono_data = std::vector<int16_t>(audio_data.size() / 2);
for (size_t i = 0, j = 0; i < mono_data.size(); ++i, j += 2) {
mono_data[i] = audio_data[j];
}
audio_data = std::move(mono_data);
}
// Downsample the audio data
std::vector<float> downsampled_data;
size_t last_index = 0;
if (kDownsampleStep > 1.0f) {
downsampled_data.reserve(audio_data.size() / static_cast<size_t>(kDownsampleStep));
for (size_t i = 0; i < audio_data.size(); ++i) {
size_t sample_index = static_cast<size_t>(i / kDownsampleStep);
if ((sample_index + 1) > last_index) {
downsampled_data.push_back(static_cast<float>(audio_data[i]));
last_index = sample_index + 1;
}
}
} else {
downsampled_data.reserve(audio_data.size());
for (int16_t sample : audio_data) {
downsampled_data.push_back(static_cast<float>(sample));
}
}
// Process audio samples to get probability data
auto probabilities = signal_processor.ProcessAudioSamples(downsampled_data);
// Feed probability data to the data buffer
if (data_buffer.ProcessProbabilityData(probabilities, 0.5f)) {
// If complete data was received, extract WiFi credentials
if (data_buffer.decoded_text.has_value()) {
ESP_LOGI(kLogTag, "Received text data: %s", data_buffer.decoded_text->c_str());
display->SetChatMessage("system", data_buffer.decoded_text->c_str());
// Split SSID and password by newline character
std::string wifi_ssid, wifi_password;
size_t newline_position = data_buffer.decoded_text->find('\n');
if (newline_position != std::string::npos) {
wifi_ssid = data_buffer.decoded_text->substr(0, newline_position);
wifi_password = data_buffer.decoded_text->substr(newline_position + 1);
ESP_LOGI(kLogTag, "WiFi SSID: %s, Password: %s", wifi_ssid.c_str(), wifi_password.c_str());
} else {
ESP_LOGE(kLogTag, "Invalid data format, no newline character found");
continue;
}
// Save WiFi credentials using SsidManager
auto& ssid_manager = SsidManager::GetInstance();
ssid_manager.AddSsid(wifi_ssid, wifi_password);
ESP_LOGI(kLogTag, "WiFi credentials saved successfully");
// Exit config mode (triggers ConfigModeExit event)
wifi_manager->StopConfigAp();
data_buffer.decoded_text.reset(); // Clear processed data
return; // Exit the function
}
}
vTaskDelay(pdMS_TO_TICKS(1)); // 1ms delay
}
}
// Default start and end transmission identifiers
// \x01\x02 = 00000001 00000010
const std::vector<uint8_t> kDefaultStartTransmissionPattern = {
0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0};
// \x03\x04 = 00000011 00000100
const std::vector<uint8_t> kDefaultEndTransmissionPattern = {
0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0};
// FrequencyDetector implementation
FrequencyDetector::FrequencyDetector(float frequency, size_t window_size)
: frequency_(frequency), window_size_(window_size) {
frequency_bin_ = std::floor(frequency_ * static_cast<float>(window_size_));
angular_frequency_ = 2.0f * M_PI * frequency_;
cos_coefficient_ = std::cos(angular_frequency_);
sin_coefficient_ = std::sin(angular_frequency_);
filter_coefficient_ = 2.0f * cos_coefficient_;
// Initialize state buffer
state_buffer_.push_back(0.0f);
state_buffer_.push_back(0.0f);
}
void FrequencyDetector::Reset() {
state_buffer_.clear();
state_buffer_.push_back(0.0f);
state_buffer_.push_back(0.0f);
}
void FrequencyDetector::ProcessSample(float sample) {
if (state_buffer_.size() < 2) {
return;
}
float s_minus_2 = state_buffer_.front(); // S[-2]
state_buffer_.pop_front();
float s_minus_1 = state_buffer_.front(); // S[-1]
state_buffer_.pop_front();
float s_current = sample + filter_coefficient_ * s_minus_1 - s_minus_2;
state_buffer_.push_back(s_minus_1); // Put S[-1] back
state_buffer_.push_back(s_current); // Add new S[0]
}
float FrequencyDetector::GetAmplitude() const {
if (state_buffer_.size() < 2) {
return 0.0f;
}
float s_minus_1 = state_buffer_[1]; // S[-1]
float s_minus_2 = state_buffer_[0]; // S[-2]
float real_part = cos_coefficient_ * s_minus_1 - s_minus_2; // Real part
float imaginary_part = sin_coefficient_ * s_minus_1; // Imaginary part
return std::sqrt(real_part * real_part + imaginary_part * imaginary_part) /
(static_cast<float>(window_size_) / 2.0f);
}
// AudioSignalProcessor implementation
AudioSignalProcessor::AudioSignalProcessor(size_t sample_rate, size_t mark_frequency, size_t space_frequency,
size_t bit_rate, size_t window_size)
: input_buffer_size_(window_size), output_sample_count_(0) {
if (sample_rate % bit_rate != 0) {
// On ESP32 we can continue execution, but log the error
ESP_LOGW(kLogTag, "Sample rate %zu is not divisible by bit rate %zu", sample_rate, bit_rate);
}
float normalized_mark_freq = static_cast<float>(mark_frequency) / static_cast<float>(sample_rate);
float normalized_space_freq = static_cast<float>(space_frequency) / static_cast<float>(sample_rate);
mark_detector_ = std::make_unique<FrequencyDetector>(normalized_mark_freq, window_size);
space_detector_ = std::make_unique<FrequencyDetector>(normalized_space_freq, window_size);
samples_per_bit_ = sample_rate / bit_rate; // Number of samples per bit
}
std::vector<float> AudioSignalProcessor::ProcessAudioSamples(const std::vector<float> &samples) {
std::vector<float> result;
for (float sample : samples) {
if (input_buffer_.size() < input_buffer_size_) {
input_buffer_.push_back(sample); // Just add, don't process yet
} else {
// Input buffer is full, process the data
input_buffer_.pop_front(); // Remove oldest sample
input_buffer_.push_back(sample); // Add new sample
output_sample_count_++;
if (output_sample_count_ >= samples_per_bit_) {
// Process all samples in the window using Goertzel algorithm
for (float window_sample : input_buffer_) {
mark_detector_->ProcessSample(window_sample);
space_detector_->ProcessSample(window_sample);
}
float mark_amplitude = mark_detector_->GetAmplitude(); // Mark amplitude
float space_amplitude = space_detector_->GetAmplitude(); // Space amplitude
// Avoid division by zero
float mark_probability = mark_amplitude /
(space_amplitude + mark_amplitude + std::numeric_limits<float>::epsilon());
result.push_back(mark_probability);
// Reset detector windows
mark_detector_->Reset();
space_detector_->Reset();
output_sample_count_ = 0; // Reset output counter
}
}
}
return result;
}
// AudioDataBuffer implementation
AudioDataBuffer::AudioDataBuffer()
: current_state_(DataReceptionState::kInactive),
start_of_transmission_(kDefaultStartTransmissionPattern),
end_of_transmission_(kDefaultEndTransmissionPattern),
enable_checksum_validation_(true) {
identifier_buffer_size_ = std::max(start_of_transmission_.size(), end_of_transmission_.size());
max_bit_buffer_size_ = 776; // Preset bit buffer size, 776 bits = (32 + 1 + 63 + 1) * 8 = 776
bit_buffer_.reserve(max_bit_buffer_size_);
}
AudioDataBuffer::AudioDataBuffer(size_t max_byte_size, const std::vector<uint8_t> &start_identifier,
const std::vector<uint8_t> &end_identifier, bool enable_checksum)
: current_state_(DataReceptionState::kInactive),
start_of_transmission_(start_identifier),
end_of_transmission_(end_identifier),
enable_checksum_validation_(enable_checksum) {
identifier_buffer_size_ = std::max(start_of_transmission_.size(), end_of_transmission_.size());
max_bit_buffer_size_ = max_byte_size * 8; // Bit buffer size in bytes
bit_buffer_.reserve(max_bit_buffer_size_);
}
uint8_t AudioDataBuffer::CalculateChecksum(const std::string &text) {
uint8_t checksum = 0;
for (char character : text) {
checksum += static_cast<uint8_t>(character);
}
return checksum;
}
void AudioDataBuffer::ClearBuffers() {
identifier_buffer_.clear();
bit_buffer_.clear();
}
bool AudioDataBuffer::ProcessProbabilityData(const std::vector<float> &probabilities, float threshold) {
for (float probability : probabilities) {
uint8_t bit = (probability > threshold) ? 1 : 0;
if (identifier_buffer_.size() >= identifier_buffer_size_) {
identifier_buffer_.pop_front(); // Maintain buffer size
}
identifier_buffer_.push_back(bit);
// Process received bit based on state machine
switch (current_state_) {
case DataReceptionState::kInactive:
if (identifier_buffer_.size() >= start_of_transmission_.size()) {
current_state_ = DataReceptionState::kWaiting; // Enter waiting state
ESP_LOGI(kLogTag, "Entering Waiting state");
}
break;
case DataReceptionState::kWaiting:
// Waiting state, possibly waiting for transmission end
if (identifier_buffer_.size() >= start_of_transmission_.size()) {
std::vector<uint8_t> identifier_snapshot(identifier_buffer_.begin(), identifier_buffer_.end());
if (identifier_snapshot == start_of_transmission_)
{
ClearBuffers(); // Clear buffers
current_state_ = DataReceptionState::kReceiving; // Enter receiving state
ESP_LOGI(kLogTag, "Entering Receiving state");
}
}
break;
case DataReceptionState::kReceiving:
bit_buffer_.push_back(bit);
if (identifier_buffer_.size() >= end_of_transmission_.size()) {
std::vector<uint8_t> identifier_snapshot(identifier_buffer_.begin(), identifier_buffer_.end());
if (identifier_snapshot == end_of_transmission_) {
current_state_ = DataReceptionState::kInactive; // Enter inactive state
// Convert bits to bytes
std::vector<uint8_t> bytes = ConvertBitsToBytes(bit_buffer_);
uint8_t received_checksum = 0;
size_t minimum_length = 0;
if (enable_checksum_validation_) {
// If checksum is required, last byte is checksum
minimum_length = 1 + start_of_transmission_.size() / 8;
if (bytes.size() >= minimum_length)
{
received_checksum = bytes[bytes.size() - start_of_transmission_.size() / 8 - 1];
}
} else {
minimum_length = start_of_transmission_.size() / 8;
}
if (bytes.size() < minimum_length) {
ClearBuffers();
ESP_LOGW(kLogTag, "Data too short, clearing buffer");
return false; // Data too short, return failure
}
// Extract text data (remove trailing identifier part)
std::vector<uint8_t> text_bytes(
bytes.begin(), bytes.begin() + bytes.size() - minimum_length);
std::string result(text_bytes.begin(), text_bytes.end());
// Validate checksum if required
if (enable_checksum_validation_) {
uint8_t calculated_checksum = CalculateChecksum(result);
if (calculated_checksum != received_checksum) {
// Checksum mismatch
ESP_LOGW(kLogTag, "Checksum mismatch: expected %d, got %d",
received_checksum, calculated_checksum);
ClearBuffers();
return false;
}
}
ClearBuffers();
decoded_text = result;
return true; // Return success
} else if (bit_buffer_.size() >= max_bit_buffer_size_) {
// If not end identifier and bit buffer is full, reset
ClearBuffers();
ESP_LOGW(kLogTag, "Buffer overflow, clearing buffer");
current_state_ = DataReceptionState::kInactive; // Reset state machine
}
}
break;
}
}
return false;
}
std::vector<uint8_t> AudioDataBuffer::ConvertBitsToBytes(const std::vector<uint8_t> &bits) const {
std::vector<uint8_t> bytes;
// Ensure number of bits is a multiple of 8
size_t complete_bytes_count = bits.size() / 8;
bytes.reserve(complete_bytes_count);
for (size_t i = 0; i < complete_bytes_count; ++i) {
uint8_t byte_value = 0;
for (size_t j = 0; j < 8; ++j) {
byte_value |= bits[i * 8 + j] << (7 - j);
}
bytes.push_back(byte_value);
}
return bytes;
}
}
-177
View File
@@ -1,177 +0,0 @@
#pragma once
#include <vector>
#include <deque>
#include <string>
#include <memory>
#include <optional>
#include <cmath>
#include "wifi_manager.h"
#include "application.h"
// Audio signal processing constants for WiFi configuration via audio
const size_t kAudioSampleRate = 6400;
const size_t kMarkFrequency = 1800;
const size_t kSpaceFrequency = 1500;
const size_t kBitRate = 100;
const size_t kWindowSize = 64;
namespace audio_wifi_config
{
// Main function to receive WiFi credentials through audio signal
void ReceiveWifiCredentialsFromAudio(Application *app, WifiManager *wifi_manager, Display *display,
size_t input_channels = 1);
/**
* Goertzel algorithm implementation for single frequency detection
* Used to detect specific audio frequencies in the AFSK demodulation process
*/
class FrequencyDetector
{
private:
float frequency_; // Target frequency (normalized, i.e., f / fs)
size_t window_size_; // Window size for analysis
float frequency_bin_; // Frequency bin
float angular_frequency_; // Angular frequency
float cos_coefficient_; // cos(w)
float sin_coefficient_; // sin(w)
float filter_coefficient_; // 2 * cos(w)
std::deque<float> state_buffer_; // Circular buffer for storing S[-1] and S[-2]
public:
/**
* Constructor
* @param frequency Normalized frequency (f / fs)
* @param window_size Window size for analysis
*/
FrequencyDetector(float frequency, size_t window_size);
/**
* Reset the detector state
*/
void Reset();
/**
* Process one audio sample
* @param sample Input audio sample
*/
void ProcessSample(float sample);
/**
* Calculate current amplitude
* @return Amplitude value
*/
float GetAmplitude() const;
};
/**
* Audio signal processor for Mark/Space frequency pair detection
* Processes audio signals to extract digital data using AFSK demodulation
*/
class AudioSignalProcessor
{
private:
std::deque<float> input_buffer_; // Input sample buffer
size_t input_buffer_size_; // Input buffer size = window size
size_t output_sample_count_; // Output sample counter
size_t samples_per_bit_; // Samples per bit threshold
std::unique_ptr<FrequencyDetector> mark_detector_; // Mark frequency detector
std::unique_ptr<FrequencyDetector> space_detector_; // Space frequency detector
public:
/**
* Constructor
* @param sample_rate Audio sampling rate
* @param mark_frequency Mark frequency for digital '1'
* @param space_frequency Space frequency for digital '0'
* @param bit_rate Data transmission bit rate
* @param window_size Analysis window size
*/
AudioSignalProcessor(size_t sample_rate, size_t mark_frequency, size_t space_frequency,
size_t bit_rate, size_t window_size);
/**
* Process input audio samples
* @param samples Input audio sample vector
* @return Vector of Mark probability values (0.0 to 1.0)
*/
std::vector<float> ProcessAudioSamples(const std::vector<float> &samples);
};
/**
* Data reception state machine states
*/
enum class DataReceptionState
{
kInactive, // Waiting for start signal
kWaiting, // Detected potential start, waiting for confirmation
kReceiving // Actively receiving data
};
/**
* Data buffer for managing audio-to-digital data conversion
* Handles the complete process from audio signal to decoded text data
*/
class AudioDataBuffer
{
private:
DataReceptionState current_state_; // Current reception state
std::deque<uint8_t> identifier_buffer_; // Buffer for start/end identifier detection
size_t identifier_buffer_size_; // Identifier buffer size
std::vector<uint8_t> bit_buffer_; // Buffer for storing bit stream
size_t max_bit_buffer_size_; // Maximum bit buffer size
const std::vector<uint8_t> start_of_transmission_; // Start-of-transmission identifier
const std::vector<uint8_t> end_of_transmission_; // End-of-transmission identifier
bool enable_checksum_validation_; // Whether to validate checksum
public:
std::optional<std::string> decoded_text; // Successfully decoded text data
/**
* Default constructor using predefined start and end identifiers
*/
AudioDataBuffer();
/**
* Constructor with custom parameters
* @param max_byte_size Expected maximum data size in bytes
* @param start_identifier Start-of-transmission identifier
* @param end_identifier End-of-transmission identifier
* @param enable_checksum Whether to enable checksum validation
*/
AudioDataBuffer(size_t max_byte_size, const std::vector<uint8_t> &start_identifier,
const std::vector<uint8_t> &end_identifier, bool enable_checksum = false);
/**
* Process probability data and attempt to decode
* @param probabilities Vector of Mark probabilities
* @param threshold Decision threshold for bit detection
* @return true if complete data was successfully received and decoded
*/
bool ProcessProbabilityData(const std::vector<float> &probabilities, float threshold = 0.5f);
/**
* Calculate checksum for ASCII text
* @param text Input text string
* @return Checksum value (0-255)
*/
static uint8_t CalculateChecksum(const std::string &text);
private:
/**
* Convert bit vector to byte vector
* @param bits Input bit vector
* @return Converted byte vector
*/
std::vector<uint8_t> ConvertBitsToBytes(const std::vector<uint8_t> &bits) const;
/**
* Clear all buffers and reset state
*/
void ClearBuffers();
};
// Default start and end transmission identifiers
extern const std::vector<uint8_t> kDefaultStartTransmissionPattern;
extern const std::vector<uint8_t> kDefaultEndTransmissionPattern;
}
+227 -82
View File
@@ -57,8 +57,6 @@ void esp_blufi_btc_deinit(void);
#include <wifi_station.h>
#include "esp_crc.h"
#include "esp_random.h"
#include "mbedtls/md5.h"
#include "ssid_manager.h"
static const char* BLUFI_TAG = "BLUFI_CLASS";
@@ -345,51 +343,70 @@ esp_err_t Blufi::_controller_deinit() {
}
#endif
static int myrand(void* rng_state, unsigned char* output, size_t len) {
esp_fill_random(output, len);
return 0;
}
namespace {
constexpr uint8_t kDhParamLength = 0x00;
constexpr uint8_t kDhParamData = 0x01;
constexpr size_t kBlufiIvSize = 16;
constexpr size_t kBlufiHashSize = 32;
constexpr char kBlufiEncryptDomain[] = "blufi_enc";
constexpr char kBlufiDecryptDomain[] = "blufi_dec";
#if ESP_IDF_VERSION >= ESP_IDF_VERSION_VAL(6, 0, 0)
constexpr auto kBlufiHashError = ESP_BLUFI_CALC_SHA_256_ERROR;
#else
// IDF 5 does not expose a SHA-256-specific BluFi error code.
constexpr auto kBlufiHashError = ESP_BLUFI_CALC_MD5_ERROR;
#endif
} // namespace
void Blufi::_security_init() {
_security_deinit();
psa_status_t status = psa_crypto_init();
if (status != PSA_SUCCESS) {
ESP_LOGE(BLUFI_TAG, "psa_crypto_init failed: %d", status);
btc_blufi_report_error(ESP_BLUFI_INIT_SECURITY_ERROR);
return;
}
m_sec = new BlufiSecurity();
if (m_sec == nullptr) {
ESP_LOGE(BLUFI_TAG, "Failed to allocate security context");
btc_blufi_report_error(ESP_BLUFI_INIT_SECURITY_ERROR);
return;
}
memset(m_sec, 0, sizeof(BlufiSecurity));
m_sec->dhm = new mbedtls_dhm_context();
m_sec->aes = new mbedtls_aes_context();
mbedtls_dhm_init(m_sec->dhm);
mbedtls_aes_init(m_sec->aes);
memset(m_sec->iv, 0x0, sizeof(m_sec->iv));
m_sec->enc_operation = psa_cipher_operation_init();
m_sec->dec_operation = psa_cipher_operation_init();
}
void Blufi::_security_deinit() {
if (m_sec == nullptr)
if (m_sec == nullptr) {
return;
if (m_sec->dh_param) {
free(m_sec->dh_param);
}
mbedtls_dhm_free(m_sec->dhm);
mbedtls_aes_free(m_sec->aes);
delete m_sec->dhm;
delete m_sec->aes;
psa_cipher_abort(&m_sec->enc_operation);
psa_cipher_abort(&m_sec->dec_operation);
if (m_sec->aes_key != PSA_KEY_ID_NULL) {
psa_destroy_key(m_sec->aes_key);
}
free(m_sec->dh_param);
memset(m_sec, 0, sizeof(BlufiSecurity));
delete m_sec;
m_sec = nullptr;
}
void Blufi::_dh_negotiate_data_handler(uint8_t* data, int len, uint8_t** output_data,
int* output_len, bool* need_free) {
if (m_sec == nullptr) {
if (m_sec == nullptr || data == nullptr || output_data == nullptr || output_len == nullptr ||
need_free == nullptr) {
ESP_LOGE(BLUFI_TAG, "Security not initialized in DH handler");
btc_blufi_report_error(ESP_BLUFI_INIT_SECURITY_ERROR);
return;
}
if (len < 1) {
if (len < 3) {
ESP_LOGE(BLUFI_TAG, "DH handler: data too short");
btc_blufi_report_error(ESP_BLUFI_DATA_FORMAT_ERROR);
return;
@@ -397,70 +414,191 @@ void Blufi::_dh_negotiate_data_handler(uint8_t* data, int len, uint8_t** output_
uint8_t type = data[0];
switch (type) {
case 0x00:
if (len < 3) {
ESP_LOGE(BLUFI_TAG, "DH_PARAM_LEN packet too short");
btc_blufi_report_error(ESP_BLUFI_DATA_FORMAT_ERROR);
case kDhParamLength:
m_sec->dh_param_len = (data[1] << 8) | data[2];
if (m_sec->dh_param_len <= 0 || m_sec->dh_param_len > DH_PARAM_LEN_MAX) {
ESP_LOGE(BLUFI_TAG, "Invalid DH parameter length: %d", m_sec->dh_param_len);
m_sec->dh_param_len = 0;
btc_blufi_report_error(ESP_BLUFI_DH_PARAM_ERROR);
return;
}
m_sec->dh_param_len = (data[1] << 8) | data[2];
if (m_sec->dh_param) {
free(m_sec->dh_param);
m_sec->dh_param = nullptr;
free(m_sec->dh_param);
m_sec->dh_param = nullptr;
psa_cipher_abort(&m_sec->enc_operation);
psa_cipher_abort(&m_sec->dec_operation);
m_sec->enc_operation = psa_cipher_operation_init();
m_sec->dec_operation = psa_cipher_operation_init();
if (m_sec->aes_key != PSA_KEY_ID_NULL) {
psa_destroy_key(m_sec->aes_key);
m_sec->aes_key = PSA_KEY_ID_NULL;
}
m_sec->dh_param = (uint8_t*)malloc(m_sec->dh_param_len);
if (m_sec->dh_param == nullptr) {
ESP_LOGE(BLUFI_TAG, "DH malloc failed");
m_sec->dh_param_len = 0;
btc_blufi_report_error(ESP_BLUFI_DH_MALLOC_ERROR);
return;
}
break;
case 0x01: {
case kDhParamData: {
if (m_sec->dh_param == nullptr) {
ESP_LOGE(BLUFI_TAG, "DH param not allocated");
btc_blufi_report_error(ESP_BLUFI_DH_PARAM_ERROR);
return;
}
uint8_t* param = m_sec->dh_param;
if (len != m_sec->dh_param_len + 1) {
ESP_LOGE(BLUFI_TAG, "Invalid DH parameter packet length: %d, expected %d", len,
m_sec->dh_param_len + 1);
btc_blufi_report_error(ESP_BLUFI_DH_PARAM_ERROR);
return;
}
memcpy(m_sec->dh_param, &data[1], m_sec->dh_param_len);
int ret = mbedtls_dhm_read_params(m_sec->dhm, &param, &param[m_sec->dh_param_len]);
if (ret) {
ESP_LOGE(BLUFI_TAG, "mbedtls_dhm_read_params failed %d", ret);
const uint8_t* param = m_sec->dh_param;
const uint8_t* const end = param + m_sec->dh_param_len;
auto read_field = [&param, end](const uint8_t** value, size_t* value_len) -> bool {
if (end - param < 2) {
return false;
}
*value_len = (static_cast<size_t>(param[0]) << 8) | param[1];
param += 2;
if (*value_len == 0 || static_cast<size_t>(end - param) < *value_len) {
return false;
}
*value = param;
param += *value_len;
return true;
};
const uint8_t* prime = nullptr;
const uint8_t* generator = nullptr;
const uint8_t* peer_public_key = nullptr;
size_t prime_len = 0;
size_t generator_len = 0;
size_t peer_public_key_len = 0;
if (!read_field(&prime, &prime_len) || !read_field(&generator, &generator_len) ||
!read_field(&peer_public_key, &peer_public_key_len) || param != end) {
ESP_LOGE(BLUFI_TAG, "Malformed DH parameter data");
btc_blufi_report_error(ESP_BLUFI_READ_PARAM_ERROR);
return;
}
const int dhm_len = mbedtls_dhm_get_len(m_sec->dhm);
// PSA FFDH supports named RFC 7919 groups instead of caller-supplied P/G. The IDF 6
// BluFi protocol uses ffdhe3072; P and G remain in the packet for framing compatibility.
constexpr size_t kDhKeyBits = 3072;
constexpr size_t kDhKeyBytes = kDhKeyBits / 8;
if (prime_len != kDhKeyBytes || peer_public_key_len != kDhKeyBytes ||
generator_len != 1 || generator[0] != 2) {
ESP_LOGE(BLUFI_TAG,
"Unsupported DH group (P=%u, G=%u, public=%u); ffdhe3072 is required",
static_cast<unsigned>(prime_len), static_cast<unsigned>(generator_len),
static_cast<unsigned>(peer_public_key_len));
btc_blufi_report_error(ESP_BLUFI_DH_PARAM_ERROR);
return;
}
ret = mbedtls_dhm_make_public(m_sec->dhm, dhm_len, m_sec->self_public_key, dhm_len,
myrand, NULL);
if (ret != 0) {
ESP_LOGE(BLUFI_TAG, "mbedtls_dhm_make_public failed: %d", ret);
psa_key_attributes_t attributes = psa_key_attributes_init();
psa_set_key_type(&attributes, PSA_KEY_TYPE_DH_KEY_PAIR(PSA_DH_FAMILY_RFC7919));
psa_set_key_bits(&attributes, kDhKeyBits);
psa_set_key_algorithm(&attributes, PSA_ALG_FFDH);
psa_set_key_usage_flags(&attributes, PSA_KEY_USAGE_DERIVE);
psa_key_id_t private_key = PSA_KEY_ID_NULL;
psa_status_t status = psa_generate_key(&attributes, &private_key);
psa_reset_key_attributes(&attributes);
if (status != PSA_SUCCESS) {
ESP_LOGE(BLUFI_TAG, "psa_generate_key failed: %d", status);
btc_blufi_report_error(ESP_BLUFI_MAKE_PUBLIC_ERROR);
return;
}
ret = mbedtls_dhm_calc_secret(m_sec->dhm, m_sec->share_key, SHARE_KEY_LEN,
&m_sec->share_len, myrand, NULL);
if (ret != 0) {
ESP_LOGE(BLUFI_TAG, "mbedtls_dhm_calc_secret failed: %d", ret);
size_t public_key_len = 0;
status = psa_export_public_key(private_key, m_sec->self_public_key,
sizeof(m_sec->self_public_key), &public_key_len);
if (status != PSA_SUCCESS || public_key_len != kDhKeyBytes) {
ESP_LOGE(BLUFI_TAG, "psa_export_public_key failed: %d, length: %u", status,
static_cast<unsigned>(public_key_len));
psa_destroy_key(private_key);
btc_blufi_report_error(ESP_BLUFI_MAKE_PUBLIC_ERROR);
return;
}
status = psa_raw_key_agreement(PSA_ALG_FFDH, private_key, peer_public_key,
peer_public_key_len, m_sec->share_key,
sizeof(m_sec->share_key), &m_sec->share_len);
psa_destroy_key(private_key);
if (status != PSA_SUCCESS) {
ESP_LOGE(BLUFI_TAG, "psa_raw_key_agreement failed: %d", status);
btc_blufi_report_error(ESP_BLUFI_ENCRYPT_ERROR);
return;
}
ret = mbedtls_md5(m_sec->share_key, m_sec->share_len, m_sec->psk);
if (ret != 0) {
ESP_LOGE(BLUFI_TAG, "mbedtls_md5 failed: %d", ret);
btc_blufi_report_error(ESP_BLUFI_CALC_MD5_ERROR);
size_t hash_len = 0;
status = psa_hash_compute(PSA_ALG_SHA_256, m_sec->share_key, m_sec->share_len,
m_sec->psk, sizeof(m_sec->psk), &hash_len);
if (status != PSA_SUCCESS || hash_len != sizeof(m_sec->psk)) {
ESP_LOGE(BLUFI_TAG, "psa_hash_compute failed: %d", status);
btc_blufi_report_error(kBlufiHashError);
return;
}
ret = mbedtls_aes_setkey_enc(m_sec->aes, m_sec->psk, PSK_LEN * 8);
if (ret != 0) {
ESP_LOGE(BLUFI_TAG, "mbedtls_aes_setkey_enc failed: -0x%04X", -ret);
attributes = psa_key_attributes_init();
psa_set_key_type(&attributes, PSA_KEY_TYPE_AES);
psa_set_key_bits(&attributes, PSK_LEN * 8);
psa_set_key_algorithm(&attributes, PSA_ALG_CTR);
psa_set_key_usage_flags(&attributes, PSA_KEY_USAGE_ENCRYPT | PSA_KEY_USAGE_DECRYPT);
status = psa_import_key(&attributes, m_sec->psk, sizeof(m_sec->psk), &m_sec->aes_key);
psa_reset_key_attributes(&attributes);
if (status != PSA_SUCCESS) {
ESP_LOGE(BLUFI_TAG, "psa_import_key failed: %d", status);
btc_blufi_report_error(ESP_BLUFI_ENCRYPT_ERROR);
return;
}
auto setup_cipher = [this](psa_cipher_operation_t* operation, const char* domain) {
uint8_t material[sizeof(kBlufiEncryptDomain) - 1 + SHARE_KEY_LEN];
memcpy(material, domain, sizeof(kBlufiEncryptDomain) - 1);
memcpy(material + sizeof(kBlufiEncryptDomain) - 1, m_sec->share_key,
m_sec->share_len);
uint8_t hash[kBlufiHashSize];
size_t hash_len = 0;
psa_status_t status = psa_hash_compute(
PSA_ALG_SHA_256, material,
sizeof(kBlufiEncryptDomain) - 1 + m_sec->share_len, hash, sizeof(hash),
&hash_len);
memset(material, 0, sizeof(material));
if (status != PSA_SUCCESS || hash_len != sizeof(hash)) {
return status == PSA_SUCCESS ? PSA_ERROR_GENERIC_ERROR : status;
}
*operation = psa_cipher_operation_init();
status = psa_cipher_encrypt_setup(operation, m_sec->aes_key, PSA_ALG_CTR);
if (status == PSA_SUCCESS) {
status = psa_cipher_set_iv(operation, hash, kBlufiIvSize);
}
memset(hash, 0, sizeof(hash));
return status;
};
status = setup_cipher(&m_sec->enc_operation, kBlufiEncryptDomain);
if (status == PSA_SUCCESS) {
// CTR decryption uses the same cipher primitive with an independent counter.
status = setup_cipher(&m_sec->dec_operation, kBlufiDecryptDomain);
}
if (status != PSA_SUCCESS) {
ESP_LOGE(BLUFI_TAG, "PSA cipher setup failed: %d", status);
psa_cipher_abort(&m_sec->enc_operation);
psa_cipher_abort(&m_sec->dec_operation);
psa_destroy_key(m_sec->aes_key);
m_sec->aes_key = PSA_KEY_ID_NULL;
btc_blufi_report_error(ESP_BLUFI_ENCRYPT_ERROR);
return;
}
*output_data = m_sec->self_public_key;
*output_len = dhm_len;
*output_len = public_key_len;
*need_free = false;
ESP_LOGI(BLUFI_TAG, "DH negotiation completed successfully");
@@ -471,49 +609,56 @@ void Blufi::_dh_negotiate_data_handler(uint8_t* data, int len, uint8_t** output_
}
default:
ESP_LOGE(BLUFI_TAG, "DH handler unknown type: %d", type);
btc_blufi_report_error(ESP_BLUFI_DATA_FORMAT_ERROR);
}
}
int Blufi::_aes_encrypt(uint8_t iv8, uint8_t* crypt_data, int crypt_len) {
if (!m_sec || !m_sec->aes || !crypt_data || crypt_len <= 0) {
(void)iv8;
if (!m_sec || m_sec->aes_key == PSA_KEY_ID_NULL || !crypt_data || crypt_len < 0) {
ESP_LOGE(BLUFI_TAG, "Invalid parameters for AES encryption");
return -ESP_ERR_INVALID_ARG;
}
size_t iv_offset = 0;
uint8_t iv0[16];
memcpy(iv0, m_sec->iv, 16);
iv0[0] = iv8;
int ret = mbedtls_aes_crypt_cfb128(m_sec->aes, MBEDTLS_AES_ENCRYPT, crypt_len, &iv_offset, iv0,
crypt_data, crypt_data);
if (ret == 0) {
return crypt_len;
} else {
ESP_LOGE(BLUFI_TAG, "AES encrypt failed: %d", ret);
return ret;
if (crypt_len == 0) {
return 0;
}
std::vector<uint8_t> output(PSA_CIPHER_ENCRYPT_OUTPUT_SIZE(
PSA_KEY_TYPE_AES, PSA_ALG_CTR, static_cast<size_t>(crypt_len)));
size_t output_len = 0;
psa_status_t status = psa_cipher_update(&m_sec->enc_operation, crypt_data, crypt_len,
output.data(), output.size(), &output_len);
if (status != PSA_SUCCESS || output_len != static_cast<size_t>(crypt_len)) {
ESP_LOGE(BLUFI_TAG, "AES encrypt failed: %d, length: %u", status,
static_cast<unsigned>(output_len));
return status == PSA_SUCCESS ? -ESP_FAIL : status;
}
memcpy(crypt_data, output.data(), output_len);
return static_cast<int>(output_len);
}
int Blufi::_aes_decrypt(uint8_t iv8, uint8_t* crypt_data, int crypt_len) {
if (!m_sec || !m_sec->aes || !crypt_data || crypt_len < 0) {
ESP_LOGE(BLUFI_TAG, "Invalid parameters for AES decryption %p %p %d", m_sec->aes,
crypt_data, crypt_len);
(void)iv8;
if (!m_sec || m_sec->aes_key == PSA_KEY_ID_NULL || !crypt_data || crypt_len < 0) {
ESP_LOGE(BLUFI_TAG, "Invalid parameters for AES decryption");
return -ESP_ERR_INVALID_ARG;
}
size_t iv_offset = 0;
uint8_t iv0[16];
memcpy(iv0, m_sec->iv, 16);
iv0[0] = iv8;
int ret = mbedtls_aes_crypt_cfb128(m_sec->aes, MBEDTLS_AES_DECRYPT, crypt_len, &iv_offset, iv0,
crypt_data, crypt_data);
if (ret != 0) {
ESP_LOGE(BLUFI_TAG, "AES decrypt failed: %d", ret);
return ret;
} else {
return crypt_len;
if (crypt_len == 0) {
return 0;
}
std::vector<uint8_t> output(PSA_CIPHER_DECRYPT_OUTPUT_SIZE(
PSA_KEY_TYPE_AES, PSA_ALG_CTR, static_cast<size_t>(crypt_len)));
size_t output_len = 0;
psa_status_t status = psa_cipher_update(&m_sec->dec_operation, crypt_data, crypt_len,
output.data(), output.size(), &output_len);
if (status != PSA_SUCCESS || output_len != static_cast<size_t>(crypt_len)) {
ESP_LOGE(BLUFI_TAG, "AES decrypt failed: %d, length: %u", status,
static_cast<unsigned>(output_len));
return status == PSA_SUCCESS ? -ESP_FAIL : status;
}
memcpy(crypt_data, output.data(), output_len);
return static_cast<int>(output_len);
}
uint16_t Blufi::_crc_checksum(uint8_t iv8, uint8_t* data, int len) {
+9 -9
View File
@@ -1,16 +1,15 @@
#pragma once
#include <aes/esp_aes.h>
#include <cassert>
#include <cstring>
#include <vector>
#include "esp_blufi_api.h"
#include "esp_err.h"
#include "esp_idf_version.h"
#include "esp_timer.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "mbedtls/aes.h"
#include "mbedtls/dhm.h"
#include "psa/crypto.h"
#include "wifi_manager.h"
class Blufi {
@@ -112,18 +111,19 @@ private:
// Security context, formerly blufi_sec struct
struct BlufiSecurity {
#define DH_SELF_PUB_KEY_LEN 128
#define DH_PARAM_LEN_MAX 1024
#define DH_SELF_PUB_KEY_LEN 384
uint8_t self_public_key[DH_SELF_PUB_KEY_LEN];
#define SHARE_KEY_LEN 128
#define SHARE_KEY_LEN 384
uint8_t share_key[SHARE_KEY_LEN];
size_t share_len;
#define PSK_LEN 16
#define PSK_LEN 32
uint8_t psk[PSK_LEN];
uint8_t *dh_param;
int dh_param_len;
uint8_t iv[16];
mbedtls_dhm_context *dhm;
esp_aes_context *aes;
psa_key_id_t aes_key;
psa_cipher_operation_t enc_operation;
psa_cipher_operation_t dec_operation;
};
BlufiSecurity *m_sec;
+17 -6
View File
@@ -57,8 +57,7 @@
#define TAG "EspVideo"
#if defined(CONFIG_CAMERA_SENSOR_SWAP_PIXEL_BYTE_ORDER) || defined(CONFIG_XIAOZHI_ENABLE_CAMERA_ENDIANNESS_SWAP)
#warning \
"CAMERA_SENSOR_SWAP_PIXEL_BYTE_ORDER or CONFIG_XIAOZHI_ENABLE_CAMERA_ENDIANNESS_SWAP is enabled, which may cause image corruption in YUV422 format!"
#pragma message("CAMERA_SENSOR_SWAP_PIXEL_BYTE_ORDER or CONFIG_XIAOZHI_ENABLE_CAMERA_ENDIANNESS_SWAP is enabled; verify YUV422 image integrity")
#endif
#if CONFIG_XIAOZHI_ENABLE_CAMERA_DEBUG_MODE
@@ -191,7 +190,8 @@ EspVideo::EspVideo(const esp_video_init_config_t& config) {
uint32_t best_fmt = 0;
int best_rank = 1 << 30; // large number
// 注: 当前版本 esp_video 中 YUV422P 实际输出为 YUYV。
// esp_video 2.x uses the packed YUYV/UYVY FOURCC values. Keep YUV422P
// for compatibility with esp_video 1.x, where it was used for YUYV data.
#if defined(CONFIG_XIAOZHI_ENABLE_ROTATE_CAMERA_IMAGE) && defined(CONFIG_SOC_PPA_SUPPORTED)
auto get_rank = [](uint32_t fmt) -> int {
switch (fmt) {
@@ -203,6 +203,9 @@ EspVideo::EspVideo(const esp_video_init_config_t& config) {
case V4L2_PIX_FMT_YUV420: // 软件 JPEG 编码器不支持 YUV420 格式
return 2;
#endif // CONFIG_XIAOZHI_ENABLE_HARDWARE_JPEG_ENCODER
case V4L2_PIX_FMT_YUYV:
case V4L2_PIX_FMT_UYVY:
return 3;
case V4L2_PIX_FMT_GREY:
case V4L2_PIX_FMT_YUV422P:
default:
@@ -212,6 +215,8 @@ EspVideo::EspVideo(const esp_video_init_config_t& config) {
#else
auto get_rank = [](uint32_t fmt) -> int {
switch (fmt) {
case V4L2_PIX_FMT_YUYV:
case V4L2_PIX_FMT_UYVY:
case V4L2_PIX_FMT_YUV422P:
return 10;
case V4L2_PIX_FMT_RGB565:
@@ -433,6 +438,7 @@ bool EspVideo::Capture() {
case V4L2_PIX_FMT_RGB565:
case V4L2_PIX_FMT_RGB24:
case V4L2_PIX_FMT_YUYV:
case V4L2_PIX_FMT_UYVY:
case V4L2_PIX_FMT_YUV420:
case V4L2_PIX_FMT_GREY:
#ifdef CONFIG_XIAOZHI_CAMERA_ALLOW_JPEG_INPUT
@@ -517,6 +523,9 @@ bool EspVideo::Capture() {
rotate_cfg.in_pixel_fmt = ESP_IMGFX_PIXEL_FMT_RGB565_LE;
break;
case V4L2_PIX_FMT_YUYV:
case V4L2_PIX_FMT_UYVY:
// Rotate packed YUV422 as opaque 16-bit pixels. The rotation
// operation only needs the pixel size and does not convert color.
rotate_cfg.in_pixel_fmt = ESP_IMGFX_PIXEL_FMT_RGB565_LE;
break;
case V4L2_PIX_FMT_GREY:
@@ -584,8 +593,9 @@ bool EspVideo::Capture() {
rotate_src = (uint8_t*)frame_.data;
ppa_color_mode = PPA_SRM_COLOR_MODE_RGB888;
break;
case V4L2_PIX_FMT_YUYV: {
ESP_LOGW(TAG, "YUYV format is not supported for PPA rotation, using software conversion to RGB888");
case V4L2_PIX_FMT_YUYV:
case V4L2_PIX_FMT_UYVY: {
ESP_LOGW(TAG, "YUV422 format is not supported for PPA rotation, using software conversion to RGB888");
rotate_src = (uint8_t*)heap_caps_malloc(frame_.width * frame_.height * 3,
MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);
if (rotate_src == nullptr) {
@@ -598,7 +608,7 @@ bool EspVideo::Capture() {
esp_imgfx_color_convert_cfg_t convert_cfg = {
.in_res = {.width = static_cast<int16_t>(frame_.width),
.height = static_cast<int16_t>(frame_.height)},
.in_pixel_fmt = ESP_IMGFX_PIXEL_FMT_YUYV,
.in_pixel_fmt = static_cast<esp_imgfx_pixel_fmt_t>(frame_.format),
.out_pixel_fmt = ESP_IMGFX_PIXEL_FMT_RGB888,
};
esp_imgfx_color_convert_handle_t convert_handle = nullptr;
@@ -746,6 +756,7 @@ bool EspVideo::Capture() {
switch (frame_.format) {
// LVGL 显示 YUV 系的图像似乎都有问题,暂时转换为 RGB565 显示
case V4L2_PIX_FMT_YUYV:
case V4L2_PIX_FMT_UYVY:
case V4L2_PIX_FMT_YUV420:
case V4L2_PIX_FMT_RGB24: {
color_format = LV_COLOR_FORMAT_RGB565;
+3 -2
View File
@@ -12,7 +12,7 @@
#include <esp_mac.h>
#include <esp_network.h>
#include <esp_netif.h>
#include <font_awesome.h>
#include <material_symbols.h>
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#include <soc/soc_caps.h>
@@ -221,7 +221,7 @@ NetworkInterface* EthernetBoard::GetNetwork() {
}
const char* EthernetBoard::GetNetworkStateIcon() {
return connected_ ? FONT_AWESOME_SIGNAL_STRONG : FONT_AWESOME_SIGNAL_OFF;
return connected_ ? MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR : MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
}
std::string EthernetBoard::GetEthernetMacAddress() {
@@ -236,6 +236,7 @@ std::string EthernetBoard::GetEthernetMacAddress() {
std::string EthernetBoard::GetBoardJson() {
std::string json = R"({"type":")" + std::string(BOARD_TYPE) + R"(",)";
json += R"("name":")" + std::string(BOARD_NAME) + R"(",)";
json += R"("manufacturer":")" + std::string(BOARD_MANUFACTURER) + R"(",)";
json += R"("network":"ethernet",)";
if (!ip_address_.empty()) {
json += R"("ip":")" + ip_address_ + R"(",)";
+9 -8
View File
@@ -7,7 +7,7 @@
#include <esp_timer.h>
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#include <font_awesome.h>
#include <material_symbols.h>
#include <utility>
static const char *TAG = "Ml307Board";
@@ -146,29 +146,30 @@ NetworkInterface* Ml307Board::GetNetwork() {
const char* Ml307Board::GetNetworkStateIcon() {
if (modem_ == nullptr || !modem_->network_ready()) {
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
}
int csq = modem_->GetCsq();
if (csq == -1) {
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
} else if (csq >= 0 && csq <= 9) {
return FONT_AWESOME_SIGNAL_WEAK;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT_1_BAR;
} else if (csq >= 10 && csq <= 14) {
return FONT_AWESOME_SIGNAL_FAIR;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT_2_BAR;
} else if (csq >= 15 && csq <= 19) {
return FONT_AWESOME_SIGNAL_GOOD;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT;
} else if (csq >= 20 && csq <= 31) {
return FONT_AWESOME_SIGNAL_STRONG;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR;
}
ESP_LOGW(TAG, "Invalid CSQ: %d", csq);
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
}
std::string Ml307Board::GetBoardJson() {
// Set the board type for OTA
std::string board_json = std::string("{\"type\":\"" BOARD_TYPE "\",");
board_json += "\"name\":\"" BOARD_NAME "\",";
board_json += "\"manufacturer\":\"" BOARD_MANUFACTURER "\",";
board_json += "\"revision\":\"" + modem_->GetModuleRevision() + "\",";
board_json += "\"carrier\":\"" + modem_->GetCarrierName() + "\",";
board_json += "\"csq\":\"" + std::to_string(modem_->GetCsq()) + "\",";
+14 -10
View File
@@ -3,7 +3,7 @@
#include "application.h"
#include "audio_codec.h"
#include <esp_log.h>
#include <font_awesome.h>
#include <material_symbols.h>
#include <cJSON.h>
#define TAG "Nt26Board"
@@ -79,7 +79,7 @@ void Nt26Board::StartNetwork() {
modem_ = std::make_unique<UartEthModem>(config);
modem_->SetDebug(false);
modem_->SetNetworkEventCallback([this](UartEthModem::UartEthModemEvent event) {
modem_->SetNetworkEventCallback([this](UartEthModem::UartEthModemEvent event, const std::string& detail) {
switch (event) {
case UartEthModem::UartEthModemEvent::Connected:
esp_timer_stop(network_ready_timer_);
@@ -105,11 +105,14 @@ void Nt26Board::StartNetwork() {
case UartEthModem::UartEthModemEvent::ErrorNoCarrier:
esp_timer_stop(network_ready_timer_);
ScheduleAsyncStop();
OnNetworkEvent(NetworkEvent::ModemErrorInitFailed);
OnNetworkEvent(NetworkEvent::ModemErrorInitFailed, detail);
break;
case UartEthModem::UartEthModemEvent::InFlightMode:
ESP_LOGW(TAG, "Modem in flight mode");
break;
case UartEthModem::UartEthModemEvent::RfTestReady:
ESP_LOGI(TAG, "Modem RF test mode ready");
break;
case UartEthModem::UartEthModemEvent::RequestingPdpContext:
break;
}
@@ -143,21 +146,21 @@ NetworkInterface* Nt26Board::GetNetwork() {
const char* Nt26Board::GetNetworkStateIcon() {
if (modem_ == nullptr || !modem_->IsInitialized()) {
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
}
int csq = modem_->GetSignalStrength();
if (csq == 99 || csq == -1) {
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
} else if (csq >= 0 && csq <= 9) {
return FONT_AWESOME_SIGNAL_WEAK;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT_1_BAR;
} else if (csq >= 10 && csq <= 14) {
return FONT_AWESOME_SIGNAL_FAIR;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT_2_BAR;
} else if (csq >= 15 && csq <= 19) {
return FONT_AWESOME_SIGNAL_GOOD;
return MATERIAL_SYMBOLS_SIGNAL_CELLULAR_ALT;
} else if (csq >= 20 && csq <= 31) {
return FONT_AWESOME_SIGNAL_STRONG;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR;
}
return FONT_AWESOME_SIGNAL_OFF;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR_OFF;
}
void Nt26Board::SetPowerSaveLevel(PowerSaveLevel level) {
@@ -183,6 +186,7 @@ std::string Nt26Board::GetBoardJson() {
// Set the board type for OTA
std::string board_json = std::string("{\"type\":\"" BOARD_TYPE "\",");
board_json += "\"name\":\"" BOARD_NAME "\",";
board_json += "\"manufacturer\":\"" BOARD_MANUFACTURER "\",";
if (modem_) {
board_json += "\"revision\":\"" + modem_->GetModuleRevision() + "\",";
board_json += "\"carrier\":\"" + modem_->GetCarrierName() + "\",";
+4 -3
View File
@@ -10,7 +10,7 @@
#include <esp_network.h>
#include <esp_log.h>
#include <utility>
#include <font_awesome.h>
#include <material_symbols.h>
#if CONFIG_IDF_TARGET_ESP32P4 || CONFIG_IDF_TARGET_ESP32S3
@@ -176,13 +176,14 @@ NetworkInterface* RndisBoard::GetNetwork() {
}
const char* RndisBoard::GetNetworkStateIcon() {
return FONT_AWESOME_SIGNAL_STRONG;
return MATERIAL_SYMBOLS_ANDROID_CELL_4_BAR;
}
std::string RndisBoard::GetBoardJson() {
std::string json = R"({"type":")" + std::string(BOARD_TYPE) + R"(",)";
json += R"("name":")" + std::string(BOARD_NAME) + R"(",)";
json += R"("manufacturer":")" + std::string(BOARD_MANUFACTURER) + R"(",)";
json += R"("mac":")" + SystemInfo::GetMacAddress() + R"("})";
return json;
@@ -244,4 +245,4 @@ std::string RndisBoard::GetDeviceStatusJson() {
cJSON_Delete(root);
return result;
}
#endif // CONFIG_IDF_TARGET_ESP32P4 || CONFIG_IDF_TARGET_ESP32S3
#endif // CONFIG_IDF_TARGET_ESP32P4 || CONFIG_IDF_TARGET_ESP32S3
+12
View File
@@ -7,6 +7,8 @@
#include <esp_log.h>
#include <esp_sleep.h>
#include <esp_lvgl_port.h>
#include <esp_idf_version.h>
#include <inttypes.h>
#define TAG "SleepTimer"
@@ -99,11 +101,21 @@ void SleepTimer::CheckTimer() {
esp_light_sleep_start();
lvgl_port_resume();
#if ESP_IDF_VERSION >= ESP_IDF_VERSION_VAL(6, 0, 0)
// IDF 6 deprecates esp_sleep_get_wakeup_cause() in favor of
// the bitmap-based API that can report simultaneous sources.
auto wakeup_causes = esp_sleep_get_wakeup_causes();
ESP_LOGI(TAG, "Wake up from light sleep, wakeup_causes: 0x%" PRIx32, wakeup_causes);
if (!(wakeup_causes & (1UL << ESP_SLEEP_WAKEUP_TIMER))) {
break;
}
#else
auto wakeup_reason = esp_sleep_get_wakeup_cause();
ESP_LOGI(TAG, "Wake up from light sleep, wakeup_reason: %d", wakeup_reason);
if (wakeup_reason != ESP_SLEEP_WAKEUP_TIMER) {
break;
}
#endif
}
WakeUp();
});
+7 -22
View File
@@ -13,11 +13,10 @@
#include <esp_mac.h>
#include <utility>
#include <font_awesome.h>
#include <material_symbols.h>
#include <wifi_manager.h>
#include <wifi_station.h>
#include <ssid_manager.h>
#include "afsk_demod.h"
#ifdef CONFIG_USE_ESP_BLUFI_WIFI_PROVISIONING
#include "blufi.h"
#endif
@@ -188,21 +187,6 @@ void WifiBoard::StartWifiConfigMode() {
// initialize esp-blufi protocol
blufi.init();
#endif
#if CONFIG_USE_ACOUSTIC_WIFI_PROVISIONING
// Start acoustic provisioning task
auto codec = Board::GetInstance().GetAudioCodec();
int channel = codec ? codec->input_channels() : 1;
ESP_LOGI(TAG, "Starting acoustic WiFi provisioning, channels: %d", channel);
xTaskCreate([](void* arg) {
auto ch = reinterpret_cast<intptr_t>(arg);
auto& app = Application::GetInstance();
auto& wifi = WifiManager::GetInstance();
auto disp = Board::GetInstance().GetDisplay();
audio_wifi_config::ReceiveWifiCredentialsFromAudio(&app, &wifi, disp, ch);
vTaskDelete(NULL);
}, "acoustic_wifi", 4096, reinterpret_cast<void*>(channel), 2, NULL);
#endif
}
void WifiBoard::EnterWifiConfigMode() {
@@ -259,25 +243,26 @@ const char* WifiBoard::GetNetworkStateIcon() {
auto& wifi = WifiManager::GetInstance();
if (wifi.IsConfigMode()) {
return FONT_AWESOME_WIFI;
return MATERIAL_SYMBOLS_WIFI;
}
if (!wifi.IsConnected()) {
return FONT_AWESOME_WIFI_SLASH;
return MATERIAL_SYMBOLS_WIFI_OFF;
}
int rssi = wifi.GetRssi();
if (rssi >= -65) {
return FONT_AWESOME_WIFI;
return MATERIAL_SYMBOLS_WIFI;
} else if (rssi >= -75) {
return FONT_AWESOME_WIFI_FAIR;
return MATERIAL_SYMBOLS_WIFI_2_BAR;
}
return FONT_AWESOME_WIFI_WEAK;
return MATERIAL_SYMBOLS_WIFI_1_BAR;
}
std::string WifiBoard::GetBoardJson() {
auto& wifi = WifiManager::GetInstance();
std::string json = R"({"type":")" + std::string(BOARD_TYPE) + R"(",)";
json += R"("name":")" + std::string(BOARD_NAME) + R"(",)";
json += R"("manufacturer":")" + std::string(BOARD_MANUFACTURER) + R"(",)";
if (!wifi.IsConfigMode()) {
json += R"("ssid":")" + wifi.GetSsid() + R"(",)";

Some files were not shown because too many files have changed in this diff Show More