72bf13a395
* add ubring transport * fix the bug for ub ring transport * fix the bug for ub ring transport and other * add the license for ub transport * Modifying the variable naming style * optimize the log message and some field name * fix some bug for ubring * add some log ubring endpoint * add todo * fix the bug for handshake for ub endpoint * fix the bug for client ub endpoint * optimize the iobuf file code * modify the log level * fix the declare_shm_ubs define not found bug * add the timer_mgr support for macos and format code style * fix the bug for macos epoll * fix the timespece bug * adaptor the itimerspec for macos platform * optimize the cmakelist config * add the ubring docs for ubring transport * modify some file name and directory structure * modify some code style and optimize code logical * add the cmake ubshm transport ci/testing * add the dependency header * bug fix the code * Fix UB shm allocation cleanup crashes (#11) Co-authored-by: 郭业昌 <lvpengfei@MacBook-Air.local> * remove the brpc_ubshm_unittest * Found and fixed two stability issues (#13) * 修复ubring server端关闭连接coredump问题 * 修复PollIn/PollOut解引用已释放Socket指针的问题 PollIn/PollOut通过ep->_socket(裸指针)读取data socket,当data socket 被销毁时该指针悬空,导致Socket::Address读到垃圾id触发SIGSEGV。 改为存储_socket_id(SocketId),用Address获取引用计数的Socket, 并在整个回调期间持有该引用,避免解引用悬空指针。 * 修复client非正常退出导致UBRING shm残留的问题 client被强杀(SIGTERM/崩溃/OOM)时teardown没跑完,localShm(_C)的 shm_unlink未执行,导致/dev/shm残留_C文件。server的remoteShm只munmap 不unlink(正确),无法清理client的名字。 在握手ESTABLISHED时(client/server都确认对方已mmap自己的localShm) 立即unlink localShm名字。此时对端已持有mmap引用,unlink只删名字不 影响通信;进程任意时刻退出都不会残留文件名。 * Address chenBright's review: use English comments and BAIDU_CACHELINE_ALIGNMENT - Convert all Chinese comments in ubshm to English (per chenBright's 'Please use English' on ub_endpoint.cpp:723, ub_ring.cpp:337, shm_ubs.cpp:316, and similar) - Replace __attribute__((aligned(64))) with BAIDU_CACHELINE_ALIGNMENT in ubr_msg.h (per chenBright's comment on ubr_msg.h:41) - Remove unnecessary TODO comment in ub_ring.cpp:551 (per chenBright's 'Unnecessary comments, please delete') * Remove unused lock macros in thread_lock.h Per chenBright's review, the functions and macros defined in thread_lock.h are largely unused. Verified usage across ubshm: - LOCK_GUARD / UnlockMutex: 8 call sites in shm_ubs.cpp and ub_ring_manager.cpp, kept. - SPIN_LOCK_GUARD, R_LOCK_GUARD, W_LOCK_GUARD, SEMAPHORE_WAIT_GUARD, SEMAPHORE_WAIT_GUARD_WITH_CLOSE and their helper functions (UnlockSpinLock, UnlockRWLock, PostSem, PostSemWithClose): 0 call sites, removed. * Apply chenBright's review on timer_mgr globals Per chenBright's review on timer_mgr.cpp:32-37: - Add explicit default values to uninitialized globals (g_total_timer_num=0, g_max_system_fd=0, g_epoll_execute_thread=0, g_timer_module_initialized=0) - Rename globals to snake_case (g_epollFd -> g_epoll_fd, g_totalTimerNum -> g_total_timer_num, g_timerFdCtxMap -> g_timer_fd_ctx_map, maxSystemFd -> g_max_system_fd, g_epollExecuteThread -> g_epoll_execute_thread, g_timerModuleInitialized -> g_timer_module_initialized) - maxSystemFd also gains the g_ prefix to match global naming style Also fix the missing std:: qualifier on atomic_fetch_sub/add/load (per chenBright's earlier comment on timer_mgr.cpp:80). * Change CloseTimerFd fd type from uint32_t to int Per chenBright's review on timer_mgr.cpp:399 (uint32_t -> int). fd is a system file descriptor; POSIX APIs use int and -1 denotes an invalid fd, which uint32_t cannot represent. Changed the CloseTimerFd signature (header + definition) and removed the now-unnecessary (uint32_t) casts at the two call sites. * Use BAIDU_LIKELY/BAIDU_UNLIKELY instead of custom __builtin_expect Per chenBright's review on common.h:27. Rather than redefine the macros with __builtin_expect directly, forward LIKELY/UNLIKELY to brpc's standard BAIDU_LIKELY/BAIDU_UNLIKELY (from butil/compiler_specific.h). The 122 call sites keep using LIKELY()/UNLIKELY() unchanged; only the macro bodies change, preserving semantics. * Add unit tests for UBShmEndpoint Per chenBright's request to add unit tests for UBShmTransport in this PR (rather than a follow-up). Adds test/brpc_ubring_unittest.cpp with tests covering the public interface of UBShmEndpoint under the g_skip_ub_init=true mode (which skips real shared-memory/poller setup): - construct_and_destruct: lifecycle safety - is_writable_false_when_skip_init: skip-mode behavior - reset_is_idempotent: Reset() is safe to call repeatedly The file follows the brpc_*_unittest.cpp naming convention so it is auto-collected by test/CMakeLists.txt's file(GLOB). Verified: compiles, links, and all 3 tests pass (g++ 15.2, C++17, gtest, BRPC_WITH_UBRING=ON). * Rewrite UBShmEndpoint unit tests with real coverage Per chenBright's feedback that the previous tests were too simple and did not cover the main methods. Source changes to enable testing: - Move HelloMessage struct declaration from ub_endpoint.cpp to ub_endpoint.h so tests can access it - Expose private members under #ifdef UNIT_TEST (precedent: butil/containers/stack_container.h) so tests can call AllocateClientResources without -Dprivate=public (which breaks GCC 15 + new libstdc++ <any>/<sstream>) Tests (9, all passing on Ubuntu 26.04 g++ 15.2 C++17 gtest): HelloMessageTest (5): serialize/deserialize roundtrip, network byte order verification, uint64 max boundary, full shm_name, toString UBShmEndpointTest (4): construct, real IPC shm AllocateClientResources (g_skip_ub_init=false), reset cleanup, reset idempotency * rename the variable to snake_case style --------- Co-authored-by: YeChang Guo <52730608+YChange01@users.noreply.github.com> Co-authored-by: 郭业昌 <lvpengfei@MacBook-Air.local> Co-authored-by: gure <740684863@qq.com>
270 lines
10 KiB
YAML
270 lines
10 KiB
YAML
name: Build and Test on Linux
|
|
|
|
on:
|
|
push:
|
|
branches: [ master ]
|
|
paths-ignore:
|
|
- '**.md'
|
|
pull_request:
|
|
branches: [ master ]
|
|
paths-ignore:
|
|
- '**.md'
|
|
|
|
env:
|
|
proc_num: $(nproc)
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
|
|
|
# https://github.com/actions/runner-images
|
|
jobs:
|
|
compile-with-make:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- uses: ./.github/actions/install-all-dependencies
|
|
|
|
- name: gcc with default options
|
|
uses: ./.github/actions/compile-with-make
|
|
with:
|
|
options: --headers=/usr/include --libs=/usr/lib /usr/lib64 --cc=gcc --cxx=g++ --werror
|
|
|
|
- name: gcc with all options
|
|
uses: ./.github/actions/compile-with-make
|
|
with:
|
|
options: --headers=/usr/include --libs=/usr/lib /usr/lib64 --cc=gcc --cxx=g++ --werror \
|
|
--with-thrift --with-glog --with-rdma --with-debug-bthread-sche-safety \
|
|
--with-debug-lock --with-bthread-tracer --with-asan
|
|
|
|
- name: clang with default options
|
|
uses: ./.github/actions/compile-with-make
|
|
with:
|
|
options: --headers=/usr/include --libs=/usr/lib /usr/lib64 --cc=clang --cxx=clang++ --werror
|
|
|
|
- name: clang with all options
|
|
uses: ./.github/actions/compile-with-make
|
|
with:
|
|
options: --headers=/usr/include --libs=/usr/lib /usr/lib64 --cc=clang --cxx=clang++ --werror \
|
|
--with-thrift --with-glog --with-rdma --with-debug-bthread-sche-safety \
|
|
--with-debug-lock --with-bthread-tracer --with-asan
|
|
|
|
compile-with-cmake:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- uses: ./.github/actions/install-all-dependencies
|
|
|
|
- name: gcc with default options
|
|
run: |
|
|
export CC=gcc && export CXX=g++
|
|
mkdir gcc_build && cd gcc_build && cmake -DCMAKE_POLICY_VERSION_MINIMUM=3.5 ..
|
|
make -j ${{env.proc_num}} && make clean
|
|
|
|
- name: gcc with all options
|
|
run: |
|
|
export CC=gcc && export CXX=g++
|
|
mkdir gcc_build_all && cd gcc_build_all
|
|
cmake -DWITH_MESALINK=OFF -DWITH_GLOG=ON -DWITH_THRIFT=ON -DWITH_RDMA=ON \
|
|
-DWITH_DEBUG_BTHREAD_SCHE_SAFETY=ON -DWITH_DEBUG_LOCK=ON -DWITH_BTHREAD_TRACER=ON \
|
|
-DWITH_ASAN=ON -DCMAKE_POLICY_VERSION_MINIMUM=3.5 ..
|
|
make -j ${{env.proc_num}} && make clean
|
|
|
|
- name: clang with default options
|
|
run: |
|
|
export CC=clang && export CXX=clang++
|
|
mkdir clang_build && cd clang_build && cmake -DCMAKE_POLICY_VERSION_MINIMUM=3.5 ..
|
|
make -j ${{env.proc_num}} && make clean
|
|
|
|
- name: clang with all options
|
|
run: |
|
|
export CC=clang && export CXX=clang++
|
|
mkdir clang_build_all && cd clang_build_all
|
|
cmake -DWITH_MESALINK=OFF -DWITH_GLOG=ON -DWITH_THRIFT=ON -DWITH_RDMA=ON -DWITH_UBRING=ON \
|
|
-DWITH_DEBUG_BTHREAD_SCHE_SAFETY=ON -DWITH_DEBUG_LOCK=ON -DWITH_BTHREAD_TRACER=ON \
|
|
-DWITH_ASAN=ON -DCMAKE_POLICY_VERSION_MINIMUM=3.5 ..
|
|
make -j ${{env.proc_num}} && make clean
|
|
|
|
gcc-compile-with-make-protobuf:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- uses: ./.github/actions/install-essential-dependencies
|
|
|
|
- name: protobuf 3.5.1
|
|
uses: ./.github/actions/compile-with-make-protobuf
|
|
with:
|
|
protobuf-version: 3.5.1
|
|
protobuf-cpp-version: 3.5.1
|
|
protobuf-install-dir: /protobuf-3.5.1
|
|
config-brpc-options: --cc=gcc --cxx=g++ --werror
|
|
|
|
- name: protobuf 3.12.4
|
|
uses: ./.github/actions/compile-with-make-protobuf
|
|
with:
|
|
protobuf-version: 3.12.4
|
|
protobuf-cpp-version: 3.12.4
|
|
protobuf-install-dir: /protobuf-3.12.4
|
|
config-brpc-options: --cc=gcc --cxx=g++ --werror
|
|
|
|
- name: protobuf 21.12
|
|
uses: ./.github/actions/compile-with-make-protobuf
|
|
with:
|
|
protobuf-version: 21.12
|
|
protobuf-cpp-version: 3.21.12
|
|
protobuf-install-dir: /protobuf-3.21.12
|
|
config-brpc-options: --cc=gcc --cxx=g++ --werror
|
|
|
|
gcc-unittest-with-bazel:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
# Install redis-server/mysql-server so the integration tests that fork a
|
|
# real server (e.g. brpc_redis_unittest) actually run under bazel instead
|
|
# of skipping. Same shared action the make-based unittest jobs use.
|
|
- uses: ./.github/actions/install-essential-dependencies
|
|
- run: bazel test //test/...
|
|
|
|
gcc-compile-with-bazel-all-options:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- run: |
|
|
bazel build --define with_mesalink=false \
|
|
--define with_glog=true \
|
|
--define with_thrift=true \
|
|
--define BRPC_WITH_BORINGSSL=true \
|
|
--define with_debug_bthread_sche_safety=true \
|
|
--define with_debug_lock=true \
|
|
--define with_asan=true \
|
|
--define with_bthread_tracer=true \
|
|
--define BRPC_WITH_NO_PTHREAD_MUTEX_HOOK=true \
|
|
--define with_babylon_counter=true \
|
|
-- //:brpc
|
|
|
|
clang-compile-with-make-protobuf:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- uses: ./.github/actions/install-essential-dependencies
|
|
|
|
- name: protobuf 3.5.1
|
|
uses: ./.github/actions/compile-with-make-protobuf
|
|
with:
|
|
protobuf-version: 3.5.1
|
|
protobuf-cpp-version: 3.5.1
|
|
protobuf-install-dir: /protobuf-3.5.1
|
|
config-brpc-options: --cc=clang --cxx=clang++ --werror
|
|
|
|
- name: protobuf 3.12.4
|
|
uses: ./.github/actions/compile-with-make-protobuf
|
|
with:
|
|
protobuf-version: 3.12.4
|
|
protobuf-cpp-version: 3.12.4
|
|
protobuf-install-dir: /protobuf-3.12.4
|
|
config-brpc-options: --cc=clang --cxx=clang++ --werror
|
|
|
|
- name: protobuf 21.12
|
|
uses: ./.github/actions/compile-with-make-protobuf
|
|
with:
|
|
protobuf-version: 21.12
|
|
protobuf-cpp-version: 3.21.12
|
|
protobuf-install-dir: /protobuf-3.21.12
|
|
config-brpc-options: --cc=clang --cxx=clang++ --werror
|
|
|
|
clang-unittest-with-bazel:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
# Install redis-server/mysql-server so the forked-server integration tests
|
|
# actually run under bazel (see gcc-unittest-with-bazel).
|
|
- uses: ./.github/actions/install-essential-dependencies
|
|
- run: |
|
|
bazel test --test_output=streamed \
|
|
--action_env=CC=clang \
|
|
//test/...
|
|
|
|
clang-compile-with-bazel-all-options:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- run: |
|
|
bazel build --action_env=CC=clang \
|
|
--define with_mesalink=false \
|
|
--define with_glog=true \
|
|
--define with_thrift=true \
|
|
--define BRPC_WITH_BORINGSSL=true \
|
|
--define with_debug_bthread_sche_safety=true \
|
|
--define with_debug_lock=true \
|
|
--define with_asan=true \
|
|
--define with_bthread_tracer=true \
|
|
--define BRPC_WITH_NO_PTHREAD_MUTEX_HOOK=true \
|
|
--define with_babylon_counter=true \
|
|
-- //:brpc
|
|
|
|
clang-unittest:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- uses: ./.github/actions/install-essential-dependencies
|
|
- uses: ./.github/actions/init-ut-make-config
|
|
with:
|
|
options: --with-bthread-tracer
|
|
- name: compile tests
|
|
run: |
|
|
cat config.mk
|
|
cd test
|
|
make -j ${{env.proc_num}}
|
|
- name: run tests
|
|
run: |
|
|
cd test
|
|
sh ./run_tests.sh
|
|
|
|
clang-unittest-asan:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- uses: ./.github/actions/install-essential-dependencies
|
|
- uses: ./.github/actions/init-ut-make-config
|
|
with:
|
|
options: --with-bthread-tracer --with-asan
|
|
- name: compile tests
|
|
run: |
|
|
cat config.mk
|
|
cd test
|
|
make NEED_GPERFTOOLS=0 -j ${{env.proc_num}}
|
|
- name: run tests
|
|
run: |
|
|
cd test
|
|
# The redis integration tests (sanity/keys_with_spaces/incr_and_decr/by_components/auth)
|
|
# fork a real redis-server and connect after a fixed 50ms wait; under ASan redis starts
|
|
# too slowly, so they flake here (connection refused). Skip just those under ASan; the
|
|
# redis codec/server tests still run, and the full suite runs in clang-unittest.
|
|
GTEST_FILTER='-RedisTest.sanity:RedisTest.keys_with_spaces:RedisTest.incr_and_decr:RedisTest.by_components:RedisTest.auth' sh ./run_tests.sh
|
|
|
|
clang-unittest-bazel-with-babylon-and-new-pb:
|
|
runs-on: ubuntu-22.04
|
|
env:
|
|
TEST_PROTOBUF_VERSION: "34.1"
|
|
# protobuf >= 34.x uses new ProtoInfo fields (option_deps,
|
|
# extension_declarations) introduced in Bazel 8.x. The repo's
|
|
# .bazelversion (7.2.1) is too old. bazelisk honors USE_BAZEL_VERSION.
|
|
USE_BAZEL_VERSION: "8.3.1"
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
# Install redis-server/mysql-server so the forked-server integration tests
|
|
# actually run under bazel (see gcc-unittest-with-bazel).
|
|
- uses: ./.github/actions/install-essential-dependencies
|
|
- name: Override protobuf version for testing
|
|
run: |
|
|
sed -i -E "s/(bazel_dep\(name = ['\"]protobuf['\"], version = ['\"])[^'\"]+/\1${TEST_PROTOBUF_VERSION}/" MODULE.bazel
|
|
echo "After override:"
|
|
grep -E "bazel_dep\(name = ['\"]protobuf['\"]" MODULE.bazel
|
|
grep -qE "bazel_dep\(name = ['\"]protobuf['\"], version = ['\"]${TEST_PROTOBUF_VERSION}['\"]" MODULE.bazel \
|
|
|| { echo "ERROR: failed to override protobuf version in MODULE.bazel to ${TEST_PROTOBUF_VERSION}"; exit 1; }
|
|
- run: |
|
|
bazel test --action_env=CC=clang --config=rdma \
|
|
--define with_bthread_tracer=true \
|
|
--define with_babylon_counter=true \
|
|
//test/... --test_arg=--gtest_filter=-RdmaRpcTest.*
|