* reject out-of-range port in SplitHostAndPort
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
* parse port forward and clamp to avoid accumulator wrap
---------
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
* skip header fields containing CR/LF in http serialization
* Gate CR/LF header check behind a flag; log at WARNING with sanitized name
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
* Log the skipped header value alongside its name
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
* Use separate IOBufs for request and response in the CR/LF header test
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
---------
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
* Fix bug when parsing zero-length string field in mcpack2pb
UnparsedValue::as_string() resizes the output string to
without checking , where is the value_size of a string
field read from the input. When value_size is 0, underflows
to SIZE_MAX and resize() throws std::length_error, which is not caught
on the request path and therefore crashes the server. Reject such
malformed fields by marking the stream bad so the caller can fail the
request gracefully instead.
* Clear output string when size error
* fix WeightedRandomizedLoadBalancer skipping the last server
SelectServer() draws random_weight from fast_rand_less_than(weight_sum),
i.e. from [0, weight_sum - 1], and then lower_bound()s it against
Server::current_weight_sum, which Add() fills with an inclusive prefix
sum. lower_bound() returns the first server whose prefix sum is >=
random_weight, but a server owns the half-open range
[prefix(i-1), prefix(i)), so the predicate has to be > random_weight.
Because of that the first server in the list also serves
random_weight == prefix(0) and the last server never serves anything at
all, since random_weight can never reach weight_sum. With four servers of
equal weight the measured distribution is 49.8/25.1/25.1/0.0 percent
instead of 25 percent each.
Search for random_weight + 1 so that lower_bound() lands on the first
prefix sum strictly greater than random_weight.
The existing weighted_randomized test does not catch this: its servers
have weights 3/2/5/10 and it only asserts that each rate is within
0.5x~2x of the expected one. The weight-10 server measures 0.448 before
this change and 0.494 after it, both inside that band. Add
weighted_randomized_equal_weight, which uses equal weights so that a
single misplaced slot is visible, and check the rates within 0.9x~1.1x.
Signed-off-by: Anas <156536069+Nas01010101@users.noreply.github.com>
* use upper_bound for the weighted prefix-sum search
upper_bound(random_weight) states the intent directly: the first server whose
inclusive prefix sum is strictly greater than random_weight. It is the same
search as lower_bound(random_weight + 1) without the increment.
Also correct the tolerance comment in the unit test: with run_times=40000 and
p=0.25 the count has sigma ~= 86.6, so the 0.9x~1.1x band is about 11 sigma,
not more than 20.
---------
Signed-off-by: Anas <156536069+Nas01010101@users.noreply.github.com>
* Support flow-controlled gRPC client requests
- Split client request DATA frames according to the peer's connection and
stream flow-control windows.
- Buffer unsent DATA and resume transmission when WINDOW_UPDATE restores
capacity.
- Track pending request bytes per H2 connection and apply
socket_max_unwritten_bytes as an upper bound.
- Reject or reroute new requests once the pending DATA limit is reached.
- Release buffered DATA when an RPC fails, times out, or its stream is
removed.
- Add tests for fragmented transmission, deferred DATA flushing,
pending-byte accounting, and buffer cleanup.
* Fix pending HTTP/2 data limit race
- Check pending DATA capacity atomically with client stream insertion.
- Leave stream and window state unchanged when the limit is exceeded.
- Use an ephemeral port in the gRPC flow-control test.
- Avoid accessing an empty payload buffer in H2 frame tests.
ParseSofaMessage only checked body_size against max_body_size, while
meta_size and the total frame size were left unbounded. A frame with a
large meta_size and zero body_size passed the body_size check and made
the connection keep buffering far beyond the configured limit before the
invalid metadata was rejected. Bound meta_size by max_body_size as well,
consistent with other protocols such as baidu_std and hulu_pbrpc.
Add unit tests covering oversized body and oversized metadata.
* Limit mcpack2pb array item count to the actual payload size
The item count in an mcpack array header is read directly from the
request and was used as-is by the generated parsing code to Reserve()
memory for repeated protobuf fields. A malformed request could claim an
item count up to INT32_MAX and force the server to preallocate ~16GB of
virtual memory, which may abort the process on memory-constrained hosts.
Cap the item count by the remaining bytes of the array (each item
occupies at least one byte) so that the preallocation is bounded by the
request size.
* Fix underflow in mcpack2pb array item count clamping
Wait for the asynchronous span collector to release root client spans
after Controller reset or submission. This prevents LeakSanitizer from
reporting pending test spans when the short-lived test binary exits
before the collector’s first polling interval.
Signed-off-by: Zhengwei Zhu <141622927+ZhengweiZhu@users.noreply.github.com>
Keep the current RPC span alive from Controller until the RPC finishes,
SubmitSpan runs, or the Controller is reset. This lets root client spans
without a local parent be submitted to rpcz instead of being destroyed
after the caller-side temporary shared_ptr goes out of scope.
Child client spans remain linked to their parent through weak local-parent
references and parent-owned client lists, so they are still serialized
under their parent without introducing shared_ptr cycles.
Co-authored-by: lh2debug <lh2debug@163.com>
* build(cxx): align all builds on a C++14 baseline
CMake, Make, tests, tools, and examples now require C++14 so every supported build path shares the same minimum language level. The compiler checks and user-facing documentation reflect that baseline.
* fix(test): keep cores scoped to the failing test
Clear prior cores before each test binary. A passing death test can leave a core that would otherwise be used to diagnose an unrelated later failure.
* revert src/butil/type_traits.h
Controller::set_request/response_checksum_type() previously only
covered the serialized protobuf body; the attachment (if any) was
never protected. Add set_request/response_checksum_attachment(bool)
so callers can opt the attachment into the same checksum.
- baidu_rpc_meta.proto: add RpcMeta.checksum_with_attachment so the
receiver knows whether to fold the attachment into verification.
Defaults to false, so old peers that don't understand the field
keep verifying against the body only (backward compatible).
- Controller: add the two setters/getters, thread the flag through
ClientSettings (Save/ApplyClientSettings) so ParallelChannel/
SelectiveChannel sub-controllers inherit it correctly, and reset it
in ResetPods().
- ChecksumIn: add an optional `attachment' field consumed by checksum
handlers.
- crc32c_checksum.cpp: extend the crc32c over body then attachment (in
that fixed order) when requested.
- baidu_rpc_protocol.cpp: wire checksum_attachment through
SerializeRpcMessage/DeserializeRpcMessage and every client/server
send/receive path; skip it when progressive attachment reading is
enabled since there's no single complete IOBuf to checksum in that
case.
Add brpc_checksum_unittest.cpp covering Crc32cCompute/Crc32cVerify
directly (including corruption/omission/order sensitivity) and an
end-to-end Server/Channel test for both request- and response-side
attachment checksums.
* guard negative value_size in memcache PopStore
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
* memcache: drop only the declared message when value_size is negative
Move the value_size < 0 check ahead of the pop_front so a malformed
reply no longer pops sizeof(header) + extras_length + key_length past
the declared message boundary. Discard exactly
sizeof(header) + total_body_length instead, keeping the following
pipelined responses aligned.
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
---------
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
* add ubring transport
* fix the bug for ub ring transport
* fix the bug for ub ring transport and other
* add the license for ub transport
* Modifying the variable naming style
* optimize the log message and some field name
* fix some bug for ubring
* add some log ubring endpoint
* add todo
* fix the bug for handshake for ub endpoint
* fix the bug for client ub endpoint
* optimize the iobuf file code
* modify the log level
* fix the declare_shm_ubs define not found bug
* add the timer_mgr support for macos and format code style
* fix the bug for macos epoll
* fix the timespece bug
* adaptor the itimerspec for macos platform
* optimize the cmakelist config
* add the ubring docs for ubring transport
* modify some file name and directory structure
* modify some code style and optimize code logical
* add the cmake ubshm transport ci/testing
* add the dependency header
* bug fix the code
* Fix UB shm allocation cleanup crashes (#11)
Co-authored-by: 郭业昌 <lvpengfei@MacBook-Air.local>
* remove the brpc_ubshm_unittest
* Found and fixed two stability issues (#13)
* 修复ubring server端关闭连接coredump问题
* 修复PollIn/PollOut解引用已释放Socket指针的问题
PollIn/PollOut通过ep->_socket(裸指针)读取data socket,当data socket
被销毁时该指针悬空,导致Socket::Address读到垃圾id触发SIGSEGV。
改为存储_socket_id(SocketId),用Address获取引用计数的Socket,
并在整个回调期间持有该引用,避免解引用悬空指针。
* 修复client非正常退出导致UBRING shm残留的问题
client被强杀(SIGTERM/崩溃/OOM)时teardown没跑完,localShm(_C)的
shm_unlink未执行,导致/dev/shm残留_C文件。server的remoteShm只munmap
不unlink(正确),无法清理client的名字。
在握手ESTABLISHED时(client/server都确认对方已mmap自己的localShm)
立即unlink localShm名字。此时对端已持有mmap引用,unlink只删名字不
影响通信;进程任意时刻退出都不会残留文件名。
* Address chenBright's review: use English comments and BAIDU_CACHELINE_ALIGNMENT
- Convert all Chinese comments in ubshm to English (per chenBright's
'Please use English' on ub_endpoint.cpp:723, ub_ring.cpp:337,
shm_ubs.cpp:316, and similar)
- Replace __attribute__((aligned(64))) with BAIDU_CACHELINE_ALIGNMENT
in ubr_msg.h (per chenBright's comment on ubr_msg.h:41)
- Remove unnecessary TODO comment in ub_ring.cpp:551 (per chenBright's
'Unnecessary comments, please delete')
* Remove unused lock macros in thread_lock.h
Per chenBright's review, the functions and macros defined in
thread_lock.h are largely unused. Verified usage across ubshm:
- LOCK_GUARD / UnlockMutex: 8 call sites in shm_ubs.cpp and
ub_ring_manager.cpp, kept.
- SPIN_LOCK_GUARD, R_LOCK_GUARD, W_LOCK_GUARD, SEMAPHORE_WAIT_GUARD,
SEMAPHORE_WAIT_GUARD_WITH_CLOSE and their helper functions
(UnlockSpinLock, UnlockRWLock, PostSem, PostSemWithClose): 0 call
sites, removed.
* Apply chenBright's review on timer_mgr globals
Per chenBright's review on timer_mgr.cpp:32-37:
- Add explicit default values to uninitialized globals
(g_total_timer_num=0, g_max_system_fd=0, g_epoll_execute_thread=0,
g_timer_module_initialized=0)
- Rename globals to snake_case (g_epollFd -> g_epoll_fd,
g_totalTimerNum -> g_total_timer_num, g_timerFdCtxMap ->
g_timer_fd_ctx_map, maxSystemFd -> g_max_system_fd,
g_epollExecuteThread -> g_epoll_execute_thread,
g_timerModuleInitialized -> g_timer_module_initialized)
- maxSystemFd also gains the g_ prefix to match global naming style
Also fix the missing std:: qualifier on atomic_fetch_sub/add/load
(per chenBright's earlier comment on timer_mgr.cpp:80).
* Change CloseTimerFd fd type from uint32_t to int
Per chenBright's review on timer_mgr.cpp:399 (uint32_t -> int).
fd is a system file descriptor; POSIX APIs use int and -1 denotes an
invalid fd, which uint32_t cannot represent. Changed the CloseTimerFd
signature (header + definition) and removed the now-unnecessary
(uint32_t) casts at the two call sites.
* Use BAIDU_LIKELY/BAIDU_UNLIKELY instead of custom __builtin_expect
Per chenBright's review on common.h:27. Rather than redefine the
macros with __builtin_expect directly, forward LIKELY/UNLIKELY to
brpc's standard BAIDU_LIKELY/BAIDU_UNLIKELY (from butil/compiler_specific.h).
The 122 call sites keep using LIKELY()/UNLIKELY() unchanged; only the
macro bodies change, preserving semantics.
* Add unit tests for UBShmEndpoint
Per chenBright's request to add unit tests for UBShmTransport in this
PR (rather than a follow-up).
Adds test/brpc_ubring_unittest.cpp with tests covering the public
interface of UBShmEndpoint under the g_skip_ub_init=true mode (which
skips real shared-memory/poller setup):
- construct_and_destruct: lifecycle safety
- is_writable_false_when_skip_init: skip-mode behavior
- reset_is_idempotent: Reset() is safe to call repeatedly
The file follows the brpc_*_unittest.cpp naming convention so it is
auto-collected by test/CMakeLists.txt's file(GLOB). Verified: compiles,
links, and all 3 tests pass (g++ 15.2, C++17, gtest, BRPC_WITH_UBRING=ON).
* Rewrite UBShmEndpoint unit tests with real coverage
Per chenBright's feedback that the previous tests were too simple and
did not cover the main methods.
Source changes to enable testing:
- Move HelloMessage struct declaration from ub_endpoint.cpp to
ub_endpoint.h so tests can access it
- Expose private members under #ifdef UNIT_TEST (precedent:
butil/containers/stack_container.h) so tests can call
AllocateClientResources without -Dprivate=public (which breaks
GCC 15 + new libstdc++ <any>/<sstream>)
Tests (9, all passing on Ubuntu 26.04 g++ 15.2 C++17 gtest):
HelloMessageTest (5): serialize/deserialize roundtrip, network byte
order verification, uint64 max boundary, full shm_name, toString
UBShmEndpointTest (4): construct, real IPC shm
AllocateClientResources (g_skip_ub_init=false), reset cleanup, reset
idempotency
* rename the variable to snake_case style
---------
Co-authored-by: YeChang Guo <52730608+YChange01@users.noreply.github.com>
Co-authored-by: 郭业昌 <lvpengfei@MacBook-Air.local>
Co-authored-by: gure <740684863@qq.com>
Previously BringUpQp only did a local ibv_query_ece + ibv_set_ece
roundtrip and never exchanged ECE capabilities with the peer.
This patch wires up the standard requestor/responder ECE negotiation
flow on top of the existing v3 handshake without adding any extra
round trip:
1. Client queries local ECE, advertises it in its v3 hello.
2. Server applies the client's ECE in INIT->RTR (set_ece), then
after RTS queries the reduced/negotiated ECE and sends it back
in the reply hello.
3. Client applies the server's reduced ECE in INIT->RTR.
TimerThread only reclaims a Task's pooled slot when the task is popped at
its run_time. A task unscheduled after being pulled into the internal heap
therefore keeps its slot until run_time, and tasks that pile up in the
buckets while the thread sleeps on a far-future deadline are not consumed
(and thus not reclaimed) until that deadline. With large timeouts the live
Task count grows to ~ qps * timeout even though almost all of those tasks
were unscheduled long ago.
Add two independent, bounded reclamation paths:
- Heap sweep: when the internal heap grows past
brpc_timer_heap_sweep_min_size (default 4096) and has roughly doubled
since the last sweep, drop unscheduled tasks from it. The trigger is a
timer-thread-local heuristic, so the unschedule() hot path is unchanged
(no new shared atomics/contention). Amortized O(1) per task.
- Periodic wakeup: cap the sleep at brpc_timer_max_wakeup_interval_ms
(default 0 = disabled, legacy behavior) so the thread wakes up to drain
the buckets and sweep the heap even when every pending task is far in
the future. An empty heap still sleeps until woken by schedule().
Expose pending_task_count() and allocated_task_count() for observability,
and add unit tests covering both the heap-retention and bucket-accumulation
cases (each verified to fail with the corresponding fix disabled).
* check field length against remaining buffer in mysql Field::Parse
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
* test: reject oversized text field length in MysqlReply parse
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
---------
Signed-off-by: ubeddulla khan <ubed@bugqore.com>
Route accesses to tls_task_group, tls_task_group_nosignal, tls_bls and
several mutex-profiling thread-locals (tls_inside_lock, tls_warn_up,
tls_pthread_lock_count, tls_csites, tls_ever_created_keytable) through
the BAIDU_(GET|SET|GET_PTR)_VOLATILE_THREAD_LOCAL macros instead of
touching the raw __thread variables directly.
On aarch64/clang the compiler can incorrectly cache the address of a
thread_local variable across a suspend point (e.g. when a bthread is
rescheduled onto another worker), so raw accesses can silently read a
stale TaskGroup*/LocalStorage. Going through the accessor macros
everywhere forces a fresh, non-cached load/store on those platforms
while staying a plain variable access elsewhere.
Add bthread::tls_bls_ptr() as the single helper for obtaining the
current LocalStorage*, and update all call sites (bthread.cpp,
butex.cpp, fd.cpp, key.cpp, mutex.cpp, task_control.cpp,
brpc/controller.cpp, test/bthread_unittest.cpp) to use it instead of
declaring their own stale/mistyped extern references to tls_task_group
or tls_bls.
Also hardens borrow_keytable()/return_keytable() in key.cpp: the
KeyTableList pointer obtained under the read lock is no longer reused
after re-acquiring the write lock, since pool->list/pool->destroyed
may change concurrently via bthread_keytable_pool_destroy(); the
thread-local list is drained first before falling back to the pool's
global free list.
Controller::HandleStreamConnection called s->SetHostSocket() on the first
stream inside the extra-stream loop. When an extra stream carried no
downstream data (nothing triggered Stream::OnReceived to set host_socket
for it), Stream::SetConnected() hit CHECK(_host_socket != NULL).
Set host_socket on the extra stream itself, matching the server-side logic
in baidu_rpc_protocol.
Co-authored-by: jiangyt-git <64257436+jiangyt-git@users.noreply.github.com>
Co-authored-by: jiangyuting <jiangyutingwangyi@163.com>
* Add Power-of-Two-Choices Peak-EWMA load balancer (p2c)
Implements the p2c load balancing policy proposed in #3340: each
selection samples two random servers (configurable via choices=N) and
routes to the lower peak-EWMA latency * (inflight+1) / weight score.
Upward latency spikes take effect immediately while recovery decays
over tau_ms (default 10s), so a degraded server is shed within one
observation at O(1) selection cost.
Uses only existing LoadBalancer hooks (SelectServer/Feedback,
need_feedback) with DoublyBufferedData membership like rr/la; per-node
stats are shared_ptr-owned by both buffers. Registered as "p2c" in
global.cpp. Includes unit tests (functional, weighted, exclusion,
error punishment, concurrency churn) and docs in cn/en client.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reuse SelectIn.begin_time_us to avoid extra clock read per selection
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: gflags for tunables, cap error punishment, drop FlatMap init
- p2c_default_choices/p2c_default_tau_ms gflags as channel-overridable defaults
- p2c_max_punish_ms(default 30s) caps the doubled-EWMA error punishment so a
persistently failing server recovers in bounded time after turning healthy
- Servers::server_map relies on FlatMap small-map auto-init
- tests: error_punish_is_capped + feedback_lock_overhead benchmark
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: rajvarun77 <287367605+rajvarun77@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* build(cmake): keep modern CMake policy compatibility
Raise the minimum CMake version range to 3.16...3.28 so CMake 4.x no longer treats the project as relying on removed pre-3.5 policy compatibility. The range preserves a conservative 3.16 runtime floor while documenting policy validation through 3.28.
* build(cmake): modernize target-scoped CMake configuration
Move shared compile definitions, include paths, options, and link dependencies onto interface targets so top-level, src, tools, tests, and standalone examples consume consistent build settings without relying on global directory state.
Examples now share a small CMake helper, and the gtest download path is updated to configure with modern CMake while keeping downloaded gtest headers ahead of Homebrew's C++17-only installation.
* Fix StreamCreate failure handling
* Add ScopedStream guard to stream test
* Move stream regression test into streaming UT
* Fix StreamCreate failure handling
* Add ScopedStream guard to stream test
* Move stream regression test into streaming UT
UB