This commit adds detailed documentation for all benchmark command-line
options to the user guide. Each option is documented with:
- Description of what the option does
- Default value (where applicable)
- Valid values (where applicable)
- Example usage
The documentation is organized into logical categories:
- Benchmark Selection and Execution
- Timing and Repetition Control
- Output Formatting
- Reporting Options
- Performance Counters and Context
- Miscellaneous
This addresses issue #2156 where users requested public documentation
of command-line options instead of having to run --help.
Closes#2156
* Add BENCHMARK_NAMED macro for named benchmarks without lambda
Closes#2128. BENCHMARK_CAPTURE creates a lambda even when no arguments
are captured, causing compiler/linker scalability issues with thousands
of benchmarks. BENCHMARK_NAMED provides the same func/name format but
passes the function pointer directly (no lambda), consistent with the
existing BENCHMARK macro.
* Move BENCHMARK_NAMED test to register_benchmark_test with name assertions
* Add Benjamin King to AUTHORS and CONTRIBUTORS
* Fix clang-format: remove trailing spaces from BENCHMARK_NAMED macro
* Fix clang-format: remove extra blank line left after test removal
---------
Co-authored-by: Roman Lebedev <lebedev.ri@gmail.com>
While ASLR is a useful security hardening feature,
it introduces unreproducible noise into benchmarks,
and we really really really don't want any noise,
especially easily avoidable one.
Unless prevented by some other security hardening,
we can disable ASLR for the current process,
and restart it, thus eliminating this noise.
Fixes https://github.com/google/benchmark/issues/461
* `CMakeLists.txt`: drop hopefully obsolete code
* README.md: update
* Unbreak `BENCHMARK_HAS_CXX11` macro
835365f99a stopped defining it,
but didn't un-conditionalize the code guarded under it...
* Drop `BENCHMARK_HAS_NO_VARIADIC_REGISTER_BENCHMARK`
We no longer support such an old gcc version
* `docs/user_guide.md`: proofread
* Add a test to ensure that `benchmark.h` remains C++14 header
* Revert `[[maybe_unused]]` changes - it requires C++17
* Also support C++11 standard for using the library
I don't think we want to support C++03 though,
but i suppose C++11 is palatable, at least right now.
This API is akin to the MemoryManager API and lets tools provide
their own profiler which is wrapped in the same way MemoryManager is
wrapped. Namely, the profiler provides Start/Stop methods that are called
at the start/end of running the benchmark in a separate pass.
Co-authored-by: dominic <510002+dmah42@users.noreply.github.com>
* Clarify BENCHMARK_REGISTER_F
Add comments highlighting the difference between `BENCHMARK_TEMPLATE_F` and `BENCHMARK_TEMPLATE_DEFINE_F`, mirroring those of `BENCHMARK_F ` and `BENCHMARK_DEFINE_F`.
* More informative comments.
* Update user_guide.md
Test coverage isn't great, but not worse than the existing one.
You'd think `BENCHMARK_CAPTURE` would suffice,
but you can't pass `func<targs>` to it (due to the `<` and `>`),
and when passing `(func<targs>)` we get issues with brackets.
So i'm not sure if we can fully avoid this helper.
That being said, if there is only a single template argument,
`BENCHMARK_CAPTURE()` works fine if we avoid using function name.
* Update AUTHORS/CONTRIBUTORS
* Fix examples with deprecated DoNotOptimize API
The const-reference API to DoNotOptimize was deprecated with #1493. Some
examples in the user guide are using exactly that deprecated interface.
This fixes that by passing non-const lvalues instead. Fixes#1566
* Discuss sources of variance in the user guide
* Mention cpufreq/boost
* Pull variance material into a new document
Add reducing_variance.md as a place to discuss things related to variance
and, in the future, statistical interpretation of benchmark results.
Co-authored-by: Dominic Hamon <dominichamon@users.noreply.github.com>
* Introduce warmup phase to BenchmarkRunner (#1130)
In order to account for caching effects in user
benchmarks introduce a new command line option
"--benchmark_min_warmup_time"
which allows to specify an amount of time for
which the benchmark should be run before results
are meaningful.
* Adapt review suggestions regarding introduction of warmup phase (#1130)
* Fix BM_CHECK call in MinWarmUpTime (#1130)
* Fix comment on requirements of MinWarmUpTime (#1130)
* Add basic description of warmup phase mechanism to user guide (#1130)
* Filter out benchmarks that start with "DISABLED_"
This could be slightly more elegant, in that the registration and the
benchmark definition names have to change. Ideally, we'd still register
without the DISABLED_ prefix and it would all "just work".
Fixes#1365
* add some documentation
* Add option to set the default time unit globally
This commit introduces the `--benchmark_time_unit={ns|us|ms|s}` command line argument. The argument only affects benchmarks where the time unit is not set explicitly.
* Update AUTHORS and CONTRIBUTORS
* Test `SetDefaultTimeUnit`
* clang format
* Use `GetDefaultTimeUnit()` for initializing `TimeUnit` variables
* Review fixes
* Export functions
* Add comment
* Revert "Refine docs on changing cpufreq governor (#1325)"
This reverts commit 9e859f5bf5.
* Refine the User Guide CPU Frequency Scaling section
The text now describes the cpupower command, so users in a hurry
have something to copy/paste that will likely work. It then
suggests that there are probably more convenient optons available
that people can look into.
This reverts the prior commit, which introduced a shell script
that doesn't work. It also retains the spirit of the original
fix: no longer recommend setting the frequency governor to
"powersave", which might not be appropriate or available.
Note: I did attempt to write a bash script that set the govenor
to "powersave" for the duration of a single command, but I gave
up for many reasons:
1) it got complex, in part because the cpupower command does not
seem to be designed for scripts (e.g. it prints out complex
English phrases).
2) munging /proc/sys files directly feels unstable and less than
universal. The libcpupower and cpupower are designed to abstract
those away, because the details can vary.
3) there are better options. E.g. various GUI programs, and
even Gnome's core Settings UI, let you adjust the system's
performance mode without root access.
Fixes#1325, #1327
* Add Setup/Teardown option on Benchmark.
Motivations:
- feature parity with our internal library. (which has ~718 callers)
- more flexible than cordinating setup/teardown inside the benchmark routine.
* change Setup/Teardown callback type to raw function pointers
* add test file to cmake file
* move b.Teardown() up
* add const to param of Setup/Teardown callbacks
* fix comment and add doc to user_guide
* fix typo
* fix doc, fix test and add bindings to python/benchmark.cc
* fix binding again
* remove explicit C cast - that was wrong
* change policy to reference_internal
* try removing the bindinds ...
* clean up
* add more tests with repetitions and fixtures
* more comments
* init setup/teardown callbacks to NULL
* s/nullptr/NULL
* removed unused var
* change assertion on fixture_interaction::fixture_setup
* move NULL init to .cc file
* Allow template arguments to be specifed directly on the BENCHMARK macro/
Use cases:
- more convenient (than having to use a separate BENCHMARK_TEMPLATE)
- feature parity with our internal library.
* fix tests
* updated docs
* Introduce Coefficient of variation aggregate
I believe, it is much more useful / use to understand,
because it is already normalized by the mean,
so it is not affected by the duration of the benchmark,
unlike the standard deviation.
Example of real-world output:
```
raw.pixls.us-unique/GoPro/HERO6 Black$ ~/rawspeed/build-old/src/utilities/rsbench/rsbench GOPR9172.GPR --benchmark_repetitions=27 --benchmark_display_aggregates_only=true --benchmark_counters_tabular=true
2021-09-03T18:05:56+03:00
Running /home/lebedevri/rawspeed/build-old/src/utilities/rsbench/rsbench
Run on (32 X 3596.16 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x16)
L1 Instruction 32 KiB (x16)
L2 Unified 512 KiB (x16)
L3 Unified 32768 KiB (x2)
Load Average: 7.00, 2.99, 1.85
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations CPUTime,s CPUTime/WallTime Pixels Pixels/CPUTime Pixels/WallTime Raws/CPUTime Raws/WallTime WallTime,s
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
GOPR9172.GPR/threads:32/process_time/real_time_mean 11.1 ms 353 ms 27 0.353122 31.9473 12M 33.9879M 1085.84M 2.83232 90.4864 0.0110535
GOPR9172.GPR/threads:32/process_time/real_time_median 11.0 ms 352 ms 27 0.351696 31.9599 12M 34.1203M 1090.11M 2.84336 90.8425 0.0110081
GOPR9172.GPR/threads:32/process_time/real_time_stddev 0.159 ms 4.60 ms 27 4.59539m 0.0462064 0 426.371k 14.9631M 0.0355309 1.24692 158.944u
GOPR9172.GPR/threads:32/process_time/real_time_cv 1.44 % 1.30 % 27 0.0130136 1.44633m 0 0.0125448 0.0137802 0.0125448 0.0137802 0.0143795
```
Fixes https://github.com/google/benchmark/issues/1146
* Be consistent, it's CV, not 'rel std dev'
* Statistics: add support for percentage unit in addition to time
I think, `stddev` statistic is useful, but confusing.
What does it mean if `stddev` of `1ms` is reported?
Is that good or bad? If the `median` is `1s`,
then that means that the measurements are pretty noise-less.
And what about `stddev` of `100ms` is reported?
If the `median` is `1s` - awful, if the `median` is `10s` - good.
And hurray, there is just the statistic that we need:
https://en.wikipedia.org/wiki/Coefficient_of_variation
But, naturally, that produces a value in percents,
but the statistics are currently hardcoded to produce time.
So this refactors thinkgs a bit, and allows a percentage unit for statistics.
I'm not sure whether or not `benchmark` would be okay
with adding this `RSD` statistic by default,
but regales, that is a separate patch.
Refs. https://github.com/google/benchmark/issues/1146
* Address review notes
Refactoring in 201b981a moved most of the documentation from `README.md` to `docs/user_guide.md`. Some links from `README.md` to other `docs/*.md` files ended up unchanged in `docs/user_guide.md`. Those links were now broken as they did not link from outside the `docs` directory anymore, but from inside it. Removing the leading `docs/` for these links fixes this.
Haihan Jiang