* dev: Update `nanobind_bazel` and pre-commit hooks
Also add `uv.lock` to the list of ignored Python build artifacts.
* Fix mypy unused-variable warnings
An underscore is the preferred option for ignoring an unused variable obtained
from a structured binding (i.e., tuple unpacking) in Python.
The changes are an autofix added in Buildifier 8.0.1, designed to future-proof
Bazel projects against the eventual removal of these rules from the native
Bazel namespace.
As a fix, also turn the comment in libpfm's build file into a proper Starlark
docstring.
Co-authored-by: dominic <510002+dmah42@users.noreply.github.com>
* Add a bzlmod Python bindings build
Uses the newly started `@nanobind_bazel` project to build nanobind
extensions. This means that we can drop all in-tree custom build defs
and build files for nanobind and the C++ Python headers.
Additionally, the temporary WORKSPACE overwrite hack naturally goes away
due to the WORKSPACE system being obsolete.
* Bump ruff -> v0.3.1, change ruff settings
The latest minor releases incurred some formatting and configuration
changes, this commit rolls them out.
---------
Co-authored-by: dominic <510002+dmah42@users.noreply.github.com>
* CMake: `get_git_version()`: just use `--dirty` flag of `git describe`
* CMake: move version normalization out of `get_git_version()`
Mainly, i want `get_git_version()` to return true version,
not something sanitized.
* JSON reporter: store library version and schema version in `context`
* Tools: discard inputs with unexpected `json_schema_version`
* Extract version string into `GetBenchmarkVersiom()`
---------
Co-authored-by: dominic <510002+dmah42@users.noreply.github.com>
Also fix a mypy error in `tools.gbench.util` - the condition behaves the
same as before, but in the new mypy version, the old condition results
in an unreachable code error for the final `return False` statement.
This is most likely a bug in mypy's reachability analysis, but the fix
is easy enough here to circumvent it.
* tools/compare: when dumping json, pretty-print it
It's rather completely non-human-readable otherwise.
I can't imagine the filesize really matters,
and if it does, it should just be compressed later on.
* tools/compare: add failing test
* tools/compare: don't actually discard valid (but zero) `pvalue`
So, this is embarressing. For a very large number of repetitions,
we can end up with pvalue of a true zero, and it obviously compares false,
and we treat it as-if we failed to compute it...
* Add pre-commit config and GitHub Actions job
Contains the following hooks:
* buildifier - for formatting and linting Bazel files.
* mypy, ruff, isort, black - for Python typechecking, import hygiene,
static analysis, and formatting.
The pylint CI job was changed to be a pre-commit CI job, where pre-commit
is bootstrapped via Python.
Pylint is currently no longer part of the
code checks, but can be re-added if requested. The reason to drop was
that it does not play nicely with pre-commit, and lots of its
functionality and responsibilities are actually covered in ruff.
* Add dev extra to pyproject.toml for development installs
* Clarify that pre-commit contains only Python and Bazel hooks
* Add one-line docstrings to Bazel modules
* Apply buildifier pre-commit fixes to Bazel files
* Apply pre-commit fixes to Python files
* Supply --profile=black to isort to prevent conflicts
* Fix nanobind build file formatting
* Add tooling configs to `pyproject.toml`
In particular, set line length 80 for all Python files.
* Reformat all Python files to line length 80, fix return type annotations
Also ignores the `tools/compare.py` and `tools/gbench/report.py` files
for mypy, since they emit a barrage of errors which we can deal with
later. The errors are mostly related to dynamic classmethod definition.
* Migrate to bzlmod
* Update Python version to PY3, as indicated by the actual source file.
* Migrate more libraries & first draft of direct pywheel rule usage in Bazel
* Integrate with nanobind and libpfm
* Make Python toolchain a dev dependency
* Undo py_wheel usage until later
* Added support for bzlmod for C++ parts of google_benchmark.
* Make //tools:all buildable with --enable_bzlmod
---------
Co-authored-by: Andy Christiansen <achristiansen@google.com>
Previously compare.py ignored the --benchmarks_filter
argument when loading JSON. This defeated any workflow when
a single run of the benchmark was run, followed by multiple
"subset reports" run against it with the 'benchmarks'
command.
Concretely this came up with the simple case:
compare.py benchmarks a.json b.json --benchmarks_filter=BM_Example
This has no practical impact on the 'filters' and
'benchmarkfiltered' comand, which do their thing at a later
stage.
Fixes#1484
Co-authored-by: dominic <510002+dmah42@users.noreply.github.com>
* Build libpfm as a dependency to allow collection of perf counters
This commit builds libpfm using rules_foreign_cc and lets the default
build of the benchmark library support perf counter collection without
needing additional work from users.
Tested with a custom target:
```
bazel run \
--override_repository=com_github_google_benchmark=/home/raghu/benchmark \
-c opt :test-bench -- "--benchmark_perf_counters=INSTRUCTIONS,CYCLES"
Using profile: local
<snip>
----------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
----------------------------------------------------------------------
BM_Test 0.279 ns 0.279 ns 1000000000 CYCLES=1.00888 INSTRUCTIONS=2
```
Signed-off-by: Raghu Raja <raghu@enfabrica.net>
* Adding myself to the CONTRIBUTORS file per CLA guidance
Enfabrica has already signed a corporate CLA.
Signed-off-by: Raghu Raja <raghu@enfabrica.net>
Signed-off-by: Raghu Raja <raghu@enfabrica.net>
Some linux distributions no longer provide `python` binary and require
usage of `python3` instead. This changes the scripts here and uses
cmake `find_package(Python3` when running python.
Co-authored-by: Dominic Hamon <dominichamon@users.noreply.github.com>
The pandas.Timedelta class truncates to integral nanoseconds, which throws
away sub-nanosecond precision present in benchmark JSON. Switch to
floating point multiplication, which preserves it.
Fixes#1482
Tentatively fixes#1477.
* Add possibility to ask for libbenchmark version number (#1004)
Add a header which holds the current major, minor, and
patch number of the library. The header is auto generated
by CMake.
* Do not generate unused functions (#1004)
* Add support for version number in bazel (#1004)
* Fix clang format #1004
* Fix more clang format problems (#1004)
* Use git version feature of cmake to determine current lib version
* Rename version_config header to version
* Bake git version into bazel build
* Use same input config header as in cmake for version.h
* Adapt the releasing.md to include versioning in bazel
Despite the wide variety of the features we provide,
some people still have the audacity to complain and demand more.
Concretely, i *very* often would like to see the overall result
of the benchmark. Is the 'new' better or worse, overall,
over all the non-aggregate time/cpu measurements.
This comes up for me most often when i want to quickly see
what effect some LLVM optimization change has on the benchmark.
The idea is straight-forward, just produce four lists:
wall times for LHS benchmark, CPU times for LHS benchmark,
wall times for RHS benchmark, CPU times for RHS benchmark;
then compute geomean for each one of those four lists,
and compute the two percentage change between
* geomean wall time for LHS benchmark and geomean wall time for RHS benchmark
* geomean CPU time for LHS benchmark and geomean CPU time for RHS benchmark
and voila!
It is complicated by the fact that it needs to graciously handle
different time units, so pandas.Timedelta dependency is introduced.
That is the only library that does not barf upon floating times,
i have tried numpy.timedelta64 (only takes integers)
and python's datetime.timedelta (does not take nanosecons),
and they won't do.
Fixes https://github.com/google/benchmark/issues/1147
* Statistics: add support for percentage unit in addition to time
I think, `stddev` statistic is useful, but confusing.
What does it mean if `stddev` of `1ms` is reported?
Is that good or bad? If the `median` is `1s`,
then that means that the measurements are pretty noise-less.
And what about `stddev` of `100ms` is reported?
If the `median` is `1s` - awful, if the `median` is `10s` - good.
And hurray, there is just the statistic that we need:
https://en.wikipedia.org/wiki/Coefficient_of_variation
But, naturally, that produces a value in percents,
but the statistics are currently hardcoded to produce time.
So this refactors thinkgs a bit, and allows a percentage unit for statistics.
I'm not sure whether or not `benchmark` would be okay
with adding this `RSD` statistic by default,
but regales, that is a separate patch.
Refs. https://github.com/google/benchmark/issues/1146
* Address review notes
The test used to work with scipy 1.6, but does not work with 1.7.1
I believe this is because of https://github.com/scipy/scipy/pull/4933
Since there doesn't appear that there is anything wrong with how
we call the `mannwhitneyu()`, i guess we should just expect the new values.
Notably, the tests now fail with earlier scipy versions.
Currently, the tooling just keeps the whatever benchmark order
that was present, and this is fine nowadays, but once the benchmarks
will be optionally run interleaved, that will be rather suboptimal.
So, now that i have introduced family index and per-family instance index,
we can define an order for the benchmarks, and sort them accordingly.
There is a caveat with aggregates, we assume that they are in-order,
and hopefully we won't mess that order up..
dependabot[bot]