Files
bplaxco 782f310478
gitleaks / gitleaks (push) Has been cancelled
Archive support (#1872)
* Start Files Source

* Keep cmd the same for now

* Get tests to pass

* s.MaxTargetMegaBytes -> s.MaxFileSize

* Tweak some log messages & use q semgroup for files

* Just call yield inside the semgroup

* Refactor a check and revert some log lines

* Move mg && misc tuning

* Add `make profile`

(Benchmark results are only added if tools are installed)

Example output:

```
========================================================================
generating profile data
------------------------------------------------------------------------
- mode: dir
  benchmark:
    tool: hyperfine
    path: profile/1748019612/dir/benchmark.json
    results:
      mean: 1.12852930166
      stddev: 0.12163775966037399
      median: 1.07242928416
      user: 3.9699382799999996
      system: 0.04382341999999999
      min: 1.03207673016
      max: 1.3913475181600001
  profile:
    - mode: cpu
      path: profile/1748019612/dir/cpu.pprof
      view: go tool pprof -http=localhost: ./gitleaks profile/1748019612/dir/cpu.pprof
    - mode: mem
      path: profile/1748019612/dir/mem.pprof
      view: go tool pprof -http=localhost: ./gitleaks profile/1748019612/dir/mem.pprof
    - mode: trace
      path: profile/1748019612/dir/trace.out
      view: go tool trace profile/1748019612/dir/trace.out
- mode: git
  ...snip...
```

* Archive support for File based sources

* Move Git to Source interface

* Scan archives in git

* Misc linter fixes

* Exclude certain diffs: --diff-filter=tuxdb

* Add git archives tests

* Log error instead of fatal

* Fix spacing in test config

* Refine source comments/wording

* Make MB conversions more readable

* Format cmd

* Adjust DetectSource logging

* MiB -> MB in logs

* Add testdata/repos/archives/dotGit/refs/.gitkeep

Without it the refs folder gets deleted and the
and then it isn't detected as a git repo ¯\_(ツ)_/¯

So then the test starts scanning the whole gitleaks
repo instead :D

* Handle finding links for archives

* Working on archive depth

* Fixes from failfast test

* cmd max-archive-depth

* added comments, fixed warnings

* removed test print

* Update README.md

* Set archive depth for DetectFiles

* Tweak logs & pass max archive for readers

* Pass ctx to Fragments

* Add make lint target and address some issues

* Fix compat issue

* Report path in fragment errors

* Apply PR suggestions & misc changes

- Update blobReader.Close() to discard the buffer
- Misc logger issues & uses
- Tweak .golangci.yaml to default to none
- Discard remaining blobReader data on close
- Undo a De Morgan's Law suggestion (and disable QF1001)

* Add http mode for diagnostics

gitleaks --diagnostics=http ...
-----

From the net/http/pprof docs:

Use the pprof tool to look at the heap profile:
```
go tool pprof http://localhost:6060/debug/pprof/heap
```

Or to look at a 30-second CPU profile:
```
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
```

Or to look at the goroutine blocking profile, after calling runtime.SetBlockProfileRate in your program:
```
go tool pprof http://localhost:6060/debug/pprof/block
```

Or to look at the holders of contended mutexes, after calling runtime.SetMutexProfileFraction in your program:
```
go tool pprof http://localhost:6060/debug/pprof/mutex
```

(For more info see https://pkg.go.dev/net/http/pprof)

---------

Co-authored-by: Alex Layne <alayne@redhat.com>
2025-06-01 11:29:34 -05:00
..
2025-06-01 11:29:34 -05:00