Skip to main content

​
Internal testing audit

Pro. For “how do I know a Penthera release is sound?”
Penthera is a security tool, so its own release bar has to be easy to check. The internal audit is that bar: one command that runs every check a release needs and writes the result to audit/REPORT.md.
It exits 1 if any gate fails. A release is only tagged from a commit whose audit passed (see the release checklist in CONTRIBUTING.md). The same audit runs every Monday in CI (internal-audit.yml) so drift shows up between releases.

​
The gates

Everything runs on this machine against this repo or 127.0.0.1. The only network use is npm audit (the npm registry) and, with --with-docker, pulling the Juice Shop image.

​
Options

Command output from each run goes to audit/logs/ (ignored by git, uploaded with the report as the CI artifact).

​
Optional targets: Juice Shop and crAPI

--with-docker starts OWASP Juice Shop on 127.0.0.1:3300, scans it with the benchmark’s juice-shop target, and removes the container. Pin the image for reproducible numbers with JUICE_SHOP_IMAGE=bkimminich/juice-shop@sha256:.... OWASP crAPI needs its own Docker Compose stack, so the audit scans it only if it is already running at CRAPI_URL (default http://127.0.0.1:8888). Start it with the project’s compose file first. Both are skipped by default. Neither has labelled ground truth yet, so the report lists their finding counts and shows precision and recall as “unlabelled” until they are labelled in bench/ground-truth.json (see bench/README.md).

​
When a gate fails

  • Tests, smoke, offline, packaging, supply chain, versions: fix the cause. These describe what Penthera promises its users.
  • Benchmark: if the change is intended (a new rule that trades a little precision for recall, a relabelled target), run npm run bench, commit the new bench/RESULTS.md, and say why in the PR.
  • Self-scan: fix the finding. If it is a deliberate fixture, run --update-baseline and replace the TODO with the reason, so the acceptance is reviewed like any other change.
Thresholds (the 0.02 benchmark drop, the package size budget, the allow-list) live at the top of scripts/audit/run.mjs. Change them in their own PR, never to make a release pass.

​
The report

audit/REPORT.md has a pass/fail table with the numbers behind each gate, the changes since the previous report (read from a data block at the end of the previous file), and per-gate details. Commit it with each release so the history of the gate lives in git.