Every entry says what the number was before and what it is now. Where a fix made our own score worse, the old number is still printed next to it — a changelog you can only win on isn't a changelog. Method and code are published so anything here can be reproduced or disputed.
Our docs always said this endpoint needs a key, but it answered without one. The new contract check caught the difference. Each enrichment call queries several third-party reputation services on our accounts, so we closed it rather than changing the docs. If you call it without a key, register for a free key (about 30 seconds) and send it as a Bearer header. The same applies to /api/v1/threat-journey/{ip}/enrichment, which calls it underneath. The Jeevesus MCP enrich-ioc tool, which reads only our own corpus, stays public.
A collector that reads vendor research pulled every IP address out of the prose, including the ones a write-up uses as lab or example addresses. 46 of them reached the default ips.csv at confidence 75, above its default floor of 50: private ranges such as 192.168.1.1 and 10.0.0.0, the cloud metadata address 169.254.169.254, documentation ranges, multicast, and the public resolvers 8.8.8.8 and 1.1.1.1. They first appear in the feed on October 13 and October 15. Our own edge blocks at 80, so it never acted on them. A customer blocking from ips.csv at the default floor could have blocked their own gateway or DNS.
Fixed three ways: the records are suppressed (kept, at confidence 0, so the history stays auditable); the collectors drop non-global and resolver addresses before they are written; and every feed lane now withholds them at any confidence, so one that slips in some other way still never ships. If you block from ips.csv, check your blocklist for private and resolver addresses and re-pull.
The endpoint catalog behind /docs and its OpenAPI file is checked against production: a documented endpoint that answers 404, or whose real authentication differs from what the docs say, fails the check. Its first run found one of each kind. /api/v1/search/stats was documented as needing a key and is public, so the docs now say public. The docs and this page now publish without a deploy, and a breaking change to the documented API cannot publish without an entry here.
The package check used to judge every non-npm package as if it were from PyPI, so a RubyGems name got an answer about a different registry. It now checks RubyGems on its own terms, and an ecosystem it does not cover returns unknown, which is not a safety judgment, instead of a borrowed one. The malicious-package feed now syncs RubyGems, crates.io and Go from OSV every hour alongside npm and PyPI.
Sixteen days of autonomous blocking, counted by distinct hosts rather than requests — one scanner knocking four thousand times is one attacker, not four thousand saves.
A blocklist working alone would have stopped 55 distinct hosts. The behavioural shield stopped 1,638. Roughly thirty times more, zero human decisions, effectively zero marginal cost. 81.3% of the machines that actually attacked us were on no blocklist anywhere.
18.7% were held before contact. 97.5% of the armed list never fired at all — that is coverage in reserve, which is the point of holding an indicator rather than a failure to use it.
Rows are not indicators. Our store keeps point observations, so one address can carry dozens of records. Corpus-wide inflation was 2.6×.
It is not spread evenly, and that turned out to be the more useful finding: one aggregated feed accounted for 953,833 rows describing 19,579 indicators — a factor of 48.7. Every curated feed we carry, and every source we built ourselves, came in at 1.00×. Aggregation inflates. Primary research does not.
Most of that corpus carries no request path and no user-agent, because nobody ever knocked: they are proactive listing decisions taken on reputation. Real observed attacks were the remaining 1.5%. Quoting the big number as “threats stopped” would have been defensible word by word and false as an impression.
Block records carried a timestamp set when a scheduled sync ran, not when the block happened — eight consecutive events spanning thirty-five real minutes shared two timestamps. Attack peaks were flattened by about a third and smeared into neighbouring hours.
Backfilled a true event time across the corpus. Every temporal claim made before this was quietly measuring job scheduling.
Legitimate crawlers are verified by Cloudflare's verified-bot signal or forward-confirmed reverse DNS, never by the user-agent string they claim. Fails closed.
Worth reporting what the measurement found: of four crawler families we first flagged as wrongly blocked, three were impostors — every sampled “Amazonbot” was a rented cloud VM, every “Googlebot” a bulletproof host. The shield was right; our first read was not. Real error rate was one operator, not four.
AIPM exists to give a small business the same audit a large company buys from consultants. Its scoring function did not treat them the same.
ground_truth so any score can be reproduced or disputed.What it cost us: our own accuracy fell from 18 to 15 once we stopped handing ourselves the answers, against 59–79 for the vendors we compared against. An earlier reading of “rank 4 of 200, ahead of CrowdStrike” came off scores that were not like-for-like and does not survive.
2.0.0 scores are not comparable to any earlier audit and are never charted on one axis with them.
The facts we score models against still asserted a consumer count we publicly retired in May, because it counted one-off curious pulls rather than real users. Since accuracy is scored by whether a model repeats those facts, we had been marking models wrong for correctly refusing to repeat our own inflated number — and rewarding any model credulous enough to parrot it.
That is a gullibility metric, not an accuracy metric. All grounding is now current and deflated. The same retired claim was being injected as structured data by our edge worker and was removed there too.
Single-sample scores. Two back-to-back audits of the same domain returned sentiment 46 and then 56. Production model APIs batch requests, and the underlying kernels are not batch-invariant, so identical input can produce different output. Every AIPM score published today is one draw from a distribution we have not yet characterised. Repeat-sampling with reported intervals is the next piece of work, and until it lands, treat close rankings as ties.
Behavioural lead time is not yet answerable. Our edge blocking began in June while external feed ingest runs from February, so a naive comparison measures the boundary between two eras rather than detection speed. We are not publishing a lead figure until the windows match.
We cap confidence at 95%. Something in here is wrong in a way we have not found yet. That has been true every time we have gone looking, including today, eight times.