Verifying a sample offline: the selection algorithm

Every sample Axis Assure draws can be drawn again by anyone, without Axis, from three things: the population document, the sample's seed and the algorithm on this page. A SHA-256 tool, an HMAC-SHA256 and a sort are enough. This page is the complete specification of the algorithm Axis records as hmac-sha256-rank@1. Each sample stores the name of the algorithm that drew it, so a later algorithm never changes how an earlier sample reads.

TL;DR — Hash the population document you were given and compare it with the sample's record. Rank every key by HMAC-SHA256 with the seed as the key. A ranked sample is the lowest n ranks; a systematic sample walks the document at a fixed interval from a start the seed decides. If you get the same keys in the same order, the sample is the one the population and the seed give.

Where the inputs come from. The engagement's workpapers (a signed ZIP built from the packages hub) carry every input on one page of this specification: samples.csv lists each sample's population, its two hashes, the method, size, algorithm and seed; samples/<key>.csv lists the items drawn, in order, with any replacement marked; and populations/<key>.jsonl is each population document, byte for byte — the manifest lists its SHA-256, which is the population's itemsHash. The assessment results, in OSCAL, carry the same hashes, method, size and seed for each sample.

As an external auditor with a seat, the engagement portal's Samples tab gives the same inputs: Record saves a sample's record as <key>.record.json — exactly the fields the reference script in section 7 reads (method, seed, n, keys, count, keysHash, itemsHash) — and Population file, when your seat lets files leave, saves the population document. A population whose details name people of the workspace comes as its keys alone, one {"key": …} line each: its SHA-256 differs from itemsHash, its keysHash is the record's, and section 5 verifies it with a note that the attributes differ.

Checking it in the product first. On the engagement's Sampling tab, each sample shows its method, its size, its seed (abridged, with whether Axis or the assessor supplied it) and the population it was drawn from, whose keys hash is in the population list. Verify draws the sample again on the server from the stored population document, by the sample's own algorithm, and says whether the same keys came back in the same order — and, if not, why: the stored population is not the one captured, the draw cannot be repeated from its parameters, or the keys differ. That is the same computation this page specifies; the offline check below is the one that does not rely on Axis.

What you need

The seed is 32 random bytes from the server, or a value the assessor supplied — for example the output of a public randomness beacon. A beacon value published after the population was frozen is one nobody could have chosen to favour an item.

1. The population document

The document is JSON Lines in UTF-8: one item per line, each line ending in a line feed (0x0A), the last one included. The lines are in the population's declared order, which is the order a systematic sample walks. Each line is one JSON object with a key member (the item's identifier in its source: a record id, a change number, a CSV cell) and, usually, an attrs member (the few attributes a tester needs, such as a date or a status).

Every key is a non-empty string, and no key appears twice. A document that breaks either rule is not a population.

Axis writes each line in canonical form — object members sorted by name, no whitespace, strings and numbers as ECMAScript's JSON.stringify writes them — but you never need to re-create a line. You hash the file as you received it, and you read the keys with any JSON parser.

Its two hashes

• itemsHash is the SHA-256 of the document's bytes, as stored. It says the document is the one captured.
• keysHash is the SHA-256 of the keys alone: for each key in document order, the key written as a JSON string, followed by a line feed, all concatenated and encoded as UTF-8. It says the keys are the ones the sample was drawn from.
• count is the number of lines.

A selection depends on the keys and their order only. That is why there are two hashes: if a copy of the population has an attribute redacted — a person's name, for example — its itemsHash changes and its keysHash does not, and the sample still verifies against it.

The JSON string of a key is the key between double quotes, with exactly these characters escaped:

Every other character — /, U+007F, U+2028, U+2029, any letter or symbol in any script — is written as itself, in UTF-8. This is what JSON.stringify does; Python's json.dumps(key, ensure_ascii=False) writes the same for every key that is well-formed Unicode text.

2. Ranks

An item's rank is the HMAC-SHA256 of its key, written as 64 lower-case hex digits:

• the HMAC key is the seed's bytes (the hex decoded), not the hex text;
• the message is the item's key in UTF-8. A key holding an unpaired surrogate code unit is encoded with U+FFFD (EF BF BD) in its place.

Rank order sorts the keys by rank, lowest first. Comparing the hex strings is the same as comparing the ranks as 256-bit unsigned integers. Two equal ranks — which would need an HMAC-SHA256 collision, and have never been seen — are ordered by key, comparing UTF-16 code units.

With OpenSSL or LibreSSL, one rank is:

printf '%s' 'CHG-1023' | openssl dgst -sha256 -mac HMAC -macopt hexkey:<seed>

3. Selection

Ranked

The sample is the first n keys in rank order, in that order. The document's order does not matter.

Systematic

Over the document's order, with N items and a sample of n (never more than N):

1. k = ⌊N / n⌋, the interval (at least 1).
2. r = the rank of the five-byte message start (73 74 61 72 74), read as a 256-bit unsigned integer, modulo k.
3. For i from 0 to n − 1, the position pᵢ = ⌊(r·n + i·N) / n⌋, counted from 0.
4. The sample is the keys at p₀ … pₙ₋₁, in that order.

The positions are always distinct and below N. Use whole-number arithmetic with enough precision for a 256-bit r.

Judgmental

A person picked the keys and wrote down why. There is no seed, so there is nothing to draw again: a verifier checks that there are n picks, each a key of the population, none repeated, and reads the rationale.

Replacements

A sampled item that cannot be tested (it is void, or does not apply) is replaced, and the reason is recorded. An item already tested — a pass or an exception — is never replaced.

• In a ranked or systematic sample, the replacement is the first key in rank order (the sample's seed, the whole population) that is not already in the sample. "Already in the sample" counts every key drawn, every key replaced and every earlier replacement. A systematic sample's replacements follow rank order too, not its interval.
• In a judgmental sample, the person picks the replacement: a key of the population not already in the sample.

4. Sample size

The sample's record keeps n and how it was decided. To check n:

• Attribute sampling. n is the smallest size, from e + 1 up, for which the probability of finding at most e deviations is no more than 1 − confidence when the population deviates at the tolerable rate. e is the number of deviations the assessor accepts in the sample. The record names the model:

  • binomial: the number of deviations found follows Binomial(n, tolerable rate);
  • hypergeometric: it follows Hypergeometric(N, D, n), with D = ⌈tolerable rate × N⌉ deviating items — the fewest at which the population reaches the tolerable rate.

  When n would exceed a known N, every item is tested (n = N). Confidence, tolerable rate, expected deviations and the model are the assessor's choices for each sample; Axis computes the probabilities in double precision. Two well-known values: binomial, 95% confidence, 5% tolerable rate, no deviations: 59; 90% and 10%: 22.

• A firm's table. The firm's own frequency table, loaded with its citation: the one row whose population range holds N gives n (never more than N). No publisher's table ships with Axis.

• Fixed: the n the assessor stated, never more than N. All: N.

5. Verifying a sample

1. Hash the document. If its count or keysHash differs from the record, the population changed: these are not the keys the sample was drawn from, and the sample does not verify. If only itemsHash differs, the attributes changed and the keys did not; note it, and go on.
2. Draw again with the record's method, seed and n (section 3). If that cannot be done — a seed that is not 32 to 512 bytes of lower-case hex, an n outside 1 to N — the sample does not verify.
3. Compare. The keys must be the same, in the same order.
4. For each replacement, check that it is the key section 3 gives.

Axis's own Verify does steps 1 to 3 on the document as stored, with the code that drew the sample. Each replacement is recorded with the key it brought in and the reason.

6. A worked example

A population of 24 closed changes was captured from a CSV file whose key column is the change number, keeping its Closed and Status columns. Its first two lines are:

{"attrs":{"Closed":"2026-03-01","Status":"Closed"},"key":"CHG-1001"}
{"attrs":{"Closed":"2026-03-02","Status":"Closed"},"key":"CHG-1002"}

…and so on to CHG-1024, closed on 2026-03-24, each line ending in a line feed. Then:

• count = 24
• itemsHash = eedfff864e557bef58f7c743b1812c32e8d31ce0bf699413fd5c2476d41ccb39
• keysHash = 54c6e2a49314d9b8f3e9442cf202354ba596362bfeb6cf38282ed6b0efcd7119

The seed here is the bytes 00 to 1f, written 000102…1e1f, so that you can type it; a real seed is random. The lowest ranks are:

Ranked, n = 5: CHG-1023, CHG-1017, CHG-1022, CHG-1011, CHG-1009. If one of them cannot be tested, it is replaced by CHG-1018; if a second one cannot, by CHG-1005.

Systematic, n = 5: k = ⌊24 / 5⌋ = 4. The rank of start is 035638b12fc5ae24eacb790edffe7af97e2612865e116f4a08ed5e6465b17367; modulo 4 only its last hex digit counts (7), so r = 3. The positions are 3 + ⌊i·24 / 5⌋ = 3, 7, 12, 17 and 22, so the sample is CHG-1004, CHG-1008, CHG-1013, CHG-1018, CHG-1023. A replacement here is the best-ranked key not in it: CHG-1017.

A hypergeometric size for this population: 90% confidence, 10% tolerable rate, no deviations. D = ⌈0.1 × 24⌉ = 3, and the probability of finding none is (24−n)(23−n)(22−n) / (24·23·22). At n = 12 it is 0.109, above 0.10; at n = 13 it is 0.082. So n = 13.

The worked example's data

Axis's own test suite reads this block and runs it against the code that captures, draws, replaces and verifies samples, and against an implementation written from this page alone, so the page and the product cannot drift apart.

{
  "algorithm": "hmac-sha256-rank@1",
  "seed": "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f",
  "population": [
    { "key": "CHG-1001", "attrs": { "Closed": "2026-03-01", "Status": "Closed" } },
    { "key": "CHG-1002", "attrs": { "Closed": "2026-03-02", "Status": "Closed" } },
    { "key": "CHG-1003", "attrs": { "Closed": "2026-03-03", "Status": "Closed" } },
    { "key": "CHG-1004", "attrs": { "Closed": "2026-03-04", "Status": "Closed" } },
    { "key": "CHG-1005", "attrs": { "Closed": "2026-03-05", "Status": "Closed" } },
    { "key": "CHG-1006", "attrs": { "Closed": "2026-03-06", "Status": "Closed" } },
    { "key": "CHG-1007", "attrs": { "Closed": "2026-03-07", "Status": "Closed" } },
    { "key": "CHG-1008", "attrs": { "Closed": "2026-03-08", "Status": "Closed" } },
    { "key": "CHG-1009", "attrs": { "Closed": "2026-03-09", "Status": "Closed" } },
    { "key": "CHG-1010", "attrs": { "Closed": "2026-03-10", "Status": "Closed" } },
    { "key": "CHG-1011", "attrs": { "Closed": "2026-03-11", "Status": "Closed" } },
    { "key": "CHG-1012", "attrs": { "Closed": "2026-03-12", "Status": "Closed" } },
    { "key": "CHG-1013", "attrs": { "Closed": "2026-03-13", "Status": "Closed" } },
    { "key": "CHG-1014", "attrs": { "Closed": "2026-03-14", "Status": "Closed" } },
    { "key": "CHG-1015", "attrs": { "Closed": "2026-03-15", "Status": "Closed" } },
    { "key": "CHG-1016", "attrs": { "Closed": "2026-03-16", "Status": "Closed" } },
    { "key": "CHG-1017", "attrs": { "Closed": "2026-03-17", "Status": "Closed" } },
    { "key": "CHG-1018", "attrs": { "Closed": "2026-03-18", "Status": "Closed" } },
    { "key": "CHG-1019", "attrs": { "Closed": "2026-03-19", "Status": "Closed" } },
    { "key": "CHG-1020", "attrs": { "Closed": "2026-03-20", "Status": "Closed" } },
    { "key": "CHG-1021", "attrs": { "Closed": "2026-03-21", "Status": "Closed" } },
    { "key": "CHG-1022", "attrs": { "Closed": "2026-03-22", "Status": "Closed" } },
    { "key": "CHG-1023", "attrs": { "Closed": "2026-03-23", "Status": "Closed" } },
    { "key": "CHG-1024", "attrs": { "Closed": "2026-03-24", "Status": "Closed" } }
  ],
  "count": 24,
  "itemsHash": "eedfff864e557bef58f7c743b1812c32e8d31ce0bf699413fd5c2476d41ccb39",
  "keysHash": "54c6e2a49314d9b8f3e9442cf202354ba596362bfeb6cf38282ed6b0efcd7119",
  "ranks": {
    "start": "035638b12fc5ae24eacb790edffe7af97e2612865e116f4a08ed5e6465b17367",
    "CHG-1023": "06573269a0b360aabba593c37b0a679166446a160c7173d49ace6f852e812e40",
    "CHG-1017": "21a4646336eb7e52d9287abebcc76625229b384047acadb364f28d31fec060cc",
    "CHG-1022": "256496c94c27c25ee4c80ce3bcd3b237ec9c23abbd75cd50aa3a57682c5d283f",
    "CHG-1011": "2cbeca3b910f2cfb1712c0f2d893d1de4bf8eb8873936e9ba913fc755f6e187c",
    "CHG-1009": "335142ef0ed8f95f55fdb593f46db7ef1884e69287a6966246e38cb904a8b8f7",
    "CHG-1018": "3c80f22f2d84a2b603e713973b1f6d9b3a0d8c6f20fe02b73bc3e615131d65c8",
    "CHG-1005": "4232cf23b39e32003d9c914749d6942a075a89aaea4b71f181c566828c888ec1"
  },
  "ranked": { "n": 5, "keys": ["CHG-1023", "CHG-1017", "CHG-1022", "CHG-1011", "CHG-1009"], "replacements": ["CHG-1018", "CHG-1005"] },
  "systematic": { "n": 5, "k": 4, "r": 3, "positions": [3, 7, 12, 17, 22], "keys": ["CHG-1004", "CHG-1008", "CHG-1013", "CHG-1018", "CHG-1023"], "replacements": ["CHG-1017"] },
  "size": { "model": "hypergeometric", "confidence": 0.9, "tolerableRate": 0.1, "expectedDeviations": 0, "populationSize": 24, "deviatingItems": 3, "n": 13 }
}

7. A reference implementation

This Python 3 script uses the standard library only. Save the record's fields as JSON (method, seed, n, keys, count, keysHash, itemsHash) and run python3 verify_sample.py population.jsonl record.json. It reproduces the worked example above. It reads keys that are well-formed Unicode text; a key holding an unpaired surrogate needs the two rules for it above.

import hashlib, hmac, json, sys

def read_population(path):
    data = open(path, "rb").read()
    text = data.decode("utf-8")
    if text and not text.endswith("\n"):
        raise ValueError("the document must end with a newline")
    keys = [json.loads(line)["key"] for line in text[:-1].split("\n")] if text else []
    if len(set(keys)) != len(keys) or not all(isinstance(k, str) and k for k in keys):
        raise ValueError("every key must be a non-empty string, none repeated")
    keys_doc = "".join(json.dumps(k, ensure_ascii=False) + "\n" for k in keys)
    return {
        "keys": keys,
        "count": len(keys),
        "itemsHash": hashlib.sha256(data).hexdigest(),
        "keysHash": hashlib.sha256(keys_doc.encode("utf-8")).hexdigest(),
    }

def rank(seed_hex, key):
    return hmac.new(bytes.fromhex(seed_hex), key.encode("utf-8"), hashlib.sha256).hexdigest()

def ranked_order(seed_hex, keys):
    return sorted(keys, key=lambda k: (rank(seed_hex, k), k.encode("utf-16-be")))

def draw(method, seed_hex, keys, n):
    N = len(keys)
    if not 1 <= n <= N:
        raise ValueError("n must be from 1 to N")
    if method == "ranked":
        return ranked_order(seed_hex, keys)[:n]
    if method == "systematic":
        r = int(rank(seed_hex, "start"), 16) % (N // n)
        return [keys[(r * n + i * N) // n] for i in range(n)]
    raise ValueError("a judgmental sample has no seed to re-derive")

def replacement(seed_hex, keys, already_in_sample):
    taken = set(already_in_sample)
    return next((k for k in ranked_order(seed_hex, keys) if k not in taken), None)

if __name__ == "__main__":
    population_path, record_path = sys.argv[1], sys.argv[2]
    record = json.load(open(record_path, encoding="utf-8"))
    pop = read_population(population_path)
    if (pop["count"], pop["keysHash"]) != (record["count"], record["keysHash"]):
        sys.exit("FAIL: these are not the keys the sample was drawn from")
    if pop["itemsHash"] != record["itemsHash"]:
        print("NOTE: the attributes changed since capture (the keys did not)")
    again = draw(record["method"], record["seed"], pop["keys"], record["n"])
    print("OK" if again == record["keys"] else "FAIL: drawing again gives " + json.dumps(again))