[2026-09-10 20:36:02] verify-lock ERROR: R2 PUT db/daily/lock-verify-20260910-123602.txt returned HTTP 400 — InvalidArgument
[2026-09-10 21:07:01] DIAGNOSE: credential file shape (no secrets are printed).
[2026-09-10 21:07:01]   line 1  Cloudflare account id  32 chars, hex          looks right
[2026-09-10 21:07:01]   line 2  R2 Access Key ID       53 chars, alnum/-_.    <<< UNEXPECTED SHAPE (expected 32 hex characters)
[2026-09-10 21:07:01]   line 3  Secret Access Key      64 chars, hex          looks right
[2026-09-10 21:07:01]   line 4  bucket name            14 chars, alnum/-_.    looks right
[2026-09-10 21:07:01]   NOTE: R2 shows THREE strings when a token is created — "Token value",
[2026-09-10 21:07:01]   "Access Key ID" and "Secret Access Key". Line 2 must be the Access Key ID
[2026-09-10 21:07:01]   (32 hex), NOT the token value. Line 1 is the account id from the R2 page.
[2026-09-10 21:07:01]   PUT UNLOCKED prefix          -> HTTP 400  :: InvalidArgument: Credential access key has length 53, should be 32
[2026-09-10 21:07:01]   PUT LOCKED prefix db/daily/  -> HTTP 400  :: InvalidArgument: Credential access key has length 53, should be 32
[2026-09-10 21:07:01] DIAGNOSE complete. Both failing the same way = credential/client fault, not the lock.
[2026-09-10 21:08:01] DIAGNOSE: credential file shape (no secrets are printed).
[2026-09-10 21:08:01]   line 1  Cloudflare account id  32 chars, hex          looks right
[2026-09-10 21:08:01]   line 2  R2 Access Key ID       53 chars, alnum/-_.    <<< UNEXPECTED SHAPE (expected 32 hex characters)
[2026-09-10 21:08:01]   line 3  Secret Access Key      64 chars, hex          looks right
[2026-09-10 21:08:01]   line 4  bucket name            14 chars, alnum/-_.    looks right
[2026-09-10 21:08:01]   NOTE: R2 shows THREE strings when a token is created — "Token value",
[2026-09-10 21:08:01]   "Access Key ID" and "Secret Access Key". Line 2 must be the Access Key ID
[2026-09-10 21:08:01]   (32 hex), NOT the token value. Line 1 is the account id from the R2 page.
[2026-09-10 21:08:01]   PUT UNLOCKED prefix          -> HTTP 400  :: InvalidArgument: Credential access key has length 53, should be 32
[2026-09-10 21:08:02]   PUT LOCKED prefix db/daily/  -> HTTP 400  :: InvalidArgument: Credential access key has length 53, should be 32
[2026-09-10 21:08:02] DIAGNOSE complete. Both failing the same way = credential/client fault, not the lock.
[2026-09-10 21:28:02] verify-lock: marker stored at db/daily/lock-verify-20260910-132801.txt.
[2026-09-10 21:28:02] verify-lock: DELETE answered HTTP 409 :: ObjectLockedByBucketPolicy: The object is locked by the bucket policy.; the marker SURVIVED.
[2026-09-10 21:28:02] verify-lock PASS: the object is still there after a delete attempt with the host's own credential (HTTP 409) — the bucket lock is enforcing.
[2026-09-10 21:50:01] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-10 21:50:02] dump: stored db/daily/kualifi-2026-09-10.sql.gz (40,650 B gzipped).
[2026-09-10 21:50:02] dump: first of the month — also stored db/monthly/kualifi-2026-09.sql.gz.
[2026-09-10 21:50:02] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-10 21:50:02] DONE: backup healthy.
[2026-09-11 03:30:05] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-11 03:30:05] dump: stored db/daily/kualifi-2026-09-11.sql.gz (40,701 B gzipped).
[2026-09-11 03:30:06] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-11 03:30:06] DONE: backup healthy.
[2026-09-12 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-12 03:30:04] dump: stored db/daily/kualifi-2026-09-12.sql.gz (40,700 B gzipped).
[2026-09-12 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-12 03:30:04] DONE: backup healthy.
[2026-09-13 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-13 03:30:03] dump: stored db/daily/kualifi-2026-09-13.sql.gz (40,698 B gzipped).
[2026-09-13 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-13 03:30:04] DONE: backup healthy.
[2026-09-14 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-14 03:30:03] dump: stored db/daily/kualifi-2026-09-14.sql.gz (40,700 B gzipped).
[2026-09-14 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-14 03:30:03] DONE: backup healthy.
[2026-09-15 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-15 03:30:03] dump: stored db/daily/kualifi-2026-09-15.sql.gz (40,700 B gzipped).
[2026-09-15 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-15 03:30:03] DONE: backup healthy.
[2026-09-16 03:30:03] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-16 03:30:04] dump: stored db/daily/kualifi-2026-09-16.sql.gz (40,700 B gzipped).
[2026-09-16 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-16 03:30:04] DONE: backup healthy.
[2026-09-17 03:30:03] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-17 03:30:04] dump: stored db/daily/kualifi-2026-09-17.sql.gz (40,700 B gzipped).
[2026-09-17 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-17 03:30:04] DONE: backup healthy.
[2026-09-18 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-18 03:30:04] dump: stored db/daily/kualifi-2026-09-18.sql.gz (40,700 B gzipped).
[2026-09-18 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-18 03:30:04] DONE: backup healthy.
[2026-09-19 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-19 03:30:02] dump: stored db/daily/kualifi-2026-09-19.sql.gz (40,700 B gzipped).
[2026-09-19 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-19 03:30:03] DONE: backup healthy.
[2026-09-20 03:30:03] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-20 03:30:04] dump: stored db/daily/kualifi-2026-09-20.sql.gz (40,700 B gzipped).
[2026-09-20 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-20 03:30:04] DONE: backup healthy.
[2026-09-21 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-21 03:30:03] dump: stored db/daily/kualifi-2026-09-21.sql.gz (40,701 B gzipped).
[2026-09-21 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-21 03:30:03] DONE: backup healthy.
[2026-09-22 03:30:03] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-22 03:30:04] dump: stored db/daily/kualifi-2026-09-22.sql.gz (40,701 B gzipped).
[2026-09-22 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-22 03:30:04] DONE: backup healthy.
[2026-09-23 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-23 03:30:03] dump: stored db/daily/kualifi-2026-09-23.sql.gz (40,701 B gzipped).
[2026-09-23 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-23 03:30:03] DONE: backup healthy.
[2026-09-24 03:30:03] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-24 03:30:04] dump: stored db/daily/kualifi-2026-09-24.sql.gz (40,701 B gzipped).
[2026-09-24 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-24 03:30:04] DONE: backup healthy.
[2026-09-25 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-25 03:30:03] dump: stored db/daily/kualifi-2026-09-25.sql.gz (40,701 B gzipped).
[2026-09-25 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-25 03:30:03] DONE: backup healthy.
[2026-09-26 03:30:03] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-26 03:30:04] dump: stored db/daily/kualifi-2026-09-26.sql.gz (40,701 B gzipped).
[2026-09-26 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-26 03:30:04] DONE: backup healthy.
[2026-09-27 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-27 03:30:03] dump: stored db/daily/kualifi-2026-09-27.sql.gz (40,697 B gzipped).
[2026-09-27 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-27 03:30:03] DONE: backup healthy.
[2026-09-28 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-28 03:30:03] dump: stored db/daily/kualifi-2026-09-28.sql.gz (40,697 B gzipped).
[2026-09-28 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-28 03:30:03] DONE: backup healthy.
[2026-09-29 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-29 03:30:03] dump: stored db/daily/kualifi-2026-09-29.sql.gz (40,697 B gzipped).
[2026-09-29 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-29 03:30:03] DONE: backup healthy.
[2026-09-30 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-09-30 03:30:02] dump: stored db/daily/kualifi-2026-09-30.sql.gz (40,697 B gzipped).
[2026-09-30 03:30:02] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-09-30 03:30:02] DONE: backup healthy.
[2026-10-01 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-01 03:30:03] dump: stored db/daily/kualifi-2026-10-01.sql.gz (40,697 B gzipped).
[2026-10-01 03:30:03] dump: first of the month — also stored db/monthly/kualifi-2026-10.sql.gz.
[2026-10-01 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-01 03:30:03] DONE: backup healthy.
[2026-10-02 03:30:05] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-02 03:30:06] dump: stored db/daily/kualifi-2026-10-02.sql.gz (40,697 B gzipped).
[2026-10-02 03:30:07] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-02 03:30:07] DONE: backup healthy.
[2026-10-03 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-03 03:30:02] dump: stored db/daily/kualifi-2026-10-03.sql.gz (40,698 B gzipped).
[2026-10-03 03:30:02] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-03 03:30:02] DONE: backup healthy.
[2026-10-04 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-04 03:30:04] dump: stored db/daily/kualifi-2026-10-04.sql.gz (40,698 B gzipped).
[2026-10-04 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-04 03:30:04] DONE: backup healthy.
[2026-10-05 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-05 03:30:02] dump: stored db/daily/kualifi-2026-10-05.sql.gz (40,698 B gzipped).
[2026-10-05 03:30:02] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-05 03:30:02] DONE: backup healthy.
[2026-10-06 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-06 03:30:03] dump: stored db/daily/kualifi-2026-10-06.sql.gz (40,698 B gzipped).
[2026-10-06 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-06 03:30:03] DONE: backup healthy.
[2026-10-07 03:30:03] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-07 03:30:04] dump: stored db/daily/kualifi-2026-10-07.sql.gz (40,698 B gzipped).
[2026-10-07 03:30:04] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-07 03:30:04] DONE: backup healthy.
[2026-10-08 03:30:03] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-08 03:30:03] dump: stored db/daily/kualifi-2026-10-08.sql.gz (40,698 B gzipped).
[2026-10-08 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-08 03:30:03] DONE: backup healthy.
[2026-10-09 03:30:05] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-09 03:30:07] dump: stored db/daily/kualifi-2026-10-09.sql.gz (40,697 B gzipped).
[2026-10-09 03:30:07] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-09 03:30:07] DONE: backup healthy.
[2026-10-10 03:30:03] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-10 03:30:03] dump: stored db/daily/kualifi-2026-10-10.sql.gz (40,697 B gzipped).
[2026-10-10 03:30:03] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-10 03:30:03] DONE: backup healthy.
[2026-10-11 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-11 03:30:06] dump: stored db/daily/kualifi-2026-10-11.sql.gz (40,699 B gzipped).
[2026-10-11 03:30:06] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-11 03:30:06] DONE: backup healthy.
[2026-10-12 03:30:02] START: off-host backup to R2 bucket `kualifi-backup`.
[2026-10-12 03:30:08] dump: stored db/daily/kualifi-2026-10-12.sql.gz (40,698 B gzipped).
[2026-10-12 03:30:08] evidence: 0 row(s), 0 distinct object(s) confirmed in R2 (0 uploaded now), 0 missing locally, 0 failed, 0 without sha256.
[2026-10-12 03:30:08] DONE: backup healthy.