Skip to content

Investigation: Merchant Terminal READY but Attestation Fails

Date: 2026-05-21
Symptom: Settings show Worldline SDK credentials: READY; card charge fails at attestation (Soft Space refreshToken()). Sagio API env vars and /merchant-app/fasspay-config are believed correct.
App: io.sagio.merchant (apps/sagio)
Audience: Engineers continuing this work or onboarding Soft Space / Play signing

Operational runbooks (what to do next):

1. Problem statement

Chris Kwon reported:

  • Terminal UI and connecting animation work.
  • Attestation fails when creating / proceeding with a card payment.
  • Question: are any configs needed beyond SAGIO API credentials?

Follow-up from SAGIO: env vars are set and fetched; symptom is exactly Settings READY + attestation fails.

A Cloud HSM export was then provided:

  • File: sagio-production-ring-sagio-app-signer-1-CAVIUM_V2_COMPRESSED-attestation.dat
  • KMS: projects/sagio-484122/locations/australia-southeast1/keyRings/sagio-production-ring/cryptoKeys/sagio-app-signer/cryptoKeyVersions/1
  • Public key PEM (RSA 2048)

Chris later shared the Soft Space exchange archive on his machine (~/wl): SAGIO_id_cert, sagio_app_final.cert, sagio_application_id.txt, sagio_pk.pem, softspace-attestation.asc. See softspace-exchange-archive.md.

Soft Space email (Liew Zhi Ying, UAT onboarding) — sent to Deb after Chris shared the cert file:

  • Confirmed Chris’s cert file was verified
  • Confirmed io.sagio.merchant whitelisted in UAT
  • Issued integrator SDK credentials (accessKey + secretKey) for UAT

Store these only in apps/api/.env.local and Cloudflare Worker secrets — never in the APK or git. The accessKey in the email matched our .env.local; the secretKey did not until 2026-05-22 (updated + redeployed to sagio-api-prod and sagio-api-dev via wrangler on nzsagio@gmail.com).

Goal: determine what that material is, whether it fixes attestation, and what Soft Space / Play actually require.

2. How we narrowed the failure (methodology)

2.1 Map the card charge pipeline in code

We traced apps/sagio from UI to network:

Step Component What it proves
Auth SettingsRepository → POST /merchant-app/auth SAGIO_API_KEY / SAGIO_API_SECRET work
Config fetch GET /merchant-app/fasspay-config Fasstap env on API worker (FASSTAP_*)
Settings READY MerchantConfig.isValid() Only 4 fields: accessKey, secretKey, uniqueId, developerId
Payment request PaymentRequestRepository.createPaymentRequest() Sagio backend accepts WORLDLINE rail
SDK init FasstapManager.initialize() Soft Space SDK config applied
Attestation FasstapManager.ensureTokenRefreshed() → refreshToken() Failure reported here
NFC startTransaction() Not reached if refresh fails

Relevant files:

  • apps/sagio/app/src/main/java/io/sagio/merchant/viewmodel/PaymentViewModel.kt (steps 1–5 in processCardPayment)
  • apps/sagio/app/src/main/java/io/sagio/merchant/sdk/FasstapManager.kt
  • apps/sagio/app/src/main/java/io/sagio/merchant/data/MerchantConfig.kt (isValid())
  • apps/sagio/app/src/main/java/io/sagio/merchant/ui/screens/SettingsScreen.kt (READY = uiState.isValid)
  • apps/api/src/services/merchantAppService.ts (getFasstapConfigForMerchant)
  • apps/api/src/config/index.ts (buildFasspayConfig() — UAT hosts, cert pin, Play project 757874674469)

Conclusion: If READY and payment request creation succeed, the Sagio API + Fasstap credential fetch path is working. Attestation is a separate step inside the Soft Space SDK talking to mpos-uat.fasspay.com:9001 with Play Integrity tied to the APK signing certificate.

2.2 What “READY” does and does not mean

From MerchantConfig.kt:

fun isValid(): Boolean {
    return accessKey.isNotBlank() &&
            secretKey.isNotBlank() &&
            uniqueId.isNotBlank() &&
            developerId.isNotBlank()
}

READY means: server returned the four Soft Space integrator/portal fields and the app persisted them.

READY does not mean:

  • SSMPOSSDK.init() succeeded (check logcat: SDK initialized: version=3.6.5.0)
  • refreshToken() / attestation succeeded
  • Play Integrity passed
  • Soft Space whitelisted this device’s signing cert

This explains the reported symptom precisely: env vars OK + READY + attestation fail.

2.3 Ruled out vs still in play (verified 2026-05-22)

Cloudflare account: nzsagio@gmail.com (29d5f87ef1dc233655dc802fc1c55317) via wrangler whoami.
Workers: sagio-api-prod (default env ""), sagio-api-dev (--env=sagio-api-dev).

Secrets refreshed from apps/api/.env.local:

cd apps/api
bun run deploy:fasstap-secrets -- .env.local          # prod (sagio-api-prod)
bun run deploy:fasstap-secrets -- .env.local sagio-api-dev

Live prod smoke test (Chris Kwon merchant terminal key from apps/sagio/local.properties):

Endpoint Result
POST /merchant-app/auth success: true, merchant Chris Kwon
GET /merchant-app/fasspay-config success: true, environment: UAT, all four required fields set, attestationHost + googlePlayProjectNumber: 757874674469, keyloadingCACert present (1020 chars)

Ruled out (with evidence)

Item Why
Missing FASSTAP_* on prod worker All six secrets on sagio-api-prod; live fasspay-config returns full payload
Missing FASSTAP_* on dev worker All six secrets on sagio-api-dev after 2026-05-22 deploy
Wrong / missing Sagio terminal API key Prod auth succeeds for Chris’s minted key
buildFasspayConfig() empty on Workers Historical bug (import-time config); current worker returns hosts, pins, CA, Play project number
Sagio API layer alone blocking attestation Attestation runs in SDK after READY; API path verified end-to-end
Wrong FASSTAP_ACCESS_KEY vs Soft Space email Matched Soft Space UAT accessKey in .env.local
Wrong FASSTAP_SECRET_KEY vs Soft Space email Was mismatched — fixed 2026-05-22; redeployed to prod + dev wrangler

Fixed 2026-05-22 (was still in play)

Item Action
Stale FASSTAP_SECRET_KEY on worker Replaced with value from Soft Space email; wrangler secret put on prod + dev

Still in play (likely cause of READY + attestation fail)

Item What to do
APK signing cert ≠ Soft Space whitelist Compare apksigner on Chris’s APK vs sagio_app_final.cert in ~/wl — see softspace-exchange-archive.md
Debug / Diawi APK vs Play/HSM cert whitelisted Whitelist the cert for the installed build channel (debug SHA 8B:22:30:7E:... vs Play app signing cert)
Soft Space UAT whitelist not active Re-send fingerprint; confirm with Soft Space
Wrong MUID/SSO for portal user Rare if READY; re-check FASSTAP_UNIQUE_ID / FASSTAP_DEVELOPER_ID vs Soft Space portal
Device: emulator, root, no Play services Use physical device, updated Play services
Network to mpos-uat.fasspay.com:9001 Test from device network

Bottom line: Sagio Cloudflare + API path is verified working. Remaining work is Soft Space Play Integrity cert whitelist matched to Chris’s actual APK, not more Wrangler secrets.

2.4 Soft Space / Play Integrity requirement (from specs + SDK)

From docs/specs/tap2pay/003-fasstap-configuration.md and packages/worldline-sdk/src/constants/errorCodes.ts:

  • Play Integrity uses Soft Space GCP project 757874674469 (not SAGIO’s project).
  • Package must be whitelisted: io.sagio.merchant.
  • Signing certificate SHA-256 for the installed APK must also be whitelisted.
  • SDK error codes include ATTESTATION_FAILED (8002), PLAY_INTEGRITY_FAILED (8006).

From docs/handoffs/merchant-terminal-launch.md troubleshooting row:

SDK init fails, Settings READY → Play Integrity / cert → Send SHA-256 to Soft Space

3. Investigating the Cloud HSM artifacts

3.1 Attestation file (*-CAVIUM_V2_COMPRESSED-attestation.dat)

Location checked: ~/Downloads/sagio-production-ring-sagio-app-signer-1-CAVIUM_V2_COMPRESSED-attestation.dat

Command used:

file sagio-production-ring-sagio-app-signer-1-CAVIUM_V2_COMPRESSED-attestation.dat
# → gzip compressed data, 1002 bytes → 2528 bytes decompressed

Decompressed with Python gzip.decompress():

  • Binary TLV-style vendor payload (Cavium/Cloud HSM key attestation, format CAVIUM_V2_COMPRESSED).
  • Contains label app_key and a 128-hex key ID (two concatenated SHA-256 digests per Google docs).
  • No PEM BEGIN CERTIFICATE block; no DER certificate (30 82) in the blob.

Key ID verified against KMS resource name (2026-05-22):

RESOURCE_NAME="projects/sagio-484122/locations/australia-southeast1/keyRings/sagio-production-ring/cryptoKeys/sagio-app-signer/cryptoKeyVersions/1"
echo -n "$RESOURCE_NAME" | openssl dgst -sha256 -hex
# SHA2-256(stdin)= f679bb1bba71cf6b2a16799b4121e5554ca8cbbfe6c41a6b838399803524ed72

Second half of the attestation key-id field matches exactly → this .dat is for sagio-app-signer cryptoKeyVersions/1.

Reference: Google Cloud KMS — Verifying attestations

Conclusion: This file proves the key material resides in Cloud HSM. It is not the Android signing certificate and cannot be pasted into Soft Space as a Play Integrity allowlist fingerprint. For Soft Space, use Chris’s sagio_app_final.cert instead (see §3.4).

3.4 Chris’s Soft Space exchange archive (~/wl)

Chris reported these are the exact files used in the Soft Space exchange:

File Role
sagio_application_id.txt Package allowlist (io.sagio.merchant)
sagio_pk.pem Public key paired with app signing cert
sagio_app_final.cert X.509 cert whose SHA-256 Soft Space whitelists
SAGIO_id_cert Integrator / identity cert (onboarding)
softspace-attestation.asc Soft Space signed record (PGP/armored)

Not present on SAGIO machine — only on Chris’s ~/wl. Full mapping and verification commands: softspace-exchange-archive.md.

Investigation implication: If Soft Space whitelisted sagio_app_final.cert but Chris installs a debug or upload-key APK, Play Integrity still fails with READY. The decisive test is apksigner on Chris’s APK vs openssl x509 -fingerprint -sha256 on sagio_app_final.cert.

3.2 Public key PEM (from Cloud HSM / console)

Stored in repo: apps/sagio/signing/sagio-app-signer.public.pem

Analysis:

openssl pkey -pubin -in sagio-app-signer.public.pem -outform DER | openssl dgst -sha256
# SPKI SHA-256: 649079b06f9210a42e482a638586d292247fd471675f08921d3b90fcaac0611c
# Android format: 64:90:79:B0:6F:92:10:A4:2E:48:2A:63:85:86:D2:92:24:7F:D4:71:67:5F:08:92:1D:3B:90:FC:AA:C0:61:1C
  • Algorithm: RSA 2048 (modulus bit length 2048 via Python cryptography).

Important: Android / Play Integrity / Soft Space whitelist use the X.509 certificate SHA-256 (DER of the whole cert), not the SPKI hash above. Those differ whenever the cert has issuer/subject/validity wrapping the same public key.

Authoritative cert fingerprint must come from one of:

gcloud kms keys versions get-certificate \
  --project=sagio-484122 \
  --location=australia-southeast1 \
  --keyring=sagio-production-ring \
  --key=sagio-app-signer \
  --version=1 \
  --certificate-output-file=/tmp/sagio-app-signer.crt

openssl x509 -in /tmp/sagio-app-signer.crt -noout -fingerprint -sha256

Or Play Console → App signing key certificate SHA-256 (must match this public key).

gcloud was not available in the investigation environment; SAGIO must run the command locally or read Play Console.

3.3 Unrelated private key in Downloads

A file key_e6cde843-cbdd-44ed-b0c5-875a2117ea64.pem was present locally. We compared RSA moduli:

# HSM public key modulus == Downloads private key modulus?
False

Conclusion: That PEM is not the sagio-app-signer HSM key. Do not assume it is the Play upload key without checking against CI SAGIO_KEYSTORE / Play Console. Never commit private keys to the repo.

4. Three Android signing keys (root cause of confusion)

Investigation of apps/sagio/SIGNING.md and .github/workflows/sagio-android-build.yml shows three distinct keys:

flowchart LR
  subgraph dev [Local debug]
    D[debug.keystore] --> APKd[debug APK]
  end
  subgraph ci [CI release]
    U[SAGIO_KEYSTORE secrets] --> APKr[release APK / Diawi]
  end
  subgraph play [Google Play]
    U2[Upload key] --> AAB[AAB upload]
    AAB --> GP[Play re-sign]
    HSM[sagio-app-signer Cloud HSM] --> GP
    GP --> APKp[Play-installed APK]
  end
  APKd --> PI[Play Integrity sees debug cert]
  APKr --> PI2[Play Integrity sees upload cert]
  APKp --> PI3[Play Integrity sees app signing cert]
Key Material location Signs which APK Soft Space must whitelist for that APK
Debug ~/.android/debug.keystore ./gradlew assembleDebug Debug SHA-256 (see below)
Play upload GitHub SAGIO_KEYSTORE_* → release-keystore.jks CI assembleRelease, Diawi links Upload keystore SHA-256
Play app signing Cloud HSM sagio-app-signer APKs installed from Play after re-sign gcloud get-certificate or Play Console

Critical insight: Sending only the HSM / Play App Signing fingerprint to Soft Space does not fix attestation on a debug or Diawi release APK. Chris’s installed build type determines which fingerprint Soft Space needs.

4.1 Fingerprints computed during investigation

Source SHA-256 (colon-separated) Notes
Debug keystore (this machine) 8B:22:30:7E:BB:87:8E:A6:61:3B:51:44:AF:31:E7:EA:2A:99:49:AF:62:25:2C:E9:D5:02:8C:06:EB:FD:23:FD Standard Android debug; use for local UAT APKs
HSM public key SPKI only 64:90:79:B0:6F:92:10:A4:2E:48:2A:63:85:86:D2:92:24:7F:D4:71:67:5F:08:92:1D:3B:90:FC:AA:C0:61:1C Reference only — not a substitute for cert fingerprint
HSM X.509 cert Pending Run gcloud kms keys versions get-certificate on SAGIO workstation
Upload keystore Pending keytool -list -v on CI release-keystore.jks or apksigner on Chris’s APK

Best practice: apksigner verify --print-certs <exact.apk> on the APK Chris installed — removes all ambiguity.

Helper script added: apps/sagio/scripts/print-softspace-fingerprints.sh

5. How Sagio API config relates (layer 1 vs layer 2)

Layer 1 — Sagio API (working when READY)

Item Where
Terminal auth POST /merchant-app/auth
Fasstap config GET /merchant-app/fasspay-config
Worker secrets FASSTAP_ACCESS_KEY, SECRET_KEY, UNIQUE_ID, DEVELOPER_ID, ENVIRONMENT
Baked UAT defaults Attestation host, cert pinning, keyloading CA, 757874674469

Deploy: cd apps/api && bun run deploy:fasstap-secrets -- .env.local

Layer 2 — Soft Space SDK on device (attestation)

Item Where
SDK init FasstapManager.initialize()
Attestation refreshToken() → https://mpos-uat.fasspay.com:9001/v2
Play Integrity APK signing cert + package io.sagio.merchant
Whitelist owner Soft Space (Worldline NZ)

Chris’s case: Layer 1 OK (READY). Layer 2 blocked → signing cert whitelist (most likely) or MUID/network/device.

6. Final conclusions

# Conclusion
1 SAGIO API creds alone are insufficient for card tap; Fasstap worker secrets + terminal key are also required for READY.
2 READY + attestation fail is consistent with correct Sagio/Fasstap fetch and failed Play Integrity / Soft Space device trust.
3 The HSM .dat attestation file is not used for Soft Space whitelist.
4 The HSM public key is the Play App Signing key; cert SHA-256 must be exported via gcloud or Play Console.
5 Three signing keys exist; whitelisting the wrong one (e.g. HSM only while testing debug APK) preserves the symptom.
6 Fastest UAT unblock if Chris uses debug APK: whitelist debug SHA-256 8B:22:30:7E:... (see softspace-play-integrity-whitelist.md).
7 Production path: whitelist Play App Signing cert from HSM + distribute via Play internal testing so installed cert matches.
  1. Confirm Chris’s APK source — debug local build, Diawi release, or Play internal test.
  2. Extract fingerprint from that APK — ./apps/sagio/scripts/print-softspace-fingerprints.sh /path/to.apk or adb pull + apksigner.
  3. Email Soft Space — package io.sagio.merchant, Play project 757874674469, that SHA-256, MUID/SSO from portal. Template in softspace-play-integrity-whitelist.md.
  4. For Play/HSM track — run gcloud kms keys versions get-certificate and match Play Console app signing cert to apps/sagio/signing/sagio-app-signer.public.pem.
  5. After whitelist — reinstall APK, Settings → Refresh, retry card charge; collect logcat if still failing.

8. Artifacts added to the repo (this investigation)

Path Purpose
docs/handoffs/merchant-terminal-attestation-investigation.md This document — reasoning and evidence
docs/handoffs/softspace-play-integrity-whitelist.md Operational whitelist steps
docs/handoffs/merchant-terminal-card-attestation.md Partner FAQ / config layers
apps/sagio/signing/sagio-app-signer.public.pem HSM app signing public key (safe to commit)
apps/sagio/scripts/print-softspace-fingerprints.sh Print debug / SPKI / APK cert fingerprints
apps/sagio/SIGNING.md Updated with HSM KMS path and script link

PR: docs/merchant-terminal-card-attestation branch → GitHub PR #223 (docs only; merge to master for team visibility).

9. Open items (need SAGIO input)

  • Deploy / refresh FASSTAP_* on Cloudflare (nzsagio@gmail.com) — prod + dev, 2026-05-22
  • Align FASSTAP_SECRET_KEY with Soft Space UAT email (Liew Zhi Ying) and redeploy
  • Prod smoke: auth + fasspay-config for Chris Kwon merchant
  • Soft Space confirmed cert verified + app whitelisted in UAT (email to Deb)
  • Chris zips ~/wl and shares with SAGIO (five Soft Space exchange files).
  • Run checklist in softspace-exchange-archive.md — especially APK cert vs sagio_app_final.cert.
  • Confirm how Chris installed the app (debug / Diawi / Play).
  • Run gcloud kms keys versions get-certificate and confirm cert matches sagio_app_final.cert / sagio_pk.pem.
  • Confirm Soft Space UAT whitelist is active for the cert on Chris’s actual APK.
  • Optional: paste logcat statusMessage from failed refreshToken() for Soft Space ticket.

10. References

  • Code: PaymentViewModel.kt, FasstapManager.kt, MerchantConfig.kt, merchantAppService.ts, buildFasspayConfig()
  • Specs: docs/specs/tap2pay/003-fasstap-configuration.md, 004-fasstap-visual-architecture.md
  • Google: KMS attest-key, Android client auth / cert fingerprints
  • Launch handoff: docs/handoffs/merchant-terminal-launch.md