# Troubleshooting guide

## First-response sequence

1. Record UTC time, release ID, affected airline/user/module, request ID, and exact safe error text.
2. Run `php cli health:check` and `php cli migrate:status` as the service account.
3. Check web server/PHP-FPM status and restricted application logs. Do not paste secrets, tokens, cookies, full document content, or `.env` values into tickets.
4. Confirm the issue in a non-production environment with sanitized data.
5. Preserve audit logs and failed-job records before corrective action.

## Common conditions

| Symptom | Checks | Safe response |
|---|---|---|
| 500 page | PHP-FPM/web logs, writable storage, PHP extensions, migration status | Restore permissions/config; roll back code if release-specific. Keep debug off. |
| 419 response | Session lifetime, cookie domain/HTTPS, CSRF token, stale tab | Reauthenticate and resubmit; never disable CSRF. |
| 403 response | User status, role/base assignment, service permission, API scope/IP | Correct least-privilege assignment; do not bypass middleware. |
| Login throttled | Login attempts, source IP, account state, clock | Wait for decay, verify identity, inspect attack indicators. |
| Database unavailable | MySQL service/TLS, runtime grant, connection count, disk, DNS | Fail over or restore service; never broaden to global grants. |
| Pending migration | Release ID, backup, migration review | Enter change window, back up, apply with migrator identity. |
| Upload fails | size, MIME, PHP limits, writable private storage, free disk | Correct limits/permissions; do not move uploads into public. |
| Aircraft cannot assign | status, critical/AOG defect, mandatory due task, return-to-service, insurance/airworthiness | Resolve and approve the actual blocker; do not edit database flags manually. |
| Expense cannot edit | approval/payment state | Use reversal/adjustment workflow; preserve approved record. |
| API 401/403/409/429 | token expiry/revocation, scope/IP, idempotency key, rate | Rotate/re-scope safely; retain idempotency and rate controls. |
| Webhook failing | HTTPS DNS, destination response, signature agreement, attempts | Correct endpoint, then authorized replay; do not expose secret in logs. |
| Cron warning | heartbeat file age, crontab owner/path/PHP binary | Restore scheduled heartbeat and alerting. |
| Backup warning | job logs, checksum, remote target, marker age | Run and verify a full matched backup before clearing alert. |

## Escalation and rollback

Escalate immediately for suspected cross-airline exposure, unauthorized approval/payment/release, missing audit history, private-file disclosure, credential compromise, database corruption, or aircraft eligibility bypass. Stop affected writes, preserve evidence, revoke compromised access, and invoke incident response.

Rollback code by switching the immutable `current` link only when schema compatibility is confirmed. Restore a matched pre-migration database/upload snapshot when rollback migrations are unsafe. Validate tenant isolation, critical workflows, counts, attachments, and health before reopening traffic.