Incident response#

Something is wrong on a live install: pages 500, leads stopped, payments are not flowing, the queue is backed up. This page is the calm version of those moments. Read the severity table, do the first five minutes, then jump to the matching runbook. Everything here uses tools that already exist in the app — no improvisation required.

Who this is for#

• Super admins and developers handling a live incident.
• Whoever is on call when the support ticket lands.

Difficulty: Medium · Time: 20 minutes to read; runbooks are minutes each · Prerequisites: Super Admin access, shell/hosting access, and a way to reach the workspace owner.

Step 0 — Set the severity#

The first five minutes (any severity)#

1. Confirm what users see — open GET {app URL}/api/v1/health and the login page. Degraded or 503 tells you it is infrastructure; 200 tells you it is application logic.
2. Look at the log — tail storage/logs/laravel.log. The newest ERROR lines usually name the exact class or integration. Keep the lines; they are evidence.
3. Check the boring things — disk space (System Health shows total/free/used), database reachable (php artisan db:show), and whether a deployment or update happened recently (Super Admin → Settings → Updates).
4. Check the queue — php artisan queue:failed; is the worker running; is the driver sync when it should not be?
5. Write down the timeline — when it started, what you tried, what changed. You will need it for the post-incident note, and it stops you retrying the same fix.

Runbook — the app is down or 500s#

1. Put the site into maintenance mode while you work: php artisan down (this shows a clean maintenance page instead of stack traces).
2. Re-read the newest errors in storage/logs/laravel.log. If they mention config/view/route caches built under a different environment, open System Health and use Clear caches (optimize:clear), then Rebuild caches only after the app is healthy.
3. If a deployment just happened: run Run migrations on System Health (migrate --force), confirm php artisan migrate:status is clean, and if the upgrade is broken, roll back by restoring the pre-update backup (see the backups runbook below). Deployment specifics live in Deployment & server errors.
4. If complaints mention missing images/uploads: Storage link on System Health (storage:link).
5. When healthy: php artisan up, then confirm health endpoint ok and a login works before telling anyone it is fixed.

Runbook — queue and scheduler stopped#

Symptoms: notifications late, WhatsApp not sending, digests missing, queue_failures climbing, jobs in failed_jobs.

1. Restart the worker — System Health → Restart queue (queue:restart; when the driver is sync the page warns you that this does not apply — fix the driver first).
2. Start the worker properly on the server: php artisan queue:work --queue=default,webhooks,automations,integrations,notifications,exports,ai-sdr,mail All queues must be listed; a missing one silently starves that feature.
3. Read the failed jobs (php artisan queue:failed) and map the exception to a troubleshooting page before retrying. What failed once will fail again unchanged.
4. Scheduler: on shared hosting make sure the single cron.php entry point is still being called every minute (it drains the queue with a time budget, then runs the scheduler). On a VPS, confirm the schedule:run cron line. Look for schedule task failed: lines in the log.

Runbook — leads stopped arriving#

Follow the decision path in Leads not entering the CRM. The short version: connection health → page token → form allowlist → webhook logs → permissions → test lead → assignment. Check Webhook Logs before touching anything — it tells you whether Meta is reaching you at all.

Runbook — payments failing or billing drift#

1. Super Admin → Billing → Commercial → Payments (/super-admin/billing/commercial-payments): what is the provider actually saying?
2. Read any open reconciliation alert on the billing overview — a BillingReconciliationAlert means the provider status and the CRM disagree; resolve the alert deliberately rather than assuming.
3. Check the provider configuration: Commercial Providers (/super-admin/billing/commercial-providers) — environment (sandbox vs production), keys, and the webhook secret. A sandbox/production mix-up is the classic cause.
4. Confirm webhooks are arriving: the endpoints are POST /api/webhooks/razorpay and POST /api/webhooks/cashfree; duplicates are ignored safely (idempotent processing), signature mismatches are rejected.
5. Full detail and per-symptom fixes: Payments failing.

Runbook — backups and restore#

1. Super Admin → Settings → Backups (/super-admin/settings/backups): check the newest backup exists and Verify it (health + SQL statement count).
2. Restore: use the row's Restore action — it extracts the zip and imports the SQL, then reports files restored and rows imported. Do this on a test copy first when the production wipe is not the safer option.
3. If the situation is a bad update: the updater took a pre-update backup automatically — restore exactly that one.
4. After any restore, re-check the health endpoint, log a login test, and re-run a queue restart.

Runbook — suspected data exposure or abuse#

1. Rotate secrets: Meta App Secret, payment provider keys/webhook secrets, SMTP password, WhatsApp tokens, CRON_SECRET. Assume anything readable is burned.
2. Review the audit trail: admin actions and sensitive changes are recorded in the audit log (retained 180 days by default). The tenant's Data & Privacy page (/admin/privacy) is where exports and deletion requests live; do not delete anything while an incident is open.
3. Limit blast radius: disable a compromised integration connection, revoke affected users, and check Super Admin → Android App → Devices for unexpected device registrations.
4. Escalate — treat this as SEV1 and involve the platform owner before communicating.

Communicating during an incident#

• One channel, one person speaking: name an incident lead even if it is just you.
• Short updates, timestamped: what is affected, what we are doing, when the next update comes. Do not promise a root cause you do not have.
• When resolved: say what was fixed and what users need to do (usually only "reload" or "log in again").

After it is fixed (post-incident)#

1. Re-check the health endpoint and the specific thing that broke (a real lead through Meta, a real WhatsApp message, a real payment).
2. Write down: timeline, root cause, what fixed it, and the check that would have caught it earlier.
3. Add that check to Daily checks or Monitoring & alerts so the next occurrence is a non-event.

Common mistakes#

Technical details#

• Maintenance mode: php artisan down / php artisan up.
• System Health actions: Clear caches (optimize:clear), Run migrations (migrate --force), Rebuild caches, Storage link (storage:link), Restart queue (queue:restart) — /super-admin/settings/system-health.
• Backups: leadhub:backup (nightly, if enabled), Backups page for Verify/Restore/Download/Delete; BackupService::restore reports files restored and rows imported.
• Suspension vs outage: a suspended workspace redirects to /admin/subscription-required (not an outage) — see Payments failing.

Related guides#

• Daily checks
• Monitoring & alerts
• Deployment & server errors
• Queues, jobs & scheduler