Troubleshooting Matrix¶
← Back to Home • Deployment Guide • Configuration Reference • API Reference
HTTP Status Codes & Solutions¶
| Status | Cause | Resolution |
|---|---|---|
400 Hook rules were not satisfied. |
Trigger rule mismatch (e.g. invalid HMAC secret, missing header, payload branch mismatch). | Run server with --verbose to inspect trigger evaluation logs. Verify HMAC secret matches provider. |
405 Method Not Allowed |
Webhook sent with an HTTP method not listed in hook's http-methods. |
Update hook's http-methods array or send the request using POST. |
413 Payload Too Large |
Request payload exceeds server.max_body_bytes. |
Increase server.max_body_bytes in config.toml (default is 10 MB). |
429 Queue capacity full |
In-flight queued jobs have reached queue.max_pending_jobs. |
Increase worker concurrency (worker.concurrency), add dedicated worker nodes, or increase queue.max_pending_jobs. |
429 Rate limit exceeded |
Exceeded 60 requests/minute on administrative endpoints. | Wait for the duration in Retry-After: 60 or throttle management scripts. |
500 Internal Server Error |
Synchronous command returned non-zero exit code or rule evaluation threw an unhandled error. | Inspect stdout/stderr in response (if include-command-output-in-response-on-error: true) or inspect server logs. |
503 Service Unavailable (On Hook Post) |
Storage backend is unreachable when accepting an asynchronous webhook. | Check SQLite filesystem permissions or verify Valkey/Redis network connectivity. Webhook was not acknowledged. |
503 Admin token not configured |
Calling /api/v1/* without configuring admin_token, admin_token_env, or admin_token_file. |
Set admin_token = "..." directly in config.toml, or set WEBHOOK_ADMIN_TOKEN in the environment. |
Worker & Execution Issues¶
Job Transitions Immediately to dead_letter¶
- Cause: Executable path or working directory does not exist or lacks execute permissions.
- Why:
webhook-asyncclassifies missing binaries as non-transientConfigurationerrors and dead-letters immediately to prevent infinite retry loops. - Solution: Use absolute executable paths (e.g.
/usr/local/bin/deploy.sh) and ensure thewebhookuser has read and execute permissions.
Worker Heartbeat / Lease Expiry¶
- Symptom: Job stays in
runningand is reclaimed by another worker after 60s. - Cause: Task process completely locked up the CPU or network, preventing the asynchronous heartbeat loop from renewing the lease.
- Solution: Ensure system resources are sufficient, or increase
worker.lease_duration_seconds(e.g. to 120s).
database is locked on SQLite¶
- Cause: High concurrency without Write-Ahead Logging (WAL) or missing busy timeout.
- Solution: Verify that
wal_mode = trueandbusy_timeout_ms = 5000are configured in[storage.sqlite].
Diagnostic Commands¶
Validate Configuration Files¶
webhook-async config validate --config config.toml --hooks hooks.json
Check Storage Health¶
webhook-async storage status --config config.toml
Test Webhook Triggers Locally¶
curl -v -X POST http://localhost:9000/hooks/test-hook \
-H "Content-Type: application/json" \
-d '{"test": true}'