Troubleshooting Administration

This document provides guidance for diagnosing and resolving administrative issues with integrations.

Before You Start

Information to Gather

When investigating issues, collect: - API key name and last 4 characters - Webhook subscription ID (if applicable) - Timestamp of failure - Error messages or HTTP status codes - Expected vs actual behavior

Request IDs

Include request IDs (from response headers) when contacting support for troubleshooting.

Common Issues

API Key Not Working

Symptom: All requests return 401 Unauthorized

Possible causes: - API key was revoked - API key copied incorrectly (truncated or extra spaces) - Wrong header format

Diagnostic steps:

1. Verify key exists and is active in Admin Portal
2. Check key format: wlcro_live_{companyId}_{secret}
3. Confirm header is X-Api-Key (exact capitalization)
4. Test with curl or Postman to isolate client code issues

Resolution: - If revoked: Create new key - If malformed: Re-copy from secure storage - If header wrong: Fix header name/format

Insufficient Permissions (403)

Symptom: 401 works for some endpoints, 403 for others

Cause: API key lacks required scope

Check: - Error response indicates which scope is required - Compare with scopes assigned to your API key

Resolution: - Create new API key with additional scopes - Cannot add scopes to existing keys

Common examples: - events:read required for GET /api/v1/events - webhooks:write required for subscription management - integrations:write required for creating records

Webhooks Not Arriving

Symptom: Events occur but webhook deliveries not received

Diagnostic steps:

1. Check subscription is enabled
2. Disabled subscriptions do not deliver

3. Circuit breaker may have auto-disabled after failures

4. Verify event types

5. Event must match one of the subscription's event types

6. Event type is case-sensitive (lowercase)

7. Check endpoint accessibility

8. URL must be HTTPS
9. Must be publicly accessible
10. Must respond within 10 seconds

11. Must return 2xx status

12. Use test endpoint

13. Send test delivery: POST /api/v1/webhooks/subscriptions/{id}/test

14. Confirms URL, signature, and endpoint response

15. Review delivery logs

16. Check for errors, timeouts, or non-2xx responses

Common causes: - Endpoint returns 500 or 400 - Endpoint times out (> 10 seconds) - Firewall blocking WhiteLabelCRO IPs - Signature verification failing (returns 401)

Resolution: - Fix endpoint to return 200-299 for valid webhooks - Optimize endpoint to respond under 10 seconds - Whitelist webhook source IPs (if firewalled) - Fix signature verification logic

Webhook Signature Verification Failing

Symptom: Webhooks arrive but fail signature check

Common mistakes:

1. Using wrong signing secret (from different subscription?)
2. Parsing request body as JSON before verification (modifies body)
3. Not using raw request body bytes
4. Incorrect HMAC computation format

Verification checklist: - Compute HMAC over "{timestamp}.{raw_body}" (exact format with dot) - Use raw request body exactly as received (no parsing, no whitespace changes) - Use constant-time comparison for signatures - Signing secret from subscription creation response

Test: - Temporarily disable signature verification to confirm deliveries arrive - Log the computed vs received signatures to debug mismatch

Rate Limits Being Hit

Symptom: Frequent 429 responses

Diagnostic steps:

1. Check which bucket
2. Error response includes policy field (e.g., read:60/1m)

3. Confirms which limit is being hit

4. Monitor X-RateLimit-Remaining

5. Log remaining count from response headers

6. Identify request patterns that exhaust quota

7. Check for multiple integrations

8. Is the same API key shared across systems?
9. Each system counts toward the same limit

Resolution: - Implement Retry-After header handling - Slow down request rate (longer polling intervals, batch operations spread over time) - Use separate API keys for separate integrations - Switch to webhooks instead of polling (if applicable)

Events Not Appearing When Polled

Symptom: GET /api/v1/events returns empty or missing expected events

Possible causes:

1. Events haven't occurred yet (confirm action happened in CRM)
2. Using since or until parameters that exclude the events
3. API key lacks events:read scope

Diagnostic steps: - Confirm event actually occurred (check CRM UI) - Remove since/until filters to see all recent events - Check event type filter if using type parameter - Verify API key has events:read scope

Resolution: - Adjust query parameters - Confirm event type is one of the 10 v1 types - Wait for event to occur, then poll again

Circuit Breaker Disabled Subscription

Symptom: Webhook subscription shows as disabled; you didn't disable it

Cause: 20 consecutive delivery failures triggered auto-disable

Resolution:

1. Review delivery logs to identify failure pattern
2. Fix root cause (endpoint error, timeout, signature issue)
3. Test with /test endpoint to confirm fix
4. Re-enable subscription via PATCH request

Prevention: - Monitor delivery success rate - Set up alerts for repeated failures - Fix issues before circuit breaker triggers

Tool-Managed Subscription Cannot Be Edited

Symptom: PATCH request returns 403 with "managed by Zapier" message

Cause: Zapier (or n8n, Make) created and owns this subscription

Resolution: - Edit the Zap/workflow in Zapier/n8n to change configuration - Cannot manually edit tool-managed subscriptions - You can still disable/enable via isEnabled field

Lost Signing Secret

Symptom: Need signing secret but didn't save it at creation

Cause: Signing secrets are returned only once at subscription creation

Resolution: - No way to retrieve the original secret - Create new subscription (returns new secret) - Update endpoint configuration with new secret - Delete old subscription

Timeouts During Bulk Operations

Symptom: Some requests time out during large batch imports

Possible causes: - Server under heavy load - Individual requests too complex - Network instability

Resolution: - Reduce batch size (process fewer records per minute) - Retry with exponential backoff - Spread operations over longer time window - Check for 503 responses (system contention, not rate limit)

Admin Portal Navigation

Accessing Integration Settings

Navigate to Settings > Integrations in the Admin Portal to access API keys, webhook subscriptions, and integration settings.

Viewing API Keys

Access Settings > Integrations > API Keys to view, create, and revoke API keys.

Viewing Webhook Subscriptions

Access webhook subscription management through the Integration API. Contact your administrator for UI-based subscription viewing if required.

Viewing Delivery Logs

Delivery logs are available to administrators. Contact your administrator for log access and filtering options.

Monitoring

Proactive Monitoring

Set up monitoring for: - API key last used timestamp: Identify dormant keys - Webhook delivery success rate: Detect failing subscriptions before circuit breaker - Rate limit consumption: Alert before hitting limits - Integration errors: Track 4xx/5xx responses

Use external monitoring tools or log aggregation to track these metrics.

Health Checks

Periodically verify: - Active API keys are still in use - Webhook subscriptions are delivering successfully - No subscriptions stuck in disabled state - No unexpected 429 or 503 responses

Escalation

When to Contact Support

Escalate to support when: - Issue persists after following troubleshooting steps - Server returns repeated 500 errors - Behavior contradicts documented API behavior - Rate limits seem incorrectly enforced - Data discrepancies (event not fired when CRM action occurred)

Information to Provide

When opening support ticket, include: - API key name (NOT the full key value) - Webhook subscription ID (if applicable) - Exact timestamps of failures (UTC) - Full error responses (JSON) - Request ID header (if available) - Steps to reproduce

What NOT to Share

Do not share in support tickets: - Full API key secret - Signing secrets - Other sensitive credentials

Share only key names, IDs, and last 4 characters for identification.

Preventive Measures

Regular Reviews

Monthly or quarterly: - Audit active API keys; revoke unused ones - Review webhook subscription health - Check for integrations with low success rates - Update integration partner credentials if rotated

Documentation for Integration Partners

Provide partners with: - API key scopes required for their integration - Expected event types they'll receive - Rate limits that apply to their usage - Signature verification examples - Retry and error handling guidance

Backup Integration Patterns

For critical integrations: - Use both webhooks and polling (webhook primary, polling as fallback) - Maintain separate API keys for redundancy - Monitor both channels for discrepancies

Change Management

Before making changes: - Test in non-production environment if available - Create new API key before revoking old one - Monitor delivery logs after subscription changes - Keep old configuration accessible for rollback