Webhooks - Mission Control

Overview

Webhooks enable Mission Control to push events to your external services in real-time. Configure webhook endpoints to receive notifications about tasks, agents, activities, and security events.

Webhooks use HMAC-SHA256 signatures for security and include automatic retry with exponential backoff.

Quick Start

Create Webhook

curl -X POST http://localhost:3000/api/webhooks \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_ADMIN_KEY" \
  -d '{
    "name": "Production Alerts",
    "url": "https://your-app.com/webhooks/mission-control",
    "events": ["activity.task_created", "agent.status_change", "security.login_failed"],
    "generate_secret": true
  }'

Response:

{
  "id": 1,
  "name": "Production Alerts",
  "url": "https://your-app.com/webhooks/mission-control",
  "secret": "3f7a8c2e9d1b4a6f5e8c7d9a2b4c6e8f1a3b5c7d9e1f3a5b7c9d1e3f5a7b9c1",
  "events": ["activity.task_created", "agent.status_change", "security.login_failed"],
  "enabled": true,
  "message": "Webhook created. Save the secret - it won't be shown again in full."
}

Save the secret immediately! It’s only shown once. You’ll need it to verify webhook signatures.

Verify Signature in Your Handler

Validate incoming webhooks using the HMAC signature:

const crypto = require('crypto');
const express = require('express');

const app = express();

// RAW body parser (required for signature verification)
app.use(express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf.toString('utf8');
  }
}));

app.post('/webhooks/mission-control', (req, res) => {
  const signature = req.headers['x-mc-signature'];
  const rawBody = req.rawBody;
  const secret = process.env.WEBHOOK_SECRET;

  // Verify signature
  const expectedSignature = 'sha256=' + 
    crypto.createHmac('sha256', secret)
      .update(rawBody)
      .digest('hex');

  if (!crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expectedSignature)
  )) {
    return res.status(401).json({ error: 'Invalid signature' });
  }

  // Process event
  const { event, data, timestamp } = req.body;
  console.log(`Received event: ${event}`, data);

  res.json({ ok: true });
});

app.listen(3001);

from flask import Flask, request, jsonify
import hmac
import hashlib
import os

app = Flask(__name__)

@app.route('/webhooks/mission-control', methods=['POST'])
def webhook():
    signature = request.headers.get('X-MC-Signature')
    raw_body = request.get_data(as_text=True)
    secret = os.getenv('WEBHOOK_SECRET')
    
    # Verify signature
    expected_signature = 'sha256=' + hmac.new(
        secret.encode('utf-8'),
        raw_body.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()
    
    if not hmac.compare_digest(signature, expected_signature):
        return jsonify({'error': 'Invalid signature'}), 401
    
    # Process event
    payload = request.json
    event = payload['event']
    data = payload['data']
    
    print(f'Received event: {event}', data)
    
    return jsonify({'ok': True})

if __name__ == '__main__':
    app.run(port=3001)

package main

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
)

type WebhookPayload struct {
    Event     string                 `json:"event"`
    Timestamp int64                  `json:"timestamp"`
    Data      map[string]interface{} `json:"data"`
}

func verifySignature(rawBody []byte, signature string, secret string) bool {
    mac := hmac.New(sha256.New, []byte(secret))
    mac.Write(rawBody)
    expectedMAC := "sha256=" + hex.EncodeToString(mac.Sum(nil))
    return hmac.Equal([]byte(signature), []byte(expectedMAC))
}

func webhookHandler(w http.ResponseWriter, r *http.Request) {
    rawBody, _ := ioutil.ReadAll(r.Body)
    signature := r.Header.Get("X-MC-Signature")
    secret := os.Getenv("WEBHOOK_SECRET")

    if !verifySignature(rawBody, signature, secret) {
        http.Error(w, "Invalid signature", http.StatusUnauthorized)
        return
    }

    var payload WebhookPayload
    json.Unmarshal(rawBody, &payload)

    fmt.Printf("Received event: %s\n", payload.Event)

    json.NewEncoder(w).Encode(map[string]bool{"ok": true})
}

func main() {
    http.HandleFunc("/webhooks/mission-control", webhookHandler)
    http.ListenAndServe(":3001", nil)
}

Test Webhook

curl -X POST http://localhost:3000/api/webhooks/test \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_ADMIN_KEY" \
  -d '{"id": 1}'

Response:

{
  "success": true,
  "status_code": 200,
  "response_body": "{\"ok\":true}",
  "error": null,
  "duration_ms": 142,
  "delivery_id": 1
}

Event Types

Subscribe to specific events or use * for all events.

Activity Events

Event	Description
`activity.task_created`	New task created
`activity.task_updated`	Task modified
`activity.task_deleted`	Task deleted
`activity.task_status_changed`	Task status changed
`activity.task_assigned`	Task assigned to agent
`activity.comment_added`	Comment added to task
`activity.agent_created`	New agent provisioned
`activity.connection_created`	CLI connection established
`activity.connection_disconnected`	CLI connection closed

Agent Events

Event	Description
`agent.status_change`	Agent status changed (online, offline, idle, busy, error)
`agent.error`	Agent entered error state

Notification Events

Event	Description
`notification.mention`	Agent mentioned in comment
`notification.task_assigned`	Task assigned notification
`notification.info`	Informational notification
`notification.warning`	Warning notification
`notification.error`	Error notification

Security Events

Event	Description
`security.login_success`	Successful login
`security.login_failed`	Failed login attempt
`security.api_key_used`	API key authentication
`security.unauthorized_access`	Unauthorized access attempt
`security.role_changed`	User role modified

Wildcard

Event	Description
`*`	Subscribe to all events

Start with * during development, then narrow to specific events for production to reduce traffic.

Webhook Payload Format

All webhooks deliver JSON with this structure:

{
  "event": "activity.task_created",
  "timestamp": 1709582400,
  "data": {
    "id": 123,
    "title": "Fix authentication bug",
    "status": "inbox",
    "priority": "high",
    "assigned_to": "debugging-agent",
    "created_by": "alice",
    "created_at": 1709582400
  }
}

string

Event type identifier

number

Unix timestamp (seconds) when event occurred

object

Event-specific payload. Structure varies by event type.

HTTP Headers

Mission Control sends these headers with every webhook:

Header	Value	Purpose
`Content-Type`	`application/json`	Payload format
`User-Agent`	`MissionControl-Webhook/1.0`	Identify sender
`X-MC-Event`	`activity.task_created`	Event type (for routing)
`X-MC-Signature`	`sha256=abc123...`	HMAC-SHA256 signature

Signature Verification

Mission Control signs all webhook payloads using HMAC-SHA256.

Signature Format

X-MC-Signature: sha256=<hex_digest>

The signature is computed as:

HMAC-SHA256(secret, raw_json_body)

Verification Algorithm

Extract Components

Get X-MC-Signature header
Get raw request body (UTF-8 string, before parsing JSON)
Get webhook secret from creation response

Compute Expected Signature

const expectedSignature = 'sha256=' + 
  crypto.createHmac('sha256', secret)
    .update(rawBody)
    .digest('hex');

Compare Using Constant-Time Function

if (!crypto.timingSafeEqual(
  Buffer.from(receivedSignature),
  Buffer.from(expectedSignature)
)) {
  throw new Error('Invalid signature');
}

Always use constant-time comparison to prevent timing attacks:

Node.js: crypto.timingSafeEqual()
Python: hmac.compare_digest()
Go: hmac.Equal()

Never use === or == for signature comparison!

Helper Function

Mission Control exports a verification helper:

import { verifyWebhookSignature } from '@/lib/webhooks';

const isValid = verifyWebhookSignature(
  secret,
  rawBody,
  req.headers['x-mc-signature']
);

Retry Logic

Webhooks automatically retry on failure with exponential backoff.

Retry Schedule

Attempt	Delay	Cumulative Time
1	Immediate	0s
2	30s ± 20%	~30s
3	5m ± 20%	~5.5m
4	30m ± 20%	~35m
5	2h ± 20%	~2h 35m
6	8h ± 20%	~10h 35m

Total attempts: 6 (1 initial + 5 retries)

Retries include ±20% jitter to prevent thundering herd issues.

Success Criteria

A delivery is considered successful if:

HTTP status code: 200-299
Response received within 10 seconds

Failure Criteria

A delivery fails if:

HTTP status code: ≥300 or connection error
Request times out after 10 seconds
Network error (DNS, connection refused, etc.)

Circuit Breaker

After 5 consecutive failures (configurable via MC_WEBHOOK_MAX_RETRIES):

Webhook is automatically disabled
No further deliveries are attempted
Log entry: Webhook circuit breaker tripped — disabled after exhausting retries

To re-enable:

curl -X PUT http://localhost:3000/api/webhooks \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_ADMIN_KEY" \
  -d '{"id": 1, "reset_circuit": true}'

This:

Resets consecutive_failures to 0
Sets enabled to true
Allows new deliveries

Management API

List Webhooks

curl http://localhost:3000/api/webhooks \
  -H "x-api-key: YOUR_ADMIN_KEY"

Response:

{
  "webhooks": [
    {
      "id": 1,
      "name": "Production Alerts",
      "url": "https://your-app.com/webhooks/mission-control",
      "secret": "••••••9c1",
      "events": ["activity.task_created", "agent.status_change"],
      "enabled": true,
      "consecutive_failures": 0,
      "circuit_open": false,
      "last_fired_at": 1709582400,
      "last_status": 200,
      "total_deliveries": 142,
      "successful_deliveries": 140,
      "failed_deliveries": 2,
      "created_at": 1709400000,
      "created_by": "admin"
    }
  ]
}

Update Webhook

curl -X PUT http://localhost:3000/api/webhooks \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_ADMIN_KEY" \
  -d '{
    "id": 1,
    "name": "Updated Name",
    "events": ["agent.status_change", "agent.error"],
    "enabled": false
  }'

number

required

Webhook ID to update

string

New webhook name

string

New endpoint URL

array

New event subscriptions

boolean

Enable or disable webhook

boolean

Generate new secret (returns in response)

boolean

Reset circuit breaker and re-enable webhook

Delete Webhook

curl -X DELETE http://localhost:3000/api/webhooks \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_ADMIN_KEY" \
  -d '{"id": 1}'

Deletes webhook and all delivery history.

View Delivery History

curl "http://localhost:3000/api/webhooks/deliveries?webhook_id=1&limit=20" \
  -H "x-api-key: YOUR_ADMIN_KEY"

Response:

{
  "deliveries": [
    {
      "id": 123,
      "webhook_id": 1,
      "event_type": "activity.task_created",
      "status_code": 200,
      "response_body": "{\"ok\":true}",
      "error": null,
      "duration_ms": 142,
      "attempt": 0,
      "is_retry": 0,
      "next_retry_at": null,
      "created_at": 1709582400
    }
  ],
  "total": 142
}

Mission Control keeps the last 200 deliveries per webhook.

Manual Retry

curl -X POST http://localhost:3000/api/webhooks/retry \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_ADMIN_KEY" \
  -d '{"delivery_id": 123}'

Manually retry a failed delivery (useful for debugging).

Environment Configuration

# Maximum retry attempts before circuit breaker trips (default: 5)
MC_WEBHOOK_MAX_RETRIES=5

Best Practices

Idempotency

Webhooks may deliver the same event multiple times (retries). Design handlers to be idempotent:

const processedEvents = new Set();

app.post('/webhook', (req, res) => {
  const eventId = `${req.body.event}-${req.body.data.id}`;
  
  if (processedEvents.has(eventId)) {
    return res.json({ ok: true }); // Already processed
  }
  
  processEvent(req.body);
  processedEvents.add(eventId);
  res.json({ ok: true });
});

Fast Response

Respond quickly (< 1s) to avoid timeouts:

app.post('/webhook', async (req, res) => {
  // Respond immediately
  res.json({ ok: true });
  
  // Process asynchronously
  processEventAsync(req.body).catch(err => {
    console.error('Background processing failed:', err);
  });
});

Secret Rotation

Rotate webhook secrets periodically:

Create new webhook with new secret
Update your handler to accept both secrets
Wait for old deliveries to drain (24h)
Delete old webhook

Monitor Failures

Set up alerts for:

consecutive_failures > 3
circuit_open = true
Sudden spike in failed deliveries

Query delivery stats:

SELECT webhook_id, 
  COUNT(*) as total,
  SUM(CASE WHEN status_code BETWEEN 200 AND 299 THEN 1 ELSE 0 END) as success
FROM webhook_deliveries
WHERE created_at > unixepoch() - 86400
GROUP BY webhook_id;

Troubleshooting

Webhook deliveries fail with 'Invalid signature'

Cause: Your handler’s signature verification is incorrect.Debug steps:

Ensure you’re using raw body (before JSON parsing)

Log both signatures for comparison:

console.log('Received:', req.headers['x-mc-signature']);
console.log('Expected:', expectedSignature);

Verify secret matches (check for leading/trailing whitespace)
Use timingSafeEqual() for comparison

Test with curl:

# Generate test signature
echo -n '{"event":"test","data":{}}' | \
  openssl dgst -sha256 -hmac "YOUR_SECRET" | \
  awk '{print "sha256="$2}'

Circuit breaker keeps tripping

Cause: Your endpoint is unreliable or timing out.Solutions:

Check webhook delivery logs: GET /api/webhooks/deliveries?webhook_id=1
Ensure your handler responds within 10 seconds
Return 200 status code on success
Check for network/firewall issues
Temporarily disable circuit breaker: MC_WEBHOOK_MAX_RETRIES=999

Reset circuit:

curl -X PUT http://localhost:3000/api/webhooks \
  -H "Content-Type: application/json" \
  -H "x-api-key: ADMIN_KEY" \
  -d '{"id": 1, "reset_circuit": true}'

Not receiving events for specific event types

Cause: Event not in subscription list or event mapping issue.Debug:

List webhook config: GET /api/webhooks
Check events array includes the event type

Update subscription:

curl -X PUT http://localhost:3000/api/webhooks \
  -d '{"id": 1, "events": ["*"]}'

Test delivery: POST /api/webhooks/test

Internal events vs webhook events: Some internal events are mapped to webhook event types (see EVENT_MAP in /src/lib/webhooks.ts:36).

Retries not working

Cause: Scheduler not running or deliveries not recorded.Check scheduler:

curl http://localhost:3000/api/scheduler \
  -H "x-api-key: ADMIN_KEY"

Manually trigger retry processing:

curl -X POST http://localhost:3000/api/scheduler \
  -H "x-api-key: ADMIN_KEY" \
  -d '{"action": "run_now", "job": "webhook_retries"}'

Security Considerations

Always verify signatures in production. An attacker could forge webhook payloads without signature verification.

HTTPS Only

Use HTTPS endpoints in production:

✅ https://your-app.com/webhook
❌ http://your-app.com/webhook

Mission Control allows HTTP for local development only.

IP Whitelisting

Restrict webhook requests to Mission Control server IPs:

# nginx example
location /webhooks/mission-control {
    allow 10.0.1.0/24;  # MC server subnet
    deny all;
    proxy_pass http://localhost:3001;
}

Rate Limiting

Protect your webhook endpoint:

const rateLimit = require('express-rate-limit');

const limiter = rateLimit({
  windowMs: 1 * 60 * 1000, // 1 minute
  max: 100, // 100 requests per minute
});

app.use('/webhooks', limiter);

Logging

Log all webhook deliveries for audit:

app.post('/webhook', (req, res) => {
  logger.info({
    event: req.body.event,
    timestamp: req.body.timestamp,
    signature_valid: verifySignature(...),
    source_ip: req.ip,
  }, 'Webhook received');
  
  res.json({ ok: true });
});

CLI Integration

Real-time events via Server-Sent Events

GitHub Sync

Webhook automation examples

Event Bus

Internal event system

​Overview

​Quick Start

​Event Types

​Activity Events

​Agent Events

​Notification Events

​Security Events

​Wildcard

​Webhook Payload Format

​HTTP Headers

​Signature Verification

​Signature Format

​Verification Algorithm

​Helper Function

​Retry Logic

​Retry Schedule

​Success Criteria

​Failure Criteria

​Circuit Breaker

​Management API

​List Webhooks

​Update Webhook

​Delete Webhook

​View Delivery History

​Manual Retry

​Environment Configuration

​Best Practices

Idempotency

Fast Response

Secret Rotation

Monitor Failures

​Troubleshooting

​Security Considerations

HTTPS Only

IP Whitelisting

Rate Limiting

Logging

​Related Docs

CLI Integration

GitHub Sync

Event Bus

Overview

Quick Start

Event Types

Activity Events

Agent Events

Notification Events

Security Events

Wildcard

Webhook Payload Format

HTTP Headers

Signature Verification

Signature Format

Verification Algorithm

Helper Function

Retry Logic

Retry Schedule

Success Criteria

Failure Criteria

Circuit Breaker

Management API

List Webhooks

Update Webhook

Delete Webhook

View Delivery History

Manual Retry

Environment Configuration

Best Practices

Troubleshooting

Security Considerations

Related Docs