Batches

Batches allow you to group related tasks together and monitor them as a unit. Use batches when you need to process multiple tasks in parallel while tracking their collective progress or executing callbacks based on group completion.

Understanding batches

Batches provide several key capabilities:

• Group related tasks and monitor them as a single unit
• Execute callbacks when a task completes, all tasks complete or if any task fails
• Track progress and status of the entire group
• Atomically create multiple tasks
• Add additional tasks to existing batches
• Cancel entire batches of tasks

Create a batch

curl https://api.taskhook.io/v1/batches \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Process CSV rows",
    "notifications": {
      "on_complete": {
        "url": "https://example.com/hooks/batch-complete",
        "method": "POST"
      },
      "on_success": {
        "url": "https://example.com/hooks/batch-success",
        "method": "POST"
      }
    },
    "tasks": [
      {
        "target": {
          "url": "https://example.com/process-row/1",
          "method": "POST"
        },
        "payload": {"row": 1}
      },
      {
        "target": {
          "url": "https://example.com/process-row/2",
          "method": "POST"
        },
        "payload": {"row": 2}
      }
    ],
    "options": {
      "atomic": true,
      "expiration": 24
    }
  }'

{
  "id": "batch_123",
  "status": "pending",
  "description": "Process CSV rows",
  "notifications": {
    "on_complete": {
      "url": "https://example.com/hooks/batch-complete",
      "method": "POST"
    },
    "on_success": {
      "url": "https://example.com/hooks/batch-success",
      "method": "POST"
    }
  },
  "stats": {
    "total": 2,
    "pending": 2,
    "completed": 0,
    "failed": 0
  },
  "options": {
    "atomic": true,
    "expiration": 24
  },
  "created_at": "2024-01-30T12:00:00Z",
  "updated_at": "2024-01-30T12:00:00Z",
  "completion_rate": 0
}

Retrieve a batch

curl https://api.taskhook.io/v1/batches/batch_123 \
  -H "Authorization: Bearer {token}"

{
  "id": "batch_123",
  "status": "processing",
  "description": "Process CSV rows",
  "notifications": {
    "on_complete": {
      "url": "https://example.com/hooks/batch-complete",
      "method": "POST"
    },
    "on_success": {
      "url": "https://example.com/hooks/batch-success",
      "method": "POST"
    }
  },
  "stats": {
    "total": 2,
    "pending": 1,
    "completed": 1,
    "failed": 0
  },
  "options": {
    "atomic": true,
    "expiration": 24
  },
  "created_at": "2024-01-30T12:00:00Z",
  "updated_at": "2024-01-30T12:05:00Z",
  "completion_rate": 50
}

List all batches

curl -G https://api.taskhook.io/v1/batches \
  -H "Authorization: Bearer {token}" \
  -d status=processing \
  -d limit=10

{
  "total": 1,
  "batches": [
    {
      "id": "batch_123",
      "status": "processing",
      "description": "Process CSV rows",
      "stats": {
        "total": 2,
        "pending": 1,
        "completed": 1,
        "failed": 0
      },
      "created_at": "2024-01-30T12:00:00Z",
      "updated_at": "2024-01-30T12:05:00Z",
      "completion_rate": 50
    }
  ]
}

Add tasks to batch

curl https://api.taskhook.io/v1/batches/batch_123/tasks \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{
    "tasks": [
      {
        "target": {
          "url": "https://example.com/process-row/3",
          "method": "POST"
        },
        "payload": {"row": 3}
      }
    ],
    "atomic": true
  }'

{
  "id": "batch_123",
  "status": "processing",
  "stats": {
    "total": 3,
    "pending": 0,
    "completed": 1,
    "cancelled": 2,
    "failed": 0
  },
  "updated_at": "2024-01-30T12:15:00Z",
  "completion_rate": 33.33
}

List batch tasks

curl -G https://api.taskhook.io/v1/batches/batch_123/tasks \
  -H "Authorization: Bearer {token}" \
  -d status=completed

{
  "total": 1,
  "tasks": [
    {
      "id": "task_456",
      "batch_id": "batch_123",
      "status": "completed",
      "target": {
        "url": "https://example.com/process-row/1",
        "method": "POST"
      },
      "payload": {
        "row": 1
      },
      "created_at": "2024-01-30T12:00:00Z",
      "completed_at": "2024-01-30T12:05:00Z"
    }
  ]
}

Cancel a batch

curl -X POST https://api.taskhook.io/v1/batches/batch_123/cancel \
  -H "Authorization: Bearer {token}"

{
  "id": "batch_123",
  "status": "cancelled",
  "stats": {
    "total": 3,
    "pending": 0,
    "completed": 1,
    "cancelled": 2,
    "failed": 0
  },
  "updated_at": "2024-01-30T12:15:00Z",
  "completion_rate": 33.33
}

Notification Payload Format

When batch events occur, Taskhook will make HTTP requests to your configured URLs. Each notification receives details about the batch and its current state.

Notification payload

{
  "event": "on_complete",
  "batch": {
    "id": "batch_123",
    "status": "completed",
    "stats": {
      "total": 3,
      "pending": 0,
      "completed": 3,
      "failed": 0
    },
    "created_at": "2024-01-30T12:00:00Z",
    "completed_at": "2024-01-30T12:15:00Z",
    "completion_rate": 100
  },
  "first_failure": {
    // Only included for on_death events
    "task_id": "task_789",
    "error": "Connection timeout",
    "failed_at": "2024-01-30T12:10:00Z"
  },
  "timestamp": "2024-01-30T12:15:00Z"
}

Notification retry behavior

• Notifications will be retried up to 5 times with exponential backoff
• Retries occur at approximately 30s, 2m, 5m, 10m, and 30m intervals
• A 2xx response is considered successful
• After all retries are exhausted, the notification is marked as failed but does not affect the batch status

Working with schedules

The Schedules API supports creating batches on a recurring basis. When configuring a schedule, you can specify a batch template that will be used to create new batches at the scheduled intervals.

Batch template in schedules

{
  "cron": "0 0 * * *",
  "timezone": "UTC",
  "batch_template": {
    "description": "Daily data processing batch",
    "notifications": {
      "on_complete": {
        "url": "https://example.com/hooks/batch-complete",
        "method": "POST"
      }
    },
    "task_templates": [
      {
        "target": {
          "url": "https://example.com/process/1",
          "method": "POST"
        }
      },
      {
        "target": {
          "url": "https://example.com/process/2",
          "method": "POST"
        }
      }
    ],
    "options": {
      "atomic": true,
      "expiration": 24
    }
  }
}

Error-handling

Common errors

Error responses

{
  "error": {
    "code": "batch_completed",
    "message": "Cannot add tasks to a completed batch",
    "details": {
      "batch_id": "batch_123",
      "status": "completed"
    }
  }
}

Best practices

1. Atomic creation: Keep the atomic option enabled unless you have a specific reason to disable it. This ensures consistency in batch creation.

2. Batch size: While there's no strict limit on batch size, consider these guidelines:

   • Keep batches under 10,000 tasks for optimal performance
   • For larger sets, create multiple batches or use dynamic task addition

3. Notifications: Design notification endpoints to be idempotent as they may be called multiple times due to retries.

4. Task independence: Ensure tasks within a batch can execute independently and in any order.

5. Expiration: Set appropriate expiration times based on expected completion duration:

   • 24 hours for most batches
   • Up to 7 days for very large batches
   • Consider breaking up batches that need longer processing

6. Error-handling:

   • Use the on_death notification for early failure detection
   • Implement proper retry logic in your notification endpoints
   • Monitor completion_rate for batch progress

Rate-limits

Batch-related endpoints have specific rate-limits:

• Batch creation: 100 requests per minute
• Adding tasks: 60 requests per minute per batch
• Status queries: 300 requests per minute per batch
• WebSocket connections: 10 concurrent connections per batch