Skip to main content
Use Prefect when the agent helps design or run a collection workflow, but scheduled execution needs durable state and retries.

​
Prerequisites

  • Python 3.10+
  • A Twexapi API key
  • A Prefect deployment or local flow runner
  • Storage for state such as next_cursor, last_success_at, and selected endpoint path

​
Pattern

  • Use MCP during planning and ad-hoc collection.
  • Persist the selected endpoint path, query parameters, and pagination state.
  • Run scheduled collection through Prefect tasks.
  • Use Twexapi REST directly for deterministic scheduled runs.

​
Flow skeleton

import os
import requests
from prefect import flow, task


@task(retries=3, retry_delay_seconds=30)
def fetch_trending_tweets(country: str, topic: str, content: str, count: int = 20):
    response = requests.get(
        "https://api.twexapi.io/twitter/global-trending/tweets",
        headers={"Authorization": f"Bearer {os.environ['TWEXAPI_API_KEY']}"},
        params={
            "country": country,
            "topic": topic,
            "content": content,
            "count": count,
        },
        timeout=60,
    )
    response.raise_for_status()
    return response.json()


@flow
def ai_trends_flow():
    data = fetch_trending_tweets("united-states", "technology", "AI")
    return data

​
MCP planning prompt

Use Twexapi MCP explore to choose the best endpoint for daily AI trend collection.
Return the exact method, relative path, query parameters, pagination fields, expected response fields, and retry guidance.
Do not execute write endpoints.

​
Handoff payload

Ask the planning agent to return this payload, then store it in Prefect variables or your own config. 
{
  "route_used": "/twitter/global-trending/tweets",
  "method": "GET",
  "query": {
    "country": "united-states",
    "topic": "technology",
    "content": "AI",
    "count": 20
  },
  "pagination": {
    "has_more": false,
    "next_cursor": null
  },
  "auth": {
    "header": "Authorization: Bearer TWEXAPI_API_KEY"
  }
}

​
State to persist

StatePurpose
pathStable endpoint chosen by the agent.
queryScheduled job parameters.
next_cursorResume pagination.
last_success_atRecovery after failures.
route_usedAudit trail.

​
Deployment recipe

1

Plan with MCP

Ask an agent to use explore and return the exact method, relative path, and parameters.
2

Codify the route

Store the selected endpoint in Prefect variables or your own config.
3

Run with REST

Use deterministic REST calls inside Prefect tasks.
4

Persist state

Save next_cursor, row counts, and last successful run time after every page.

​
Error handling

StatusPrefect behavior
401Fail fast and alert; the API key is missing or invalid.
403Pause the deployment until credits or permissions are fixed.
429Retry with delay and keep the same cursor.
5xxRetry the task; do not advance next_cursor until success.

​
Testing checklist

  • Run the flow locally with count=5.
  • Verify logs include the selected path and request parameters.
  • Verify state persists after each successful page.
  • Verify retries do not duplicate rows because tweet_id or user_id is used as a primary key.