How AI Agents Can Use X Content with xtomd.com

Updated March 2026 · 9 min read

AI agents are getting good at web research. They navigate sites, extract data, reason about what they find. But they all hit the same wall: X. The platform blocks automated fetches, serves JavaScript-only pages, and breaks tools that worked fine on every other website.

You want your CrewAI researcher to monitor competitors on X. Your AutoGPT workflow needs to analyze trending threads. Your OpenClaw skill should track brand mentions. None of these work out of the box.

This post shows you how to integrate xtomd.com into your AI agent stack — and why it matters.

Why do AI agents struggle with X URLs?

X isn't like other websites. When a regular web scraper hits an X URL, here's what happens:

1. Empty HTML shells — X sends minimal HTML. The actual content loads in the browser via JavaScript.
2. User agent blocking — Send a bot user agent, X refuses to respond. It knows what scrapers look like.
3. Authentication requirements — Some threads require login. APIs demand API keys. Most agents don't handle this.
4. Aggressive rate limiting — Fetch too many X URLs in sequence, and X stops responding for hours.

This isn't a bug. It's intentional. X protects its platform by making automated access expensive and unreliable.

Every AI agent framework assumes web content is readable. Crawl a news site, parse the HTML, extract the text. But X breaks that assumption. Your agent gets empty responses, timeouts, or 403 errors.

How xtomd.com solves the X problem

xtomd.com is built for this. The API endpoint (POST /api/fetch) accepts an X URL and returns structured JSON with the content already parsed, formatted, and ready for your agent to use.

Here's the flow:

1. Your agent encounters an X URL.
2. Instead of hitting X directly, it calls xtomd.com/api/fetch with the URL.
3. xtomd.com handles the browser rendering, authentication, and formatting.
4. Your agent receives clean Markdown with the thread, replies, media links, and metadata.

No rate limiting. No user agent blocking. No JavaScript rendering delays.

Integrating xtomd.com with CrewAI

CrewAI agents work by assigning tasks to specialized crew members. A researcher agent handles web research. A content curator finds relevant material.

Build a custom tool that calls xtomd.com:

from crewai import Tool
import requests

def fetch_x_content(url: str) -> str:
    """Fetch X content via xtomd.com API"""
    response = requests.post(
        "https://xtomd.com/api/fetch",
        json={"url": url}
    )
    if response.status_code == 200:
        data = response.json()
        return data.get("markdown", "")
    return "Failed to fetch content"

x_fetch_tool = Tool(
    name="fetch_x_content",
    func=fetch_x_content,
    description="Fetch and convert X URLs to readable Markdown"
)

Now your CrewAI agent can call this tool directly in a task to research competitor threads or analyze X content.

Integrating xtomd.com with AutoGPT

AutoGPT uses a configuration file to define available tools. Add xtomd.com as a web tool:

tools:
  - name: web_fetch
    type: api
    description: "Fetch web content including X threads"
    config:
      base_url: "https://xtomd.com/api"
      endpoint: "/fetch"
      method: "POST"
      headers:
        Content-Type: "application/json"
      params:
        - name: "url"
          type: "string"
          required: true
      response_type: "json"
      extract: "markdown"

AutoGPT will automatically route X URLs through xtomd.com when it detects them. The agent sees the result as clean Markdown, just like any other web page.

Integrating xtomd.com with OpenClaw

OpenClaw skills can be extended with custom handlers. When the standard web_fetch fails on an X URL, add a fallback:

class XContentSkill:
    async def handle_x_url(self, url: str):
        try:
            return await self.web_fetch(url)
        except Exception:
            # Fallback to xtomd.com
            response = await self.http_client.post(
                "https://xtomd.com/api/fetch",
                json={"url": url}
            )
            return response.json()["markdown"]

Using xtomd.com output with Claude Code

Claude Code can pipe xtomd.com output directly into context. This is useful for one-off analysis without building a full workflow.

curl -X POST https://xtomd.com/api/fetch \
  -H "Content-Type: application/json" \
  -d '{"url": "https://x.com/user/status/12345"}' | \
  jq '.markdown' | claude code --input -

The Markdown output becomes context for Claude Code analysis. Analyze sentiment, extract data, generate reports — all with X content that would normally be inaccessible.

Real-world use cases for agent + X content

Competitive monitoring — Your agent tracks competitor announcements, product launches, and strategic changes on X. Weekly reports on strategic shifts.

Trend analysis — Feed trending X threads into your agent. It identifies emerging patterns, technologies, or market movements relevant to your industry.

Content curation — Build an automated newsletter by having your agent fetch top X threads, summarize them, and organize by topic.

Lead research — When your CRM agent encounters a prospect, it can search for their X profile and historical tweets to understand their background and interests.

Brand mentions — Monitor brand mentions on X. Your agent fetches threads, assesses sentiment, and alerts you to critical customer feedback or PR issues.