Building an AI Research Pipeline with X Content and Claude

Updated March 2026 · 10 min read

If you're regularly pulling insights from X content, you need a system. Not a one-off copy-paste, but a pipeline that takes X URLs in and gives you structured analysis out.

Right now, you're probably doing this: Find an interesting article on X. Open it. Copy the text. Paste it into a doc. Maybe paste it into Claude. Get an analysis back. Move on. Do it again tomorrow. By next week, you've got 20 articles scattered across tabs, emails, and notes with no way to track what you learned.

Here's the thing: manual research doesn't scale. You lose context across articles. You can't easily compare findings or spot trends. You're spending more time organizing than analyzing.

This post walks you through building a research pipeline — a repeatable system that converts X articles to structured Markdown, feeds them to Claude for analysis, and gives you organized outputs you can actually use.

Why Ad-Hoc Research Breaks Down

When you're pulling from X, you're usually working with:

• Thread links that disappear if the author deletes them
• Screenshot threads that lose context and aren't searchable
• One-off analyses that exist in your notes but don't connect to anything else
• No version control — you're not tracking what changed between reads

Manual research is fine if you're doing it once a month. But if you're monitoring industry trends, tracking competitor moves, or synthesizing multiple sources, you need structure. The moment you hit 5-10 articles, the friction increases exponentially.

Your Research Pipeline: The Flow

Here's the architecture:

X URL → xtomd.com (conversion) → Local storage (.md) → Claude (analysis) → Paired output files

Each stage has a specific job. X URLs are messy, inconsistent, and temporary. xtomd.com standardizes them into Markdown. Claude turns that Markdown into structured insights. You store both the source and the analysis as a pair.

Step 1: Collecting X URLs (Pick Your Trigger)

You need a consistent way to capture links. Common triggers:

• Direct bookmarking — Command+D when you land on an article worth analyzing
• Lists in X — Create private lists of accounts you monitor, check them weekly
• Saved links — Use X's bookmark feature, then export them periodically
• Topic monitoring — Set up keyword searches for specific domains or keywords

The key: store all your links in one place. A text file, a Notion database, a spreadsheet. Whatever. Just not scattered across your browser bookmarks and email.

Step 2: Batch Converting to Markdown

This is where xtomd.com comes in. Instead of converting one article at a time, batch them weekly or when you've accumulated 5-10 links.

For each X URL, go to xtomd.com and paste the link. The service converts it to clean Markdown.

Naming convention matters here: Save each file as {author}-{date}-{topic}.md. Example: sama-2025-03-11-ai-scaling.md. This makes searching and sorting trivial.

Step 3: Organizing Your Markdown Library

Add a metadata header to each file. Two minutes per file saves you days of searching later.

---
source_url: https://x.com/...
author: Name
date: 2025-03-11
topic: ai-scaling
keywords: llm, inference, optimization
status: unanalyzed
---

This YAML front matter makes your files searchable. You can filter by topic, date range, or status. If you use Obsidian, Logseq, or any Markdown editor with metadata support, you can build queries on top of this.

Step 4: Feeding to Claude for Analysis

Here's where the research becomes actionable.

Single article analysis: Paste one Markdown file into Claude. Ask for: key claims, evidence, gaps, relevance to your work, follow-up questions.

Prompt example:

Analyze this article for:
1. Core claims
2. Supporting evidence
3. Gaps or weaknesses
4. How it applies to [your context]
5. Questions this raises

Multi-article synthesis: Paste three related articles into Claude. Ask for: common themes, contradictions, what's new, which author was right. You're seeing patterns across sources in one go.

Trend identification: Feed Claude 10 articles on AI safety from the past month. Ask: What's the consensus shifting toward? What's controversial? What are people missing?

Save Claude's response as a .analysis.md file paired with the source:

sama-2025-03-11-ai-scaling.md           (source)
sama-2025-03-11-ai-scaling.analysis.md  (analysis)

Step 5: Storing and Tracking Analysis Output

Create two parallel folders:

research/
  ├── sources/           (original .md files)
  └── analyses/          (Claude output)

When you analyze an article, update the metadata status to "analyzed" with the analysis date and tool used. This gives you a searchable research history.

Real Example: Monitoring AI Thought Leaders

Let's say you follow 10 accounts on X posting about AI and product strategy. Here's your weekly workflow:

Monday morning: Check those accounts, collect new article links. You've got 7 new pieces.

Tuesday: Batch convert all 7 to Markdown using xtomd.com. Save them with naming convention.

Wednesday: Skim all 7 articles quickly. Pick 3-4 that matter most. Feed them to Claude with: "Synthesize these articles. What's the consensus on enterprise adoption? Where do these authors disagree?"

Thursday: Get Claude's synthesis. Save it as a synthesis file. You now have a structured view of what top thinkers are saying this week.

Friday: Reference the synthesis when writing your own content, making product decisions, or just staying informed.

Over a year, you've built a searchable archive of insights paired with sources. You can see how the conversation evolved. You can find expert takes on specific questions instantly. You can't do this with browser bookmarks.

For Developers: Programmatic Approach with xtomd API

If you're processing 20+ URLs regularly, use xtomd.com's API endpoint directly. No web interface needed.

curl -X POST https://xtomd.com/api/fetch \
  -H "Content-Type: application/json" \
  -d '{"url": "https://x.com/..."}'

Pipe this into your local research folder with a script, then feed the results to Claude's API programmatically. Automation handles the repetitive work. You focus on synthesizing insights.