Building an AI Research Pipeline with X Content and Claude

Updated March 2026 · 10 min read

If you're regularly pulling insights from X content, you need a system. Not a one-off copy-paste, but a pipeline that takes X URLs in and gives you structured analysis out.

Right now, you're probably doing this: Find an interesting article on X. Open it. Copy the text. Paste it into a doc. Maybe paste it into Claude. Get an analysis back. Move on. Do it again tomorrow. By next week, you've got 20 articles scattered across tabs, emails, and notes with no way to track what you learned.

Here's the thing: manual research doesn't scale. You lose context across articles. You can't easily compare findings or spot trends. You're spending more time organizing than analyzing.

This post walks you through building a research pipeline—a repeatable system that converts X articles to structured Markdown, feeds them to Claude for analysis, and gives you organized outputs you can actually use.

Why Ad-Hoc Research Breaks Down

When you're pulling from X, you're usually working with:

• Thread links that disappear if the author deletes them
• Screenshot threads that lose context and aren't searchable
• One-off analyses that exist in your notes but don't connect to anything else
• No version control — you're not tracking what changed between reads

Manual research is fine if you're doing it once a month. But if you're monitoring industry trends, tracking competitor moves, or synthesizing multiple sources, you need structure. The moment you hit 5-10 articles, the friction increases exponentially.

A pipeline solves this by removing decision points. You don't decide how to save each article. You don't manually type up metadata. You don't lose the source after Claude analyzes it. The system handles it.

Your Research Pipeline: The Flow

Here's the architecture:

X URL → xtomd.com (conversion) → Local storage (.md) → Claude (analysis) → Paired output files

Each stage has a specific job. X URLs are messy, inconsistent, and temporary. xtomd.com standardizes them into Markdown. Claude turns that Markdown into structured insights. You store both the source and the analysis as a pair.

This matters because later, when you're writing a blog post or building a product, you can reference the source and the analysis at the same time. You're not re-analyzing the same article twice.

Step 1: Collecting X URLs (Pick Your Trigger)

You need a consistent way to capture links. This isn't about saving everything. It's about defining what "interesting" means to you and having a system to catch it.

Common triggers:

• Direct bookmarking — Command+D when you land on an article worth analyzing
• Lists in X — Create private lists of accounts you monitor, check them weekly
• Saved links — Use X's bookmark feature, then export them periodically
• Topic monitoring — Set up keyword searches for specific domains or keywords

The key: store all your links in one place. A text file, a Notion database, a spreadsheet. Whatever. Just not scattered across your browser bookmarks and email.

Step 2: Batch Converting to Markdown

This is where xtomd.com comes in. Instead of converting one article at a time, batch them weekly or when you've accumulated 5-10 links.

For each X URL, go to xtomd.com and paste the link. The service converts it to clean Markdown. You get structured content with metadata preserved.

Naming convention matters here: Save each file as {author}-{date}-{topic}.md. Example: sama-2025-03-11-ai-scaling.md. This makes searching and sorting trivial. You can grep for topics, sort by date, or group by author.

Create a folder structure like:

research/
  ├── ai-safety/
  ├── llm-architecture/
  ├── product-strategy/
  └── competitive-analysis/

Put converted files in the appropriate folder. Consistency now saves you hours later.

Step 3: Organizing Your Markdown Library

You've got 20 Markdown files. They're named consistently, sorted into folders. Now what?

Add a metadata header to each file. Not optional—actually do this. Two minutes per file saves you days of searching later.

---
source_url: https://x.com/...
author: Name
date: 2025-03-11
topic: ai-scaling
keywords: llm, inference, optimization
status: unanalyzed
---

# Original content here

This YAML front matter makes your files searchable. You can filter by topic, date range, or status. If you use Obsidian, Logseq, or any Markdown editor with metadata support, you can build queries on top of this.

Also: don't overthink the folder structure. Three to five top-level categories. Anything deeper and you'll spend time organizing instead of reading.

Step 4: Feeding to Claude for Analysis

Here's where the research becomes actionable.

Single article analysis: Paste one Markdown file into Claude. Ask for: key claims, evidence, gaps, relevance to your work, follow-up questions. Claude gives you structured analysis in minutes.

Multi-article synthesis: This is where a pipeline wins. Paste three related articles into Claude. Ask for: common themes, contradictions, what's new, which author was right. You're seeing patterns across sources in one go.

Trend identification: Feed Claude 10 articles on AI safety from the past month. Ask: What's the consensus shifting toward? What's controversial? What are people missing?

Save Claude's response as a .analysis.md file paired with the source. This pairing means you can always trace an insight back to its source.

Step 5: Storing and Tracking Analysis Output

Create two parallel folders:

research/
  ├── sources/           (original .md files)
  └── analyses/          (Claude output)

When you analyze an article, update the metadata with analysis date and tool. This gives you a searchable research history. Three months from now, you can query: "Show me everything I analyzed about LLM scaling."

For longer projects, add a synthesis file that pulls insights from multiple analyses into one document. You're building a knowledge base, not a pile of notes.

Real Example: Monitoring AI Thought Leaders

Let's say you follow 10 accounts on X posting about AI and product strategy. Here's your weekly workflow:

Monday morning: Check those accounts, collect new article links. You've got 7 new pieces.

Tuesday: Batch convert all 7 to Markdown using xtomd.com. Save them with consistent naming: dario-2025-03-10-ai-products.md, satya-2025-03-09-enterprise-ai.md, etc.

Wednesday: Skim all 7 articles quickly. Pick 3-4 that matter most. Feed them to Claude for synthesis.

Thursday: Get Claude's synthesis. Save it as a weekly summary. You now have a structured view of what top thinkers are saying.

Friday: Reference the synthesis when writing your own content, making product decisions, or staying informed. Drill down to original sources anytime.

Over a year, you've built a searchable archive of insights paired with sources. You can see how the conversation evolved. You can find expert takes on specific questions instantly.