How Do You Systematically Profile Competitors Without Days of Manual Research?

The Problem: Drowning in Competitor Tabs, Starving for Insights

You've just been asked to prepare a competitive analysis for next week's strategy meeting. You open a browser, type in the first competitor's URL, and begin the familiar ritual: clicking through their homepage, hunting for a pricing page, scanning their blog for recent announcements, and trying to decipher their positioning from marketing copy. You open a second tab, then a third. An hour later, you have 15 browser tabs open, a messy document of half-formed notes, and a growing sense that you're missing the bigger picture.

This is the reality for most product marketers, founders, and sales teams tasked with competitive intelligence. The core problem isn't a lack of information—it's the absence of a structured, repeatable process. Manual research is:

• Time-consuming: Profiling even one competitor thoroughly can take 2-4 hours. Multiply that by five or ten competitors, and you've lost a full work week.
• Inconsistent: Each researcher focuses on different aspects. One might emphasize pricing, another might fixate on feature lists. The resulting notes are impossible to compare side-by-side.
• Surface-level: Without access to SEO tools, traffic data, or backlink profiles, you're limited to what the competitor chooses to show you on their website. You miss their actual market position, content strategy effectiveness, and organic search strength.
• Point-in-time: The research becomes stale quickly. There's no easy way to re-run the analysis next quarter and see what changed.

A good solution would need to do several things differently. It should accept a list of URLs and automatically extract structured data from key pages—homepage, pricing, features, about, and integrations. It should pull quantitative SEO metrics like domain authority, keyword rankings, and traffic estimates to provide an objective market position. It should save raw data for auditability and future comparison. And it should output standardized profile documents that allow for true apples-to-apples comparison across competitors.

Introducing the Competitor Profiling Skill

The competitor-profiling skill is an AI agent skill designed to automate this exact workflow. It's not a standalone SaaS product you log into; it's a set of instructions and tool integrations that an AI agent (like those built on Claude or similar platforms) can follow to conduct structured competitive research.

The skill takes a list of competitor website URLs as input and produces comprehensive, structured competitor profile documents as output. It combines two data sources: live website scraping (via Firecrawl) for qualitative positioning and messaging data, and SEO/market data (via DataForSEO) for quantitative metrics like keyword rankings, backlink profiles, and traffic estimates.

Think of it as giving your AI agent a detailed research playbook. Instead of you manually clicking through sites and taking notes, the agent follows a defined process: map the site structure, scrape key pages, pull SEO data, save raw data to disk, and synthesize everything into a standardized profile template.

How the Skill Works: A Step-by-Step Breakdown

Understanding the internal workflow helps you evaluate whether it fits your needs. The process has three distinct phases.

Phase 1: Site Scraping with Firecrawl

The skill begins by using Firecrawl's mapping capability to discover the structure of each competitor's website. This identifies key pages automatically—homepage, pricing, features, about, customers, integrations, and changelog—without you needing to specify exact URLs.

Once the site map is established, it scrapes each key page individually. From each page type, it extracts specific data points:

• Homepage: Headline, value proposition, primary call-to-action, social proof claims, target audience signals
• Pricing: Tiers, prices, feature breakdowns, billing options, free tier details
• Features: Feature categories, key capabilities, how features are described
• About: Founding story, team size, funding, mission statement
• Customers: Named customers, logos, industries served, case study themes
• Integrations: Integration count, key integrations, categories

Optionally, it can also scrape review platforms like G2, Capterra, or Product Hunt to capture customer sentiment data—overall ratings, common praise themes, and common complaint themes.

Phase 2: SEO and Market Data with DataForSEO

This is where the skill moves beyond what most manual researchers can do. It uses DataForSEO tools to gather quantitative competitive intelligence:

• Domain authority and backlinks: Domain rank, total backlinks, referring domains count, spam score, and top referring domains
• Keyword intelligence: Total organic keywords ranking, keywords in top 3/10/100 positions, estimated organic traffic, and top keywords by traffic value
• Competitive positioning: Closest organic competitors (which may reveal competitors you hadn't considered), market overlap data, and highest-traffic pages

This data provides an objective lens. If a competitor claims to be "the leading solution" on their homepage, but their domain authority is low and their organic traffic is minimal, that tells you something important about the gap between their marketing claims and market reality.

Phase 3: Synthesis and Output

The skill combines the scraped content with SEO data to build a structured profile. It cross-references claims—if a competitor states they have "10,000 customers" on their site, it checks whether their traffic and backlink profile supports that scale of operation.

The output is a markdown file per competitor, following a consistent template. This consistency is critical: it means you can compare competitors side-by-side on the same dimensions. A summary document aggregates key findings across all profiled competitors.

Data Organization and Raw Data Preservation

One practical aspect worth noting is how the skill handles data storage. Before synthesizing the final profile, it persists all raw scrape data, SEO responses, and review data to disk in a structured directory:

competitor-profiles/
├── raw/
│   └── <competitor-slug>/
│       └── <YYYY-MM-DD>/
│           ├── scrapes/
│           ├── seo/
│           └── reviews/
├── <competitor-slug>.md
└── _summary.md

Each run creates a date-stamped folder, so you never overwrite previous data. This means you can:

• Audit the research: Open any raw file to see exactly what data was collected and from where
• Re-run comparisons: Run the skill again next quarter and diff the results against previous snapshots
• Reuse raw data: Feed the scraped content or SEO data into other analyses without re-running expensive API calls

The final profile document references its raw data sources, creating a clear chain from conclusion to evidence.

When This Skill Fits Your Workflow

This skill is most useful in specific scenarios. It's a good fit if you:

• Regularly need competitive intelligence for product positioning, sales enablement, or strategic planning
• Work with multiple competitors (three or more) where manual research becomes prohibitively time-consuming
• Want quantitative data (SEO metrics, traffic estimates) alongside qualitative positioning analysis
• Need consistent, comparable outputs rather than ad-hoc notes that vary by researcher
• Value auditability and want to trace profile claims back to source data

It's particularly well-suited for product marketing teams preparing positioning documents, founders entering a new market who need to understand the landscape, and sales teams who need foundational competitive research before creating battle cards.

When to Consider Alternatives

The skill has clear boundaries. It may not be the right choice if:

• You only need to profile one competitor quickly. The overhead of setting up the tooling may exceed the time saved for a single profile.
• You need real-time monitoring. This skill produces point-in-time snapshots, not continuous surveillance. For ongoing monitoring, you'd need a different tool.
• Your competitors have heavily gated content. The skill scrapes publicly accessible pages. If critical information lives behind login walls or paywalls, it won't be captured.
• You need sales battle cards specifically. The skill produces research profiles, not sales-ready battle cards. The repository notes that for sales-specific battle cards, a separate sales-enablement skill exists.
• You need comparison or alternative pages. For creating comparison content from profiles, a separate competitors skill in the same repository handles that workflow.

Setup Context and Dependencies

Before using this skill, you need to understand its dependencies. The skill relies on two external services:

1. Firecrawl — for website mapping and scraping. You'll need API access and credentials configured.
2. DataForSEO — for SEO metrics, keyword data, and competitive analysis. This also requires API credentials.

The skill is designed to work within an AI agent environment that supports tool calling (MCP tools). It's not a standalone script you run from the command line; it's a set of instructions your agent follows, using these tools as needed.

The skill also checks for a product-marketing.md context file in your project. If it exists, the agent reads it first to understand your product, reducing the number of questions it needs to ask before starting research.

Safety and Repository Signals

A few signals worth evaluating before adopting this skill:

• License: MIT — permissive and open, meaning you can modify and use it freely
• Repository activity: The marketingskills repository has over 34,000 stars, indicating significant community interest and likely ongoing maintenance
• Security level: Rated as low risk. The skill primarily orchestrates external API calls and writes files to your local project directory. It doesn't execute arbitrary code or require elevated permissions
• Topics: Tagged with marketing, codex, claude, indicating it's designed for AI agent environments with marketing use cases

As with any skill that makes external API calls, review the tool reference documentation to understand exactly what data is sent to third-party services. The raw data preservation feature is a positive signal—you can always inspect exactly what was collected.

Practical Example: What a Profile Looks Like

To make this concrete, imagine you run a project management SaaS and want to profile three competitors. You provide their URLs, and the skill produces:

• competitor-profiles/asana.md — A structured profile covering Asana's positioning, pricing tiers (Free, Starter at $10.99/user/mo, Advanced at $24.99/user/mo), key features, domain authority score, top organic keywords, estimated monthly traffic, customer review themes, and recent product direction signals from their changelog
• competitor-profiles/monday.md — Same structure, same dimensions, directly comparable
• competitor-profiles/notion.md — Same structure again
• competitor-profiles/_summary.md — A cross-competitor comparison highlighting positioning differences, pricing gaps, SEO strength rankings, and content strategy observations

Each profile includes a Raw Data Sources section linking to the date-stamped raw data folder, so you or a colleague can verify any claim in the profile.

What to Inspect Before Using It

If you're considering this skill, here's a practical checklist:

1. Review the SKILL.md file in the repository to understand the full prompt and process instructions
2. Check the template files referenced in the skill (references/templates.md) to see if the output format matches your needs
3. Verify API access — confirm you have or can obtain Firecrawl and DataForSEO API credentials
4. Test with one competitor first — run the skill against a single, well-known competitor to evaluate output quality before scaling
5. Review the raw data structure — after one run, inspect the saved raw data to understand what's being collected and stored
6. Check the related skills — if you need comparison pages or sales battle cards, look at the competitors and sales-enablement skills in the same repository

The competitor-profiling skill page provides additional context and links to the source repository.

Final Thoughts

Competitive intelligence doesn't have to mean drowning in browser tabs and producing inconsistent notes. A structured, automated approach can save hours of work while producing more thorough and comparable results. The key is matching the tool to your specific workflow: if you regularly profile multiple competitors and want both qualitative positioning data and quantitative SEO metrics in a consistent format, this skill addresses that need directly. If your needs are simpler or more specialized, the same repository offers adjacent skills worth exploring.