How to Run Autonomous Code Experiments Without Losing Your Mind

The Problem: Manual Optimization is Slow and Error-Prone

You've written a function that works, but it's too slow. Or your API response times are creeping up. Or your test suite takes 20 minutes instead of 5. You know there's room for improvement, but the process of optimizing feels like stumbling in the dark.

You try changing one thing, run the tests, check the numbers. Then you try another change, run the tests again. After an hour, you've made five different modifications, and now you're not sure which one helped and which one made things worse. You forgot to commit the good version, so you're manually reverting changes. The whole process is tedious, error-prone, and mentally exhausting.

This happens because optimization is inherently iterative. You need to try things, measure results, and keep what works. But doing this manually means you're constantly context-switching between coding, running commands, parsing output, and making decisions. Your brain gets tired. You make mistakes. You lose track of what you've tried.

What a Good Solution Should Change

A better approach would let you:

• Define clear success criteria upfront (what metric are you optimizing, and how do you measure it?)
• Automate the try-measure-keep-or-revert cycle so you don't have to babysit each experiment
• Maintain a clean git history where each experiment is a separate commit that can be reverted cleanly
• Run experiments continuously until you interrupt, rather than stopping after each one to ask permission
• Keep a log of all experiments so you can see what worked and what didn't

The goal isn't to replace your judgment—it's to handle the repetitive parts of the optimization loop while you focus on defining the problem and reviewing the results.

Introducing the Autoresearch Skill

The autoresearch skill is an autonomous experimentation loop for programming tasks. It guides you through defining your optimization goal, the metric you'll use to measure success, and the scope of changes allowed. Then it runs a continuous loop: making code changes, running tests, measuring results, and keeping or reverting changes based on whether they improved your metric.

This skill is inspired by Andrej Karpathy's autoresearch concept, but generalized from machine learning training to any programming task with a measurable outcome. It's not a magic optimizer—it's a structured framework for running experiments systematically.

How It Works in Practice

The skill operates in three phases:

Phase 1: Setup (Interactive)

Before any experimentation begins, the skill walks you through defining:

• Goal: What are you trying to improve? (execution time, memory usage, test pass rate, etc.)
• Metric: How do you measure success? What command produces the metric? Is lower or higher better?
• Scope: Which files can be modified? Which are off-limits?
• Constraints: Any limits on time per experiment, dependencies, API compatibility, etc.
• Experiment budget: How many experiments to run, or run until interrupted?

This setup phase is interactive—the skill asks you questions directly and records your answers. It doesn't assume anything or skip steps.

Phase 2: Branch & Baseline

Once you confirm the setup, the skill:

1. Creates a new git branch (e.g., autoresearch/mar17)
2. Reads all in-scope files to understand the current code
3. Initializes a results log (results.tsv) to track experiments
4. Runs the baseline measurement on your unmodified code

Phase 3: Experiment Loop

Then it enters a continuous loop:

1. Think: Analyze previous results and generate a hypothesis
2. Edit: Make focused, minimal changes to in-scope files
3. Commit: Create a git commit with a descriptive message
4. Run: Execute your metric command
5. Measure: Extract the metric from the output
6. Decide: Keep the change if it improved the metric, revert if it didn't

The loop continues until you interrupt it or it reaches your experiment budget. Each experiment is logged in results.tsv with the commit hash, metric value, status (keep/revert), and a description of what changed.

When This Skill Fits Your Workflow

The autoresearch skill is designed for specific types of problems. It works well when:

• You have a measurable metric: You can quantify what "better" means (faster execution, lower memory usage, higher test pass rate, etc.)
• The optimization is iterative: You need to try multiple approaches and see what works
• Changes are localized: You can define a clear scope of files that might be modified
• You want to explore the solution space: Rather than knowing exactly what to change, you want the agent to try different approaches

Good Use Cases

• Performance tuning: Optimizing a slow function, reducing API latency, improving database query performance
• Memory optimization: Reducing memory usage in a data processing pipeline
• Test suite optimization: Reducing test execution time while maintaining coverage
• Build time optimization: Speeding up compilation or bundling
• Code simplification: Reducing complexity while maintaining functionality
• Benchmark improvement: Improving scores on standardized benchmarks

When Not to Use It

This skill is not suitable for:

• One-shot tasks: If you know exactly what change to make, just make it
• Simple bug fixes: If the fix is obvious, don't run an experiment loop
• Code review: This isn't for reviewing existing code
• Tasks without measurable metrics: If you can't quantify success, the skill can't evaluate experiments
• Exploratory coding: If you're trying to understand a problem rather than optimize a solution

Evaluating Whether This Skill Fits Your Needs

Before using the autoresearch skill, consider these questions:

Do You Have a Clear Metric?

The most important requirement is a measurable metric. You need:

1. A command that produces the metric (e.g., npm run benchmark, pytest --tb=short, time ./build.sh)
2. A way to extract the numeric metric from the command's output
3. A direction: Is lower better (like execution time) or higher better (like test pass rate)?

If you can't define these, the skill won't work for your use case.

Can You Define the Scope?

You need to specify which files the agent can modify. This is important for safety—you don't want the agent changing configuration files, documentation, or unrelated code. Be specific about what's in scope and what's off-limits.

Are You Comfortable with Autonomous Operation?

Once the loop starts, the skill runs autonomously. It won't ask permission before each experiment. It will make changes, run tests, and revert changes automatically. You need to be comfortable with this level of automation and trust that the constraints you've defined will keep it safe.

Do You Have Time to Review Results?

The skill produces a log of all experiments, but you'll need to review the results to understand what worked. The autonomous loop saves time on the try-measure-revert cycle, but you still need to analyze the outcomes.

Practical Considerations and Safety

Setup Requirements

The skill requires:

• Git: Your project must be a git repository
• Terminal access: The skill needs to run commands in your terminal
• A measurable metric: As discussed above

Safety Features

The skill has several built-in safety mechanisms:

• Scope constraints: It only modifies files you've explicitly allowed
• Git commits: Every experiment is committed before running, so you can revert any change
• Baseline measurement: It establishes a baseline before making any changes
• Automatic reversion: Changes that don't improve the metric are automatically reverted
• Simplicity policy: By default, it prefers simpler solutions over complex ones, even if the improvement is smaller

What to Inspect Before Using

Before running the skill, you should:

1. Review the repository: The skill comes from GitHub's awesome-copilot repository, which has over 37,000 stars and is actively maintained
2. Check the license: It's MIT licensed, so you can use it freely
3. Understand the constraints: The skill won't install new dependencies or make environment changes without your approval
4. Test the metric command: Make sure your metric command works correctly and produces consistent output
5. Define clear boundaries: Be specific about what files are in scope and what constraints apply

Limitations to Consider

• Not for all tasks: As mentioned, it's only for tasks with measurable metrics
• Requires clear goals: Vague goals like "make it better" won't work—you need specific, measurable objectives
• May not find global optima: Like any hill-climbing approach, it might get stuck in local optima
• Time investment: While it automates the experiment loop, you still need time to set it up and review results

Getting Started

If you've determined that the autoresearch skill fits your workflow, here's how to approach it:

1. Start with a clear problem: Identify a specific optimization opportunity with a measurable metric
2. Prepare your environment: Make sure your project is a git repository and your metric command works
3. Define your constraints carefully: Be specific about scope, constraints, and experiment budget
4. Run the setup phase: Let the skill guide you through defining all parameters
5. Review the baseline: Make sure the baseline measurement makes sense before starting experiments
6. Let it run: Start the loop and let it work autonomously
7. Review the results: After the loop completes (or you interrupt it), review the experiment log to understand what worked

The autoresearch skill won't solve every optimization problem, but for the right use case, it can save significant time and mental effort by automating the tedious parts of iterative experimentation.