에이전트 하네스 엔지니어링 주간 리포트 — 2026-10-03

This Week's Headlines

• AI System Design Patterns: The Journey from Demo Chatbot to Reliable Product — Cubed emphasized the importance of system design choices such as routing, retrieval, guardrails, and observability for production LLM applications, proving that raw model capabilities alone are not enough.

• LangGraph vs CrewAI vs AutoGen: An Honest Comparison After 6 Months — A production agent developer compared the architectural differences (graph-based vs crew-based), continuous execution, human approval, and resume features of three frameworks after six months of hands-on experience.

• LangGraph vs CrewAI Production Agents Comparison 2026 — Using invoice workflow examples, this piece compared graph and crew flows, continuous execution, human approval, and resumption, outlining specific trade-offs for each framework.

• Awesome Harness Engineering: GitHub Community Resource Emerges — An awesome list consolidating tools, patterns, evaluation, memory, MCP permissions, observability, and orchestration for AI agent harness engineering was released two days ago, including Anthropic's April 2026 design guide.

3 sources

dev.to

dev.to

media2.dev.to

media2.dev.to

dev.to

dev.to

Framework & Tooling Updates

1 sources

matthewswong.com

matthewswong.com

LangGraph — Production Graph-Based Orchestration

• What's new: Explicitly models agent states using graph structures with native support for human-in-the-loop loops and conditional resumption. State transitions between nodes are declarative and trackable.
• Why it matters: The explicit workflow structure makes bug tracking and cost control easier in complex multi-step agents. This is essential for regulated domains like billing and content approval.
• Migration notes: When migrating from CrewAI, you need to write explicit graphs instead of role-based abstractions and define state schemas using TypedDict.

CrewAI — Crew and Role-Centric Orchestration

• What's new: Defines each agent as a "role" and declares collaboration flows within a crew. Supports sequential and hierarchical task execution.
• Why it matters: Provides a high level of abstraction that non-technical stakeholders can easily understand. Great for rapid prototyping, but complex conditional logic or human approval loops are limited.
• Migration notes: Consider switching to a graph-based framework if you need explicit error handling or retry policies.

Research & Evaluation

Harness-Bench: Measuring Harness Effects in Realistic Agent Workflows

• Authors / Org: arXiv (2605.27922)
• Core finding: Points out the limitations of existing benchmarks like SWE-bench, Terminal-Bench, WebArena, and OSWorld, which rely on static tasks. Proves that harness design choices (retry policies, context windows, tool result compression) cause a 5% to 15% performance variance on the exact same model.
• Implication for harness design: Model benchmark scores alone cannot predict production success. Harness design must be treated as an independent variable, and multiple harnesses should be evaluated on the same model.

A Comparative Evaluation of AI Agent Security Guardrails

• Authors / Org: arXiv (2604.24826)
• Core finding: Compares DKnownAI Guard with AWS Bedrock Guardrails, Azure Content Safety, and Lakera Guard, showing a 60% to 85% variance in threat detection rates (injection, policy violations) across products.
• Implication for harness design: Guardrails are a necessity, not an option. Production agents require defense-in-depth (tool schema validation + LLM guardrails + runtime monitoring).

AI Agent Systems: A Comprehensive Review of Architectures, Applications, and Evaluation

• Authors / Org: arXiv (2601.01743)
• Core finding: Identifies core challenges in agent evaluation: tool action verification, scalable memory and context management, agent decision-making interpretability, and realistic workload replication.
• Implication for harness design: Evaluate using multi-metrics for cost, latency, safety, and interpretability instead of a simple binary success/failure. Explicitly model the trade-off between tool-use accuracy and safety.

Production Patterns & Practitioner Insights

Harness vs Model: The Real Issues in Production Design Are Routing and Guardrails

• Context: Insights gathered by Cubed engineers after deploying dozens of production LLM systems.
• Problem: Discrepancy between model performance benchmarks (MMLU, GSM8K) and actual production performance. High-scoring models still fail in production due to routing failures, context pollution, and bypassed guardrails.
• Solution / Takeaway: The four core pillars of production design are (1) Routing: classifying queries to the right agent/model, (2) Retrieval: search quality for relevant context, (3) Guardrails: blocking risky behavior, and (4) Observability: tracking decisions. Models alone cannot fulfill these; each requires independent engineering. A 10% improvement in routing classification accuracy lifts overall system performance by 5% to 10%.

6 Months in Production: Real Differences Between LangGraph and CrewAI

• Context: Evaluation by a team that implemented the exact same workflow in both frameworks and ran them for 6 months.
• Problem: CrewAI is easy for rapid prototyping, but limited when complex retry policies, cost controls, or human intervention are needed. LangGraph has a steeper initial setup, but its explicit state management makes long-term maintenance superior.
• Solution / Takeaway: Recommend CrewAI for prototypes (< 2 weeks) and LangGraph for production running over 6 months. Graph-based designs track state transitions explicitly, making it easy to see where costs pile up. For example, in the same billing workflow, CrewAI has implicit retry policies (up to 3 times) while LangGraph allows explicit limits at loop nodes.

Operating AI Agents in Production for 1 Year: Lessons Learned

• Context: A team that deployed agents across multiple domains (customer support, content moderation, billing).
• Problem: Contrary to initial expectations, agent success rates stalled at 70-80%. User satisfaction did not correlate with model scores; for example, GPT-4 achieved 85% success, but users rated it with 50% trust.
• Solution / Takeaway: (1) Human-in-the-loop is essential: feeding failure cases back into training data pushes success past 90% after 3 months. (2) Explicit cost control: unlimited retries cause cost spikes, so set hard token budgets and escalate to humans when exceeded. (3) Transparency builds trust: without a harness explaining agent decisions (e.g., "Answered via FAQ search, confidence 92%"), users won't trust the results.

Trending OSS Repositories

• awesome-harness-engineering — A centralized awesome list for AI agent harness design, tools, evaluation, memory, MCP, permissions, observability, and orchestration resources. Released 2 days ago.

• LangGraph (LangChain Official) — Graph-based agent orchestration with 13K+ stars. Features built-in production patterns like human-in-the-loop, state persistence, and conditional resumption.

• CrewAI — A role and crew-centric micro-agent orchestration framework characterized by rapid prototyping and high abstraction.

Deep Dive: Three Core Patterns for Production Agent Harness Design

Anthropic's newly released "Agent Harness Design: 3 Patterns for Harnessing Claude's Intelligence" stands as the most practical harness engineering guide available today. Its core premise is unpacking the complexity hidden behind the simple formula of Agent = Model + Harness.

Pattern One: Build on tools Claude already knows. You don't need to write every tool from scratch. Claude already understands basic tasks like database queries, HTTP requests, and file system access. The harness just needs to define these clearly as API schemas with input/output validation. For instance, a customer lookup tool is a simple SELECT query wrapper, but its schema must specify constraints like "required_fields" and "max_results". Cubed's analysis shows that a 10% improvement in tool schema quality raises agent success rates by 3% to 5%.

Pattern Two: Simplify the harness as model capabilities improve. When Anthropic upgraded from Opus 4.5 to 4.6, complex retry logic, context pre-processing, and output validation that were previously necessary became largely redundant. Lowering harness complexity reduces the error surface. Production teams should generally re-evaluate their harnesses two weeks after a model upgrade.

Pattern Three: Explicitly set boundaries for UX, cost, and safety. Unlimited retries lead to cost explosions, while overly restrictive limits hurt user satisfaction. A harness must provide control points that satisfy all three goals simultaneously. For example, a customer support agent needs: (1) UX: answers delivered within 5 seconds, (2) Cost: spending no more than $0.10 per query, and (3) Safety: never returning sensitive info (passwords, credit cards). LangGraph's conditional nodes implement these three as separate control logic, making trade-offs explicit.

Most production teams operating for over 6 months are currently migrating to LangGraph. While CrewAI remains superior for rapid prototyping, the complexities of harness design have re-established the value of explicit graph structures. Harness-Bench research proves that changing only the harness on the exact same model causes a 5% to 15% performance variance, proving it is a decision point as critical as framework selection.

What to Watch Next Week

• Anticipated Anthropic Claude Opus 4.7 Technical Card Update — Check whether the harness simplification trend continues and if new tool-use capabilities are added. Each previous release modified harness design recommendations.

• LangGraph 1.x Stabilization and Persistence API Expansion — Standardization of state management for long-running agents is underway, with expected increases in multi-database backend support.

• Official Adoption of SWE-bench 2.0 and Harness-Bench — Standardization of agent evaluation, requiring objective metrics to compare harness designs regardless of the underlying framework.

Reader Action Items

• Review Harness Design: Strengthen Routing and Guardrails — If you currently run a production agent system, evaluate Cubed's four pillars (routing, retrieval, guardrails, observability) and reinforce the weakest area first. Routing classification accuracy heavily dictates overall performance.

• Re-evaluate Framework Selection Based on Operational Timeline — Plan to use CrewAI for prototypes (< 2 weeks) and migrate to LangGraph for operations lasting 6 months or longer. Long-term maintenance costs and harness explicitness matter more than initial setup complexity.

• Introduce Cost Control Mechanisms — Set hard limits on token budgets and configure your harness to explicitly escalate to a human when exceeded. Unlimited retries can be fatal for financial agents.

• Bookmark the Awesome Harness Engineering Repository — This is the latest community-curated resource for tools, patterns, and evaluation frameworks. Check it monthly to monitor new guardrails and evaluation benchmarks.