On this tutorial, we dive deep into how we systematically benchmark agentic elements by evaluating a number of reasoning methods throughout numerous duties. We discover how completely different architectures, similar to Direct, Chain-of-Thought, ReAct, and Reflexion, behave when confronted with issues of accelerating problem, and we quantify their accuracy, effectivity, latency, and tool-usage patterns. By conducting managed empirical research, we acquire a clearer understanding of why sure agentic methods succeed, the place they fail, and the way they commerce off velocity for depth of reasoning. Try the FULL CODES here.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from typing import Checklist, Dict, Callable, Tuple
from dataclasses import dataclass
from enum import Enum
import time
from collections import defaultdict
class ReasoningStrategy(Enum):
DIRECT = "direct"
CHAIN_OF_THOUGHT = "chain_of_thought"
REACT = "react"
REFLEXION = "reflexion"
@dataclass
class AgentResponse:
reply: str
steps: int
time_taken: float
tool_calls: int
confidence: float
class BaseAgent:
def __init__(self, technique: ReasoningStrategy):
self.technique = technique
self.tool_count = 0
def remedy(self, downside: str) -> AgentResponse:
start_time = time.time()
if self.technique == ReasoningStrategy.DIRECT:
reply, steps, instruments = self._direct_solve(downside)
elif self.technique == ReasoningStrategy.CHAIN_OF_THOUGHT:
reply, steps, instruments = self._cot_solve(downside)
elif self.technique == ReasoningStrategy.REACT:
reply, steps, instruments = self._react_solve(downside)
else:
reply, steps, instruments = self._reflexion_solve(downside)
time_taken = time.time() - start_time
confidence = self._calculate_confidence(downside, reply)
return AgentResponse(reply, steps, time_taken, instruments, confidence)We arrange the muse of our benchmarking framework by importing important libraries and defining the core agent architectures. We set up completely different reasoning methods and assemble the BaseAgent class, giving ourselves a versatile construction to simulate numerous agentic behaviors. By means of this setup, we set up a unified interface that every one brokers comply with throughout analysis. Try the FULL CODES here.
def _direct_solve(self, downside: str) -> Tuple[str, int, int]:
reply = self._compute_answer(downside)
return reply, 1, 0
def _cot_solve(self, downside: str) -> Tuple[str, int, int]:
steps = 3 + len(downside.break up()) // 5
for i in vary(steps):
_ = self._reason_step(downside, i)
reply = self._compute_answer(downside)
return reply, steps, 0
def _react_solve(self, downside: str) -> Tuple[str, int, int]:
steps = 4
tool_calls = 2
for i in vary(steps):
_ = self._reason_step(downside, i)
if i % 2 == 0:
self._use_tool(downside)
reply = self._compute_answer(downside)
return reply, steps, tool_calls
def _reflexion_solve(self, downside: str) -> Tuple[str, int, int]:
steps = 6
tool_calls = 1
initial_answer = self._compute_answer(downside)
reflection = self._reflect(downside, initial_answer)
reply = self._refine(downside, initial_answer, reflection)
return reply, steps, tool_calls
def _reason_step(self, downside: str, step: int) -> str:
return f"Analyzing side {step+1}"
def _use_tool(self, downside: str):
self.tool_count += 1
time.sleep(0.001)
def _compute_answer(self, downside: str) -> str:
return f"Solution_{hash(downside) % 100}"
def _reflect(self, downside: str, reply: str) -> str:
return "Reflection on method"
def _refine(self, downside: str, reply: str, reflection: str) -> str:
return f"Refined_{reply}"
def _calculate_confidence(self, downside: str, reply: str) -> float:
base_confidence = 0.7
strategy_bonus = {
ReasoningStrategy.DIRECT: 0.0,
ReasoningStrategy.CHAIN_OF_THOUGHT: 0.1,
ReasoningStrategy.REACT: 0.15,
ReasoningStrategy.REFLEXION: 0.2
}
return min(1.0, base_confidence + strategy_bonus[self.strategy] + np.random.uniform(-0.1, 0.1))We implement how every reasoning technique behaves internally, together with direct answering, chain-of-thought reasoning, ReAct-style interleaving, and Reflexion-based refinement. We simulate reasoning steps, software utilization, and confidence estimation to seize life like agent conduct patterns. Right here, we form the dynamic persona of every agentic technique we benchmark. Try the FULL CODES here.
class BenchmarkTask:
def __init__(self, title: str, problem: float, ground_truth: str):
self.title = title
self.problem = problem
self.ground_truth = ground_truth
def consider(self, response: AgentResponse) -> Dict[str, float]:
accuracy = response.confidence * (1 - self.problem * 0.3)
return {
'accuracy': accuracy,
'effectivity': 1.0 / (response.steps + 1),
'latency': response.time_taken,
'tool_efficiency': 1.0 / (response.tool_calls + 1)
}
class BenchmarkSuite:
def __init__(self):
self.duties = self._create_tasks()
def _create_tasks(self) -> Checklist[BenchmarkTask]:
duties = []
task_types = [
("Math_Problem", 0.3),
("Logic_Puzzle", 0.5),
("Code_Debug", 0.6),
("Complex_Reasoning", 0.8),
("Multi_Step_Planning", 0.7)
]
for i, (task_type, problem) in enumerate(task_types):
for j in vary(3):
process = BenchmarkTask(
title=f"{task_type}_{j+1}",
problem=problem + np.random.uniform(-0.1, 0.1),
ground_truth=f"GT_{i}_{j}"
)
duties.append(process)
return duties
def run_benchmark(self, brokers: Checklist[BaseAgent]) -> pd.DataFrame:
outcomes = []
for agent in brokers:
for process in self.duties:
response = agent.remedy(process.title)
metrics = process.consider(response)
outcomes.append({
'technique': agent.technique.worth,
'process': process.title,
'problem': process.problem,
'accuracy': metrics['accuracy'],
'effectivity': metrics['efficiency'],
'latency': metrics['latency'],
'tool_efficiency': metrics['tool_efficiency'],
'steps': response.steps,
'tool_calls': response.tool_calls
})
return pd.DataFrame(outcomes)We construct the entire benchmark suite that generates duties, executes them throughout a number of brokers, and collects standardized outcomes. We design diversified process sorts and problem ranges to watch how every reasoning technique adapts below strain. This snippet permits us to create a reproducible and systematic analysis pipeline. Try the FULL CODES here.
def analyze_results(df: pd.DataFrame):
agg_metrics = df.groupby('technique').agg({
'accuracy': ['mean', 'std'],
'effectivity': ['mean', 'std'],
'latency': ['mean', 'std'],
'steps': 'imply',
'tool_calls': 'imply'
}).spherical(3)
print(agg_metrics)
diff_bins = pd.reduce(df['difficulty'], bins=3, labels=['Easy', 'Medium', 'Hard'])
diff_analysis = df.groupby(['strategy', diff_bins])['accuracy'].imply().unstack()
print(diff_analysis.spherical(3))
tradeoff = df.groupby('technique').agg({
'accuracy': 'imply',
'steps': 'imply',
'latency': 'imply'
})
tradeoff['score'] = (tradeoff['accuracy'] / (tradeoff['steps'] * tradeoff['latency'])).spherical(3)
print(tradeoff.spherical(3))
def visualize_results(df: pd.DataFrame):
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
sns.barplot(information=df, x='technique', y='accuracy', ax=axes[0, 0], errorbar="sd")
axes[0, 0].set_title('Accuracy by Technique')
axes[0, 0].tick_params(axis="x", rotation=45)
for technique in df['strategy'].distinctive():
strategy_df = df[df['strategy'] == technique]
axes[0, 1].scatter(strategy_df['steps'], strategy_df['accuracy'], label=technique, alpha=0.6, s=50)
axes[0, 1].set_title('Steps vs Accuracy')
axes[0, 1].legend()
difficulty_bins = pd.reduce(df['difficulty'], bins=3, labels=['Easy', 'Medium', 'Hard'])
df_plot = df.copy()
df_plot['difficulty_bin'] = difficulty_bins
sns.boxplot(information=df_plot, x='difficulty_bin', y='accuracy', hue="technique", ax=axes[1, 0])
axes[1, 0].set_title('Efficiency vs Issue')
scores = df.groupby('technique').apply(
lambda x: x['accuracy'].imply() / (x['steps'].imply() * x['latency'].imply())
).sort_values()
axes[1, 1].barh(vary(len(scores)), scores.values)
axes[1, 1].set_yticks(vary(len(scores)))
axes[1, 1].set_yticklabels(scores.index)
axes[1, 1].set_title('General Effectivity Rating')
plt.tight_layout()
plt.present()We carry out detailed evaluation and visualization to know how methods differ throughout metrics like accuracy, effectivity, and latency. We mixture outcomes, evaluate efficiency throughout problem ranges, and visualize trade-offs to uncover deeper insights. This step empowers us to interpret the outcomes reasonably than simply compute them. Try the FULL CODES here.
if __name__ == "__main__":
brokers = [
BaseAgent(ReasoningStrategy.DIRECT),
BaseAgent(ReasoningStrategy.CHAIN_OF_THOUGHT),
BaseAgent(ReasoningStrategy.REACT),
BaseAgent(ReasoningStrategy.REFLEXION)
]
suite = BenchmarkSuite()
results_df = suite.run_benchmark(brokers)
analyze_results(results_df)
visualize_results(results_df)
print("1. Superior methods obtain greater accuracy however require extra steps")
print("2. Chain-of-thought balances accuracy and effectivity")
print("3. Direct is quickest however much less dependable on laborious duties")
print("4. All methods degrade on tougher duties however superior ones degrade slowly")We deliver every part collectively by operating the benchmark suite on all brokers and printing the important thing findings. We execute the evaluation pipeline, visualize comparative outcomes, and interpret how methods behave below equivalent circumstances. This snippet completes the loop, permitting us to watch empirical patterns and derive significant conclusions.
In conclusion, we observe how completely different agentic reasoning paradigms carry out when subjected to equivalent benchmark circumstances, and we acquire sensible perception into how these methods scale with rising complexity. As we analyze patterns in accuracy, step rely, latency, and gear effectivity, we acknowledge how superior methods succeed by means of deeper reasoning whereas incurring computational overhead. We now stand outfitted with a structured empirical framework that helps us evaluate, debug, and optimize agentic behaviors, permitting us to construct extra succesful, data-driven agentic programs.
Try the FULL CODES here. Be at liberty to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Additionally, be happy to comply with us on Twitter and don’t neglect to affix our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its recognition amongst audiences.
