Benchmark lab
CriogenAI, LangGraph and CrewAI, on the same work
All three in Python, in the way their own documentation shows, against one simulated model with the same replies and delays. Each scenario runs in a fresh process per framework. Everything below is read from one results file: the machine, the versions, the command and every measurement.
For what CriogenAI's Rust engine, SDK bridges and gateway cost on their own, see Benchmarks in the docs.
The recorded run
Download raw JSON- Machine
- AMD Ryzen 5 4600H with Radeon Graphics, 15.4 GB RAM, Windows 11
- Python
- 3.13.14
- Recorded
- 2026-10-21 18:47 UTC
- Frameworks
- Criogen 0.1.0
- LangGraph 1.2.16 with langchain 1.4.6, langchain-core 1.6.12
- CrewAI 1.15.28
- Command
python benches/compare/run.py
Head to head
Every measurement of the recorded run, the frameworks side by side. Hover or focus a bar for its details; “Show the numbers” under each group opens them as a table, with the best value of each row marked.
Speed
Median time per run, after a warm-up; the label says how many times longer than the fastest.
Under load
Many agent runs at once in one process (model 100 ms per call, one tool of 50 ms each): how long the whole batch takes, and the memory it needs.
Memory and CPU
What one process needs: its peak memory while running agents one by one, and the CPU time each run costs.
Reliability
The same faults injected into each framework, with its defaults. Each test's reason and log are in the matrix below.
Footprint
What you install and wait for: import time in a fresh interpreter, the size of the virtual environment, and the packages it pulls in.
Scenarios
Agent, one model call agent_call
One model call, no tools, the model answers at once: the framework's cost per agent turn.
The race
- Criogen290 µs1st
- LangGraph1.23 ms2nd
- CrewAI33.9 ms3rd
Each lane finishes in time proportional to its median per run. Replay slowed down 118×: 1 ms measured takes 118 ms on screen.
* The simulated-model adapter alone, one call, measured in the same process (so the adapters are not what differs): LangGraph 145 µs · CrewAI 3.37 µs
Every number
| Framework | Runs | Median | p95 | Min | Max | Wall (all runs) | CPU per run |
|---|---|---|---|---|---|---|---|
| Criogen | 300 | 290 µs | 687 µs | 192 µs | 1.04 ms | 101 ms | 365 µs |
| LangGraph | 300 | 1.23 ms | 2.28 ms | 1.02 ms | 3.66 ms | 419 ms | 1.35 ms |
| CrewAI | 300 | 33.9 ms | 40.4 ms | 29.4 ms | 62.1 ms | 10.3 s | 33.3 ms |
Distribution
Dots: single runs (per run). Box: middle half. Thick line: median. Thin tick: 95th percentile. Logarithmic axis: the frameworks are more than 20× apart.
Memory and CPU
Reliability
Each test injects the same fault through the shared model or tools, with each framework's defaults. Select a cell for the reason and the log.
- ✓ pass as the test defines it
- ◐ partial part of it
- ✕ fail not at all
| Test | Criogen | LangGraph | CrewAI |
|---|---|---|---|
model_transientThe model raises a rate-limit error once, then answers. Pass: The run completes without the caller's code retrying. | |||
tool_exceptionA tool raises. Pass: The model sees the error and the run completes with an answer. | |||
bad_argumentsThe model calls a tool with a wrong argument type. Pass: The tool function never sees the bad arguments, and the run completes. | |||
crash_resumeThe process is killed while the 2nd of 2 tools runs (the 1st finished). Pass: A new process continues the same conversation from storage, and the finished tool is not run again. | |||
runaway_loopThe model asks for a tool forever. Pass: The run stops by itself, with an error, within its default limit. |
Footprint
What you install, and how long the import takes in a fresh interpreter.
| Framework | Cold import | Packages installed | Install size |
|---|---|---|---|
| Criogen | 148 ms | 1 | 17.9 MB |
| LangGraph | 1.7 s | 42 | 42.9 MB |
| CrewAI | 9.3 s | 138 | 743 MB |
Check it on your machine
Download the kit: the benchmark exactly as it ran here, with the code for each framework and the method (METHOD.md). It needs Python 3.10 to 3.13 and no API keys: every framework talks to the same scripted model.
Download the benchmark kit (.zip)
cd criogen-compare-kit
python setup.py # once: a virtual environment per framework (pinned versions)
python run.py # everything, about 25 minutes → bench/results/<date>-<machine>.json
python run.py quick # a short smoke test (not for comparing)Then load the file it writes:
Drop your results file here
A .json file written by run.py (schema criogen-compare/1). It is read and checked in your browser; nothing is uploaded.
Live comparison
CriogenAI and LangChain on one job, side by side
Send a message through the same agent in both frameworks and compare latency, throughput, CPU and memory. Each framework runs in its own process against one scripted model server, and the measurements are taken from outside by the same code. For exploration on a single machine; the results above are the reference.
Recorded runs
Each model call waits 200 ms and each tool 100 ms, as a hosted model and a database would.
CriogenAI
waitingLangChain
waitingCPU and memory
- CriogenAI
- LangChain
Results
| Measurement | CriogenAI | LangChain | Difference |
|---|---|---|---|
| No results yet. | |||
How the live comparison works
- Model
- One local server speaks the OpenAI wire format and gives both frameworks identical scripted replies. Each framework builds requests and parses responses with its own client. No real model is called.
- Agents
- Each framework's standard tool-calling agent with default settings:
Agentin CriogenAI,create_agentin LangChain. Both use the same synchronous tools with the same names, arguments and delays. - Measurement
- CPU and memory are sampled every 100 ms from outside each process with
psutil, including import time. The model server runs in a separate process and is charged to neither framework. - Order
- The benchmark runs A, B, B, A, each pass in a fresh process, and averages the two passes of each framework.
Known differences and limits
- Defaults differ. CriogenAI streams the reply (
stream: true) and LangChain requests a complete one (stream: false), which adds parsing work on the CriogenAI side. This compares defaults, not feature sets. - With realistic model and tool latency, both spend most of a run waiting, so median run time converges. The differences that remain are memory, CPU and behaviour under concurrent load.
- One machine, one measurement per click. Repeat before drawing conclusions from small differences.
Run it locally
The server binds to 127.0.0.1 and accepts requests only from this site's pages. Set up the two Python environments once, then start it and reload this page; the source switches to Live.
python benches/compare/setup.py --only criogen # once
python benches/compare/setup.py --only langgraph # once
target/compare/venvs/criogen/bin/python playground/server.py # Windows: Scripts/python.exe
# serves on http://127.0.0.1:4600; reload this page