Sitemap
Open in app

Sign in

Medium Logo
Write
Search

Sign in

Building an AI Agent Mesh with Gemini 3, OpenClaw, and ACPX

Timothy

--

Press enter or click to view image in full size

I texted a Telegram bot: “Build me a dashboard.” Behind the scenes, an AI gateway powered by Google’s Gemini 3.1 Pro evaluated the request, selected the most suitable coding agent, and spawned a headless Codex session through the Agent Client Protocol. Codex wrote the code, captured Playwright screenshots, and delivered the results directly to my Telegram chat. I monitored the entire pipeline in real time from a split-pane terminal on my laptop.

I built this system over a weekend on a Google Cloud Platform virtual machine, and by the end of it, I was holding multi-turn conversations with Codex through a persistent session while observing the full orchestration pipeline through Cmux.

The Problem: Agents Don’t Talk to Each Other

Every coding agent available today operates as a standalone command-line tool:

codex "create a hello world script"
claude -p "refactor this function"
gemini "explain this error"

Each invocation is stateless. There is no mechanism to route a task from one agent to another based on the agent’s strengths. There is no way to maintain multi-turn context across prompts. You cannot orchestrate agents from a chat interface like Telegram or Discord. And there is no visibility into what agents are doing in real time.

What we need is a protocol that agents can speak, a gateway that routes requests between them, and a terminal environment that provides full observability.

The First Thing That Worked: Telegram to Codex in Two Minutes

I sent this message to my Telegram bot:

Hey Bee, spawn a Codex session and create a Python script at /tmp/fibonacci.py that prints the first 10 Fibonacci numbers, then run it and show me the output.

Within two minutes, the bot responded with the complete output: 0 1 1 2 3 5 8 13 21 34.

Here is what happened under the hood. The message hit OpenClaw, which handed it to Gemini 3.1 Pro. Gemini read its routing skill document, classified the request as a coding task, and triggered the ACPX plugin with agentId: "codex". ACPX spawned a headless Codex instance through the Agent Client Protocol, the JSON-RPC 2.0 standard created by Zed Industries for structured agent communication. Codex created the file, executed it, captured the output. The result flowed back through ACPX to OpenClaw, where Gemini formatted a human-readable response and delivered it to Telegram.

One text message in. Working code and output back. No SSH. No terminal. Just a chat.

Press enter or click to view image in full size

Persistent Sessions: Picking Up Where You Left Off

That first demo was fire-and-forget. Send a prompt, get a result, session dies. But real development is iterative. You explore, modify, test, refine. You need the agent to remember what it already did.

ACPX supports persistent sessions that maintain full context across multiple prompts:

# Create a persistent session
acpx codex sessions new --name workspace
# First prompt: explore the project
acpx codex -s workspace "list the files in /tmp/workspace-assistant/"# Second prompt: Codex remembers the context from the first
acpx codex -s workspace "add a Pi agent card to main.ts"# Third prompt: builds on all previous work
acpx codex -s workspace "take a Playwright screenshot of the updated dashboard"# Close the session when finished
acpx codex sessions close workspace

I tested this workflow live. I created a session named workspace, asked Codex to modify a Vite dashboard, then asked it to capture Playwright screenshots. All three prompts executed with full context retention. Codex remembered which files existed, what changes it had made, and could build on its previous work without re-explanation.

Press enter or click to view image in full size

The Screenshot That Sold Me

This was the moment the system stopped feeling like a prototype.

I asked Codex, through Telegram, to capture Playwright screenshots of the Vite dashboard running on the virtual machine. The workflow:

  1. I sent a message in Telegram requesting screenshots
  2. Gemini 3 routed the request to Codex through ACPX
  3. Codex installed Playwright and launched headless Chromium
  4. Codex navigated to localhost:4173, captured the light mode view
  5. Codex toggled dark mode through DOM manipulation, captured the dark mode view
  6. Both screenshots were uploaded directly to Telegram through the Bot API

Actual rendered images of a running dashboard, captured by a coding agent, delivered to a chat thread. Not text. Not logs. Screenshots.

Press enter or click to view image in full size

Making It Interactive: acpx-chat.sh

Persistent sessions are powerful, but the ergonomics are rough. Every prompt requires the full command:

acpx codex -s workspace "your prompt here"

So I built acpx-chat.sh, a readline-based wrapper that transforms ACPX into an interactive chat. Here is an actual session transcript:

View the full acpx-chat.sh source on GitHub Gist

$ ./acpx-chat.sh codex workspace
acpx-chat -- interactive ACP session
agent: codex  session: workspace
Type your prompt and press Enter. Ctrl+D or 'exit' to quit.
you> What files did we work on today?[acpx] session workspace (019cd33a...) -- agent connected
Today we worked on:
- /tmp/workspace-assistant/src/main.ts
- /tmp/workspace-updated.pngyou> What changes did we make to main.ts?[acpx] session workspace (019cd33a...) -- agent connected
We added one new entry to the agents array:
  { name: 'Pi', status: 'idle', protocol: 'ACP', lastActive: '30 min ago', tasks: 0 }
That makes the dashboard render a new Pi agent card.you> /help
Commands:
  /status   -- show session status
  /history  -- show recent turn history
  /close    -- close session and exit
  /help     -- show this help
  exit      -- exit (session stays alive)

Type a prompt, press Enter, get a response with full context retention. The wrapper supports any ACPX-compatible agent:

./acpx-chat.sh codex my-session    # Interact with Codex
./acpx-chat.sh claude my-session   # Interact with Claude Code
./acpx-chat.sh gemini my-session   # Interact with Gemini CLI

The ACPX team is actively designing a built-in solution. Their warm session owner architecture document describes a planned detached daemon model that would keep sessions warm in the background. In the meantime, acpx-chat.sh is available in the repository.

How the Stack Works

Here is what made those demos possible. Three layers, each doing one job.

OpenClaw is the gateway. It runs as a systemd service on a server, listens for messages from Telegram (or Discord, or Slack), and routes them to the right handler. When a message arrives, OpenClaw creates a session and hands the message to its configured model. The ACPX plugin ships bundled at extensions/acpx/, giving OpenClaw native support for spawning coding agents.

Gemini 3.1 Pro is the routing brain. It reads every incoming message, evaluates intent, and decides what to do. For coding tasks, it selects the best-suited agent. For general questions, it responds directly. What makes Gemini 3 effective here is its ability to internalize the acp-router skill, a multi-page routing specification describing all available agents, their capabilities, and the decision logic for selecting one. Gemini reads it once and follows it accurately across hundreds of requests.

ACPX is the protocol client. Think of it as the curl of the agent world. It speaks the Agent Client Protocol to coding agents through thin adapter wrappers. Codex is the primary agent in these demos, but ACPX ships with adapters for Claude Code, Gemini CLI, OpenCode, Pi, and Kimi. Custom agents can be registered in ~/.acpx/config.json. See the Technical Appendix for the full adapter table.

The practical flow: a user sends “build me a Fibonacci script” in Telegram. OpenClaw receives it. Gemini reads the acp-router skill and classifies it as a coding task. It triggers the ACPX plugin with agentId: "codex". ACPX spawns a headless Codex instance through its ACP adapter. Codex writes the script, executes it, captures the output. The output flows back through ACPX to OpenClaw, where Gemini formats a response and delivers it to Telegram. Under two minutes, end to end.

OpenClaw supports two routing paths: an ACP Runtime path (preferred) that creates thread-bound sessions managed by OpenClaw’s session lifecycle, and a Direct ACPX fallback that executes acpx codex exec directly when the runtime path encounters an error. The skill includes built-in recovery logic for automatic failover.

Press enter or click to view image in full size

Real-Time Monitoring with Cmux

Running agents headless on a remote VM is powerful until something goes wrong and you have no visibility. Cmux solved this.

I configured a monitoring workspace with three panes:

# Create a split pane for ACPX monitoring
cmux new-split right --json  # returns surface:N
# SSH to the VM and tail gateway logs filtered for ACPX events
cmux send --surface surface:N \
  'gcloud compute ssh codex-dev-qa --zone=us-central1-a --project=belha-ai'
cmux send-key --surface surface:N Enter

The most powerful feature for agent orchestration is cmux read-screen, which reads terminal output from any surface:

# Read the ACPX pane output
cmux read-screen --surface surface:81 --lines 30
# Read with scrollback history
cmux read-screen --surface surface:81 --scrollback --lines 50

This enabled coordination between Claude Code (composing this article on my laptop) and the ACPX pane (executing Codex sessions on the VM). One agent observing and responding to another agent’s output in real time. Cmux also delivers macOS notifications when agents complete tasks, so you do not have to stare at the terminal.

Lessons Learned

  1. Headless permission mode is non-negotiable. Setting permissionMode: "approve-all" is required. Without it, the agent hangs waiting for human approval on every file operation and the pipeline freezes.
  2. Persistent sessions are not interactive REPLs. Each acpx codex -s workspace "prompt" is a separate CLI execution. Session state persists server-side, not client-side. That is why acpx-chat.sh was necessary.
  3. Event stream files are the best debugging tool. The NDJSON files at ~/.acpx/sessions/*.stream.ndjson contain every JSON-RPC message exchanged. More reliable than gateway logs for diagnosing issues.
  4. Gemini 3’s context window enables reliable routing. The acp-router skill is a multi-page specification. Models with smaller context windows cannot execute it consistently. Gemini 3.1 Pro reads it once and follows it accurately.
  5. Cmux transforms agent monitoring. The ability to read terminal output from one pane and relay it to another (cmux read-screen) is the foundation of multi-agent coordination. Notification rings provide immediate awareness of which agent needs attention.

The Broader Perspective

What this project demonstrates is a working prototype of the AI agent mesh: a system where agents communicate through a structured protocol, orchestrated by an intelligent routing layer.

The architecture comprises three layers:

  1. Protocol layer (Agent Client Protocol): A model-agnostic standard for agent communication. JSON-RPC 2.0 over stdio. Simple, debuggable, composable.
  2. Client layer (ACPX): The universal client. Manages sessions, prompt queues, and permissions. Agent-agnostic by design.
  3. Orchestration layer (OpenClaw + Gemini 3): The gateway receives messages from chat platforms. The routing brain reads skill documents, evaluates intent, selects agents, manages sessions, and formats responses. Together, they transform a Telegram message into a multi-agent workflow.

As adoption of the Agent Client Protocol grows, the possibilities expand: intelligent routing where Gemini selects the optimal agent for each task, agent chaining where Codex writes code and Claude reviews it, persistent workspaces accessible from any chat platform, and full pipeline observability through Cmux.

Try It Yourself

# Install OpenClaw and ACPX
npm i -g openclaw
# Start the gateway
openclaw init
openclaw gateway start# Or skip the gateway and use ACPX directly
acpx --approve-all codex exec "create a hello world script and run it"# Or start an interactive chat session
./acpx-chat.sh codex my-session

For the full setup (Gemini configuration, Telegram bot connection, ACPX plugin settings, and topic bindings) see the Technical Appendix.

The complete setup guide, acpx-chat.sh, the demo video, and all screenshots are available in the GitHub repository.

References

Appendix: Technical Reference

OpenClaw Gateway Config (ACPX Plugin)

{
  "plugins": {
    "entries": {
      "acpx": {
        "enabled": true,
        "config": {
          "permissionMode": "approve-all",
          "command": "/home/tim/npm/lib/node_modules/openclaw/extensions/acpx/node_modules/.bin/acpx"
        }
      }
    }
  }
}

permissionMode: "approve-all" is required for headless operation. Without it, agents hang waiting for human approval on every file write and shell command.

ACPX Client Config (~/.acpx/config.json)

{
  "defaultPermissions": "approve-all",
  "agents": {
    "openclaw": {
      "command": "openclaw acp --url ws://127.0.0.1:8080 --token-file /home/tim/.openclaw/gateway-token"
    }
  }
}

Agent Adapters

Codex: npx @zed-industries/codex-acp (Primary coding agent)

Claude Code: npx -y @zed-industries/claude-agent-acp

Gemini CLI: gemini (Native ACP support)

OpenCode: npx -y opencode-ai acp

Pi: npx pi-acp

Kimi: kimi acp

ACP Protocol Methods

initialize (Handshake and capability negotiation), session/new (Create a new execution session), session/load (Resume a prior session), session/prompt (Send a prompt to the agent), session/update (Stream output), session/cancel (Cancel an in-flight prompt), fs/read_text_file / fs/write_text_file (Client-side file I/O), terminal/create / terminal/output / terminal/kill (Terminal process management)

Transport: JSON-RPC 2.0 over stdio | SDKs: TypeScript, Python, Rust, Kotlin | License: Apache 2.0

Session Internals

Sessions are stored in ~/.acpx/sessions/. Each session produces a metadata JSON file and an NDJSON event stream containing every JSON-RPC message exchanged.

Queue owner model: The active acpx process holds the queue. Additional invocations enqueue prompts via Unix socket. Prompts process sequentially. The owner stays alive for a configurable TTL (default 300s; --ttl 0 for indefinite).

Bindings Config (Auto-Routing Telegram Topics)

{
  "bindings": [
    {
      "type": "acp",
      "agentId": "codex",
      "match": {
        "channel": "telegram",
        "accountId": "default",
        "peer": { "kind": "group", "id": "-1003305133020:topic:8" }
      },
      "acp": { "mode": "persistent", "label": "codex-main" }
    }
  ]
}

This binds a Telegram forum topic directly to a persistent Codex session. Every message in that topic routes to the agent without requiring an explicit routing prompt.

--

--

Timothy
Timothy

Written by Timothy

920 followers
690 following

Software / DevOps Engineer | Google Developer Expert for Cloud | https://timtech4u.dev/