T.A.R.S — Voice-Driven AI Vehicle Diagnostics
A hands-free AI co-pilot for automotive diagnostics: a realtime voice agent reads a vehicle over OBD-II / CAN / UDS, narrates findings, and drives live diagnostics on screen.
- OpenAI Realtime API
- Agents SDK
- TypeScript
- Next.js
- Electron
- Node.js / Express
- Python / FastAPI
- MongoDB
- Docker
- Three.js
- OBD-II / CAN / UDS
- MCP
Demo
Overview
T.A.R.S is a hands-free, voice-driven AI co-pilot for automotive diagnostics. A technician talks to it in plain language — "pull the codes," "watch the Bank 2 oxygen sensor," "map the modules on this truck" — and a realtime voice agent reads the vehicle over OBD-II / CAN / UDS, narrates findings, drives live charts and a 3D digital twin on screen, and works the problem from complaint to evidence to repair.
It ships as a Next.js web app, a cross-platform Electron desktop app that owns the real hardware I/O, and an Expo mobile app for Bluetooth adapters in the field — all backed by a hybrid Node/Express + Python microservice platform. Under the hood it pairs OpenAI's Realtime API (speech-to-speech with tool calling) with a full OBD-II / CAN / UDS hardware stack and a Python CAN sidecar for raw bus capture and signal decoding.
What it does
- Talk to your scan tool — a realtime voice agent runs the whole diagnostic conversation: connect an adapter, scan DTCs, stream live PIDs, run Mode $06 monitor tests, check readiness, look up the vehicle, and write up findings, all by voice.
- Reads real hardware — OBD-II over Bluetooth LE, USB serial (ELM327 / vLinker), Wi-Fi / TCP gateways, plus native CAN via PEAK PCAN, CANable, and SocketCAN.
- Maps the CAN bus — a dedicated agent drives a Python sidecar to passively sniff the bus, enumerate ECUs, decode signals with comma.ai's opendbc databases, reverse-engineer unknown CAN IDs, and render a live 3D module-discovery topology.
- Advanced UDS — diagnostic session control, security access, read/write data-by-identifier, routine control, and GM-specific bidirectional actuator tests (ISO 14229 / ISO 15765).
- Closes the loop — customer and vehicle lookup, parts pricing, estimate drafting, work orders, web research for recalls / TSBs / specs, and calendar scheduling.
System architecture
T.A.R.S is a layered, hybrid system spanning a TypeScript frontend and a Node + Python backend, orchestrated with Docker Compose.
- Realtime agent layer — RealtimeAgent instances (a primary orchestrator, a low-level ELM327/UDS transport agent, a CAN-mapping agent, and a calendar agent) with ~150 Zod-typed tools, mid-conversation handoffs, and output guardrails, all driven over a WebRTC data channel.
- Frontend diagnostic stack — a UnifiedOBD2Service exposes one diagnostic API while the real I/O is Web Bluetooth in the browser, a serial/BLE/TCP bridge in Electron, or the native CAN sidecar; an OBD2ConcurrencyManager serializes the single-threaded link into prioritized lanes.
- Backend API — a Node/Express service (50+ routes) for auth, OBD-II/UDS sessions, AI agents, research, media, and Stripe billing, with Socket.IO plus Server-Sent Events for live telemetry and an OpenAI Realtime relay.
- Python microservices — a FastAPI udsoncan service (ISO-14229), a FastAgent sidecar that runs parallel hypothesis / safety / next-test reasoning lanes, a data-analysis server, and a multi-agent research server.
- Agent tooling — Model Context Protocol servers (sequential-thinking, memory, fetch, Brave search) exposed to the agents through stdio-to-HTTP bridges.
- Data — MongoDB for application data, Redis for cache and realtime, and PostgreSQL + pgvector / Chroma for long-term semantic memory and diagnostic case retrieval.
Engineering highlights
- One agent, many transports — the same tool calls hit Web Bluetooth, an Electron main-process bridge, or a native CAN sidecar; the agent never has to know which.
- Realtime tool calling on live hardware — ~150 typed tools let a speech-to-speech model issue OBD-II / UDS / CAN commands mid-conversation, kept lean enough to apply reliably over the WebRTC data channel.
- Browser-to-vehicle UDS relay — UDS frames travel browser → Node WebSocket → Python udsoncan → ELM327 → vehicle and back, with request tracking and timeout recovery, so even a cheap BLE dongle can run ISO-14229 services.
- Agent / human arbitration — a UDS arbiter mediates control of the diagnostic bus between the AI agent and the human technician so the two never collide.
- Vehicle Health State Model — OBD-II, UDS, CAN, and Mode $06 data fuse into a normalized per-vehicle state vector with predictive drift detection (days-to-failure) and daily temporal decay of older signals.
- Dual OBD-II analysis — a fast, template-filled Python path for production plus a flexible full-code-generation path, both executed in an isolated Docker sandbox.
- ISO-TP everywhere — multi-frame ISO 15765 reassembly and flow control for both the ELM327 text protocol and raw CAN.
Running pro-grade diagnostics on a $5 ELM327
The whole system runs on ~$5 ELM327 Bluetooth adapters that the automotive community widely treats as basic-only. A custom communication layer — dynamic request pacing, flow control, ISO-TP reassembly, buffer management, and retry logic — streams 65+ PIDs simultaneously and runs extended diagnostic services on hardware usually limited to simple Mode 01 reads, work typically reserved for scan tools costing hundreds to thousands of dollars.
Stack
- Frontend — Next.js, React, TypeScript, Electron, Expo, Three.js, uPlot, Redux, OpenAI Realtime + Agents SDK (Zod).
- Backend — Node.js, Express, Socket.IO, MongoDB / Mongoose, Redis, PostgreSQL + pgvector, Chroma, Stripe, Model Context Protocol.
- Python — FastAPI, uvicorn, udsoncan, FastAgent, pyserial.
- Infrastructure — Docker Compose, comma.ai opendbc.
