Bob: AI-Powered Raspberry Pi Rover

Published: July 27, 2026

Bob is an ongoing AI-powered mobile rover that combines a Raspberry Pi, Arduino motor control, computer vision, voice interaction, an animated robot face, and a custom browser dashboard in one modular robotics platform.

I started Bob as a hands-on way to explore how embedded electronics, software, artificial intelligence, and physical hardware can work together. Rather than building only a remote-controlled car or a screen-based chatbot, my goal is to create a robot that can move, see, listen, speak, respond conversationally, and gradually develop a recognisable personality.

Current status: Functional prototype under active development.

---

📸 Preview

---

🎯 Project Vision

Bob is designed as a practical robotics and embedded-systems learning platform rather than a finished consumer robot. Its modular design lets me improve individual systems without rebuilding the entire rover.

• Move around using direct dashboard or controller commands.
• Stream video and react to camera input.
• Recognise a custom "Hey Bob" wake word.
• Accept typed or spoken questions and instructions.
• Reply using conversational AI and local text-to-speech.
• Show expressions and status through an animated face.
• Progress toward safer autonomous movement and obstacle awareness.

---

⚙️ How Bob Works

Bob separates high-level intelligence from direct hardware control. The Raspberry Pi handles the dashboard, camera, audio, AI, and command validation. An Arduino-compatible controller handles the motors and pan-and-tilt servos through a small serial protocol.

Browser, microphone, wake word, or remote message
  -> Raspberry Pi dashboard and deterministic command parser
  -> conversational request OR allow-listed robot command
  -> OpenClaw with local Ollama fallback for conversation
  -> Piper text-to-speech for spoken responses
  -> bounded serial command for Arduino-controlled movement

Conversational model output is never trusted as a direct motor command. Physical actions must pass through deterministic application logic before a short allow-listed command is sent to the Arduino.

---

✅ Current Working Features

Feature	Current Capability
Rover movement	Forward, backward, pivot left, pivot right, stop, and five speed levels through an Arduino serial protocol
Motor safety	Explicit stop command and a one-second Arduino motor inactivity failsafe
Dashboard	Authenticated Flask interface for status, movement, camera, chat, and robot controls
Conversational AI	OpenClaw is the primary route, with local Ollama retained as a fallback
Speech	USB microphone capture, speech recognition, local Piper text-to-speech, and browser speech fallback
Camera	Lazy Picamera2 capture with OpenCV overlays, motion processing, and dashboard streaming
Robot face	Browser-rendered vector face with expressions, gaze, listening, thinking, speaking, and sleep states
Power monitoring	UPS integration detects its I2C fuel-gauge interface

---

🚧 Implemented but Not Fully Active

• The Arduino serial device was disconnected during the latest repository audit, so physical driving was not re-verified at that time.
• Camera and vision support was enabled, but the stream was idle during the audit.
• The custom wake-word system was installed but switched off.
• Game-controller support was configured, but no controller was connected and runtime control was disabled.
• End-to-end speech latency varies with network conditions and assistant response time.

---

🧰 Main Hardware

Component	Role	Status
Raspberry Pi 5	Dashboard, AI, camera, speech, power monitoring, and serial communication	Confirmed
Arduino-compatible AVR board	Bounded motor and servo control	Working role confirmed; exact installed board still being verified
Four DC motors and rover chassis	Physical movement	Confirmed
Motor Shield V1-compatible driver	Controls four motor channels using the AFMotor API	Software interface confirmed; exact hardware revision being verified
Raspberry Pi Camera Module 3	Video, motion, and vision input	Confirmed
Pan-and-tilt servo mount	Moves the camera horizontally and vertically	Confirmed with ongoing reliability work
USB microphone and speaker	Voice input and spoken responses	Confirmed
7-inch HDMI touchscreen	Local dashboard and robot-face display	Confirmed
20,000 mAh power bank and UPS arrangement	Portable Raspberry Pi power	Used; final power architecture still being refined

---

💻 Software and AI System

Technology	Purpose
Python and Flask	Dashboard, API endpoints, hardware bridge, and system status
OpenClaw	Primary conversational gateway and remote integration
Ollama	Local conversational fallback and experimental local vision descriptions
Picamera2 and OpenCV	Camera capture, overlays, motion, and experimental tracking
SpeechRecognition	Speech-to-text with explicit remote-recognition consent
Piper	Local text-to-speech
OpenWakeWord	Custom "Hey Bob" activation experiments
Arduino C++	Real-time motor and servo control
HTML, CSS, and JavaScript	Dashboard and animated robot-face interfaces

---

🎤 Voice and "Hey Bob"

The latest audited wake-word configuration uses a custom OpenWakeWord model with thresholding, cooldown control, and suppression while Bob is speaking. Test mode is deliberately non-actioning: detecting the wake word during a test cannot activate speech, AI, the camera, or the motors.

The working voice path records bounded microphone input, converts speech to text, routes the request to the assistant, cleans and limits the reply, then speaks it through a local Piper voice. Recordings, private datasets, transcripts, and trained model files are not published.

---

👀 Robot Face and Personality

Bob's active face is rendered in a browser and served by the dashboard. It supports idle, awake, listening, thinking, speaking, confused, happy, searching, startled, bored, and sleeping states. Gaze values are clamped to a safe range, temporary expressions return to neutral, and the display enters a sleep presentation after inactivity.

The face is more than decoration: it makes Bob's internal state easier to understand while giving the rover a recognisable personality.

---

🚗 Motor and Camera Protocol

Command	Action
F	Move forward
B	Move backward
L / R	Pivot left or right
S	Stop immediately
Q / E	Move camera pan angle
U / D	Move camera tilt angle
C	Centre the camera
V1 to V5	Select one of five movement speeds

---

🔬 Experimental and Planned Features

• More reliable custom wake-word evaluation in Bob's real acoustic environment.
• Local speech recognition to reduce reliance on external services.
• Improved object detection, face tracking, and vision-language descriptions.
• Ultrasonic sensing and cautious obstacle avoidance.
• Safer autonomous navigation with clear manual override boundaries.
• A more robust power-distribution and battery-monitoring system.
• Improved event-driven coordination between voice, face, camera, and movement.
• Accessibility and kiosk-mode improvements for the local display.

---

⚠️ Safety and Privacy

• Motor tests begin with the wheels raised and an independent power disconnect within reach.
• Natural-language output cannot send arbitrary serial bytes or unbounded movement durations.
• Camera, movement, serial, and administrative controls remain authenticated and private.
• Voice recordings, conversations, biometric images, credentials, and private network information are excluded from the public reference material.
• The source repository remains private while licensing, dashboard credentials, exact wiring, and hardware details receive a final review.

---

📝 Notes

• Bob is a functional prototype, not a finished autonomous robot.
• Some reference firmware improvements have compiled successfully but have not yet been flashed to the live rover.
• Older standalone speech, camera, wake-word, and face experiments are treated as historical work rather than current supported paths.
• This page will be updated as more hardware is confirmed and new features move from experimental to working.

← Back to Projects