Building a Self-Healing Mobile Automation Engine with LangGraph.js, Ollama, and Appium

In mobile DevOps, the "Flaky Test" is the ultimate productivity killer. Traditional automation is brittle because it relies on a static UI tree. If a developer changes a testID or a UI element shifts slightly, the entire pipeline grinds to a halt.

Agentic Appium TS changes the paradigm. By combining LangGraph.js for orchestration and Ollama (Llava) for local computer vision, we can build an automation agent that doesn't just execute commands—it reasons through UI changes.

🔗 GitHub Repository: SuneetMalhotra/Agentic-Appium-TS

---

The Tech Stack: Local AI Meets Mobile Automation

• Appium 2.x: The backbone for cross-platform (iOS/Android) interaction.
• LangGraph.js: A framework for building stateful, multi-actor applications with LLMs. It manages our "Observer → Reasoner → Executor" loop.
• Ollama + Llava: We use the llava (Large Language-and-Vision Assistant) model running locally to perform Visual Element Grounding.
• TypeScript: Provides a type-safe interface for driver implementations and state management.

---

1. The "Observer-Reasoner-Executor" Loop

Unlike a standard linear script, this framework uses a StateGraph. This allows the agent to retry, backtrack, or pivot if the UI doesn't respond as expected.

The Reasoner node is the "brain." It receives the current screenshot and the UI tree (XML), then determines the next logical step.

// Example State Definition
export interface AgentState {
  driver: IMobileDriver;
  goal: string;
  history: string[];
  lastScreenshot?: string;
  isComplete: boolean;
  healingEvents: number; // Tracks self-healing instances
}

---

2. Technical Deep Dive: Vision Fallback with Llava

The core innovation is the Vision AI fallback. When a standard accessibilityId or XPath fails, the framework triggers a call to a local Ollama instance.

By using the llava model, the framework performs a visual scan of the app's interface. It asks the model to describe the screen and find coordinates for specific elements based on their visual appearance rather than their metadata.

Why Llava?

• Local Execution: No data leaves your machine.
• Multimodal: It processes both the screenshot and the text-based prompt simultaneously.
• Cost-Effective: Zero API costs compared to GPT-4o or Claude 3.5 Sonnet.

---

3. Hybrid Locator: The Best of Both Worlds

We don't rely solely on AI because LLMs can be slower than native selectors. Instead, we implement a tiered priority system:

When the Vision AI successfully finds an element that the selectors missed, it logs a Healing Event. This provides a roadmap for developers to fix the underlying technical debt in the app's accessibility labels.

---

4. Getting Started

Prerequisites

1. Ollama: ollama pull llava
2. Appium: appium driver install xcuitest (for iOS)
3. Node.js 18+

Quick Run

import { createDriver, createIOSSimulatorConfig } from "./dist/drivers/index.js";
import { runGraph } from "./dist/graph.js";

const config = createIOSSimulatorConfig("iPhone 16 Pro", "18.2", "UDID", "com.apple.Preferences");
const driver = createDriver("ios", config);

await driver.connect();

// The Agentic Loop begins here
const result = await runGraph(
  { driver }, 
  "Navigate to Display & Brightness settings"
);

console.log(`Success: ${result.isComplete} | Healed: ${result.healingEvents}`);

---

Why This Matters

By moving from scripts to agents, we reduce the maintenance burden of automation. If a button moves from the left to the right side of the screen, the Vision AI finds it automatically, keeps the test green, and alerts you that the selector needs updating.

Check out the full framework, star the repo, and let's build more resilient mobile tests!

👉 Repo: Agentic-Appium-TS

---

Would you like me to create a "Technical Documentation" version of this for your GitHub Wiki, or perhaps an "Example Scenario" walk-through for a specific app like Settings or Safari?