Define the target domain's ontology: Identify key entities (e.g., 'product', 'customer', 'price') and their relationships relevant to your automation goal. Use or extend existing standards like Schema.org where applicable.
Extract raw web content and metadata: Utilize traditional scraping tools (Playwright/Puppeteer) to gather HTML, CSS, and JavaScript. Also, extract any available semantic annotations (e.g., JSON-LD, Microdata).
Pre-process and normalize data: Clean the extracted data, remove noise, and convert it into a structured format suitable for semantic analysis.
Apply LLMs for semantic interpretation: Use LLMs to analyze raw text and HTML, identifying entities and relationships based on the defined ontology. Prompt LLMs to extract specific semantic roles (e.g., 'identify the product title', 'find the add-to-cart button') even without explicit selectors.
Build a Knowledge Graph representation: Populate a local or remote Knowledge Graph with the semantically extracted information, linking entities and their properties. This graph serves as the agent's internal model of the web page's meaning.
Leverage WebMCP for structured interaction (if available): If the target web application exposes a WebMCP interface, use it to directly query and interact with semantically defined elements, bypassing brittle DOM selectors.
Design adaptive action planning: Based on the Knowledge Graph and LLM reasoning, the agent plans its next action by targeting the meaning of an element (e.g., 'click the purchase button') rather than its specific DOM path. The agent can infer the best element to interact with even if its visual representation changes.
Implement self-correction and feedback loops: Monitor agent execution. If an action fails (e.g., element not found), the agent can re-query its Knowledge Graph or LLM for alternative semantic interpretations or interaction methods, adjusting its plan dynamically.