https://youtu.be/5OJ5t5Nk0wQ Agency clients often ask a question legacy crawlers cannot answer: "We rank on Google, so why does ChatGPT never mention us?" Often, the blocker is invisible: robots.txt allows AI crawlers, but a CDN or firewall silently blocks them. Traditional crawlers only inspect the file directives; they do not test real-time AI user-agent responses. I built Scrawly to bridge that gap: a free, open-source (MIT License, v1.0.0) desktop crawler for macOS, Windows, and Linux. It audits the technical signals governing Google rankings alongside visibility in AI engines like ChatGPT, Claude, Perplexity, and Google AI Overviews. There are no subscriptions, account walls, or URL caps—all audit data remains stored in a local SQLite database on your machine. Scrawly is a local-first, open-source technical SEO crawler and audit platform engineered without artificial URL limits or licensing paywalls. Built from the ground up as a robust free screaming frog alternative software, Scrawly provides search professionals and technical architects with enterprise-grade crawl diagnostics, JavaScript rendering evaluation, and site architecture mapping right on their local machine. Beyond replacing traditional desktop crawlers, it functions as a modern, free Sitebulb alternative, pairing comprehensive technical issue categorization with AI-readiness auditing and native Model Context Protocol (MCP) integration. For teams seeking a transparent, community-driven screaming frog alternative, Scrawly delivers full crawl data sovereignty without the enterprise subscription tag. 250 Technical Checks Across 23 Categories Scrawly evaluates 250 distinct parameters using Playwright Chromium rendering: - Crawlability & Indexability (15): Robots blocks, noindex tags/headers, orphan pages, click depth > 4. - Status Codes & Redirects (14): 4xx/5xx responses, soft 404s, broken links, redirect loops/chains, 302 vs 301. - Canonicals (11): Missing, relative, conflicting, or dynamic post-render canonical shifts. - Titles, Meta & Headings (21): Pixel-width truncation, thin descriptions, duplicate H1s, Open Graph. - Content Quality (12): Exact/near duplicates, keyword cannibalization, placeholder copy, missing dates. - Internal Architecture (12): Link equity distribution, single-inlink pages, generic anchor text. - External Links (6): Broken outbound links, insecure HTTP, missing rel="sponsored" or ugc. - Images & Media (11): Missing alt text, files > 100 KB, missing dimensions, WebP/AVIF adoption. - Hreflang (8): Missing return tags, invalid language codes, x-default, header/sitemap conflicts. - Structured Data (15): JSON-LD validation, @id conflicts, Organization, Article, Product, and FAQ schema. - XML Sitemaps (10): Format errors, non-200 URLs listed, file size limits, stale lastmod. - Robots Directives (7): Site-wide disallows, blocked render assets, wildcard errors, high crawl delays. - Security (10): Mixed content, HSTS configuration, expiring SSL certs, server version leaks. - Core Web Vitals & Speed (14): LCP (>2.5s), INP (>200ms), CLS (>0.1), TTFB, DOM node size. - Mobile (7): Viewport setup, tap targets, horizontal scrolling, desktop/mobile parity. - JavaScript Rendering (6): DOM drift, client-only links, post-render metadata alterations. - Accessibility (9): WCAG elements impacting SEO and AI agents (ARIA, landmarks, color contrast). - AI Search & GEO (13): Silent bot blocking, client-side content barriers, entity markup, answer structure. - Agentic Web & llms.txt (13): /llms.txt quality (<800 links; 20–50 ideal), agent navigation discovery. - URL Hygiene (9): Uppercase characters, unsafe strings, tracking parameters, URLs > 115 chars. - Pagination (5): Page 2+ noindex/canonical traps, infinite scroll URL accessibility. - WordPress & HTML (17): Thin tag archives, staging leaks, dual Yoast/Rank Math conflicts, DOCTYPE. - Analytics & Config (5): GA4/GTM tagging, Search Console verification, favicon, 404 templates.

Full recording of this week's State Change Mastermind livestream with Ray Deck and Dimitris Goudis. Dimitris showed BrainGraph's digital brains at work on his own marketing: a Rory Sutherland brain auditing his website and writing three landing pages. Robert showed how he's handing video editing and animation to AI: code-built set pieces, reusable scenes, and staying on the AI until the result is right. 00:00 Intro 01:10 Dimitris: knowledge bases that help AI tools think like a human 04:24 Applying Rory Sutherland's thinking to his own marketing copy 05:03 Auditing his website with a local BrainGraph digital brain 06:08 The digital brain interviews him first, then writes three landing pages 07:05 Inconsistent calls to action, and why one next step matters 07:35 Why BrainGraph starts with a one-to-one conversation, not a sign-up 08:16 Next: ad copy from the same AI advisor 13:14 A personal Rory that knows things about you 16:25 Automating notifications with Knock inside BrainGraph 16:52 Robert asks about MCP UI for BrainGraph 18:38 Every recommendation links to its source 19:01 Updating a digital brain with new knowledge 20:53 Can it copy someone's communication style? 22:04 Robert: handing editing and animation to AI 23:53 Everything you see is code: UI recreations as set pieces 26:10 Extending a video without regenerating it 26:54 ElevenLabs voiceovers for client training videos 29:10 Reusable sets for Xano, Claude Code and ChatGPT 30:14 Clients validated it first 33:07 Getting the AI to pick the right skill, then pre-made scenes 39:18 Building it: hours upon hours of direction 41:52 "I'm gonna be on you until I don't have to be" 44:19 Finding a style outside Anthropic's 45:56 Nano Banana: one image with 30 icons, then chop it up 47:34 "Why are we doing this anymore?" 48:26 Wrap-up

Full recording of this week's State Change Mastermind livestream with Ray Deck and Dimitris Goudis. We talked about MCP connectors as the new interface for applications, turning experts' videos into knowledge-graph MCPs, and controlling a Mac from the Claude app. 00:00 Intro 01:11 Dimitris: MCP connectors are the new user interface (ClickUp to Airtable in one prompt) 02:52 Expert knowledge bases as public MCP connectors (Caleb Ralston, Rory Sutherland) 05:23 Live demo: adding the Rory Sutherland connector in Claude and asking it how to promote the livestream 16:33 How it was built: 3-4 hours, the MCP shaped around the database 19:34 Robert: notes from the MCP Dev Summit in Toronto (auth, governance, stateless and async) 22:35 Live demo: controlling my Mac from the Claude app by voice through the Snappy OS connector 28:33 How notifications get back to you (MCP tasks, the Claude app) 32:27 Does it only work on a Mac? 34:55 Experts and machines as connectors, and bespoke vs off-the-shelf MCPs 38:11 Getting things done vs gold-plating auth 41:00 Dimitris: lessons from designing the ontology for a custom knowledge graph 46:46 Wrap-up

Writing the code is getting faster. Keeping the agent focused on the right problem is becoming the harder part. My setup lately is claude.ai for working through the codebase, github.com for the actual code/history, and floment.ai for keeping the project, tasks, and next steps clear outside the agent. Feels like agentic development needs good project context almost as much as good prompts. How are you guys keeping agents aligned on longer builds?

Speed and ownability have emerged as the defining promises of modern software development, particularly in the no-code and low-code spaces. These twin commitments address clients' most pressing concerns: the need to move quickly in competitive markets and the desire to truly own their technology assets. Speed addresses the existential business concern of opportunity cost and market timing. Clients often have a limited window to capitalize on market opportunities, and delays can mean the difference between success and failure. Speed is about collapsing the timeline from concept to functional deployment. It means reducing friction throughout the entire development process, from initial meetings to final handoff. Ownability refers to the client's ability to understand, maintain, and extend the software without continuous dependency on the original developer. It manifests in clean, documented code, logical data structures, and transparent systems. Clients who feel able to understand their systems can make bolder business decisions. They value not just the functionality but the fact that they can explain it to others—the system becomes a genuine business asset rather than a black box they fear to touch. By focusing on these promises, developers can create significantly more value for their clients. You won't just deliver functional software—you will deliver business transformation through systems that launch quickly and can be confidently owned by the client.