Date: 2026-09-23
Site: https://mskazemi.com/idkmesh/
This matrix records the discovery path IDKMesh should keep healthy for the major search and answer engines. It is an engineering checklist, not a guarantee of ranking or citation. Search systems decide independently what to crawl, index, rank, summarize, or cite.
| Surface | Primary discovery path to keep healthy | Repository-controlled support |
|---|---|---|
| Google Search | Googlebot, crawlable HTML, canonical URLs, internal links, XML sitemap | static HTML/Markdown, canonical metadata, topic hub, complete sitemap |
| Gemini / Google AI search experiences | Google Search index plus Google’s Gemini-related crawling controls | same Google Search foundation; no separate keyword-stuffing path |
| Bing / Microsoft Copilot | Bingbot, XML sitemap, IndexNow freshness | sitemap + scheduled IndexNow notifier |
| Yahoo Search / Yahoo Scout discovery | Yahoo Slurp plus Bing-supplied search infrastructure; open-web crawl/index signals |
explicit Slurp robots/HTTP probe, Bing/IndexNow coverage, sitemap, canonicals |
| ChatGPT search | OAI-SearchBot plus public crawlable/indexable pages |
static pages, direct answers, topic hubs, llms.txt supplement |
| Claude web search | Claude-SearchBot; user-directed retrieval may use Claude-User |
public static content and answer-oriented topic pages |
| Perplexity | PerplexityBot; user-directed retrieval may use Perplexity-User |
public static content, topic/Q&A structure, sitemap |
| DuckDuckGo | DuckDuckBot plus downstream index/search sources |
explicit DuckDuckBot robots/HTTP probe, crawlable static HTML, sitemap |
| Apple Search / Siri / Spotlight | Applebot |
explicit Applebot robots/HTTP probe, semantic static HTML, metadata, sitemap |
| Brave Search / AI Answers | Brave intentionally does not advertise a differentiated crawler user agent; Googlebot crawlability is a prerequisite | Googlebot/robots/noindex health plus crawlable static HTML and sitemap |
| Other search/answer engines | open-web standards | semantic HTML, crawlable links, canonical URLs, sitemap, evidence-linked content |
The public discovery architecture deliberately separates three layers:
config/seo-topics-v1.json;docs/topics/;The machine-readable query list is for measurement and maintenance. It is not published as a meta-keyword dump.
The installable Python package is a separate discovery surface from the full
research repository. Its metadata in pyproject.toml therefore follows a
narrower truth boundary:
This makes package-index discovery reinforce the same semantic graph without turning repository ambitions into package capability claims.
After deployment, tools/check_public_discovery.py should be able to verify:
robots.txt is reachable;403, 429, or crawler-specific decoy page;/idkmesh/idkmesh/....The workflow is a visibility regression monitor, not a ranking monitor. It
runs from GitHub’s Pages-native page_build event and also has scheduled/manual
recovery paths. On a Pages event it requires a built status and binds the
checkout/probe to that event’s exact commit. A green run means the public site
is technically retrievable from a GitHub-hosted network path; it does not mean
an engine has indexed or cited it.
The monitor uses representative current user-agent identities for:
GooglebotbingbotSlurpDuckDuckBotApplebotOAI-SearchBotClaude-SearchBotClaude-UserPerplexityBotPerplexity-UserCrawler identities can change. Vendor documentation is the source of truth; do not freeze a user-agent string as a permanent protocol guarantee.
llms.txt is not needed for Google Search and does not positively or negatively affect Google visibility/rankings; IDKMesh keeps it only as a supplement for systems that choose to use it.Slurp) and Bing-powered search-result guidance:
https://help.yahoo.com/kb/SLN22600.html
https://help.yahoo.com/kb/SLN2245.htmlCrawler eligibility is only the input. Actual visibility is recorded separately
under the versioned evidence contract in
schemas/search-visibility-observation-v0.1.schema.json.
The canonical ledger is
evidence/search-visibility/observations.json; its deterministic report is
evidence/search-visibility/REPORT.md. This prevents a manual AI-answer check,
a Search Console impression, an index inspection, and a Bing citation from being
collapsed into one misleading score.
For the Microsoft ecosystem, Bing Webmaster Tools now exposes first-party AI
Performance data for citations across Copilot, Bing AI-generated answers, and
selected partners. Its 2026 preview includes cited URLs and grounding queries,
and later preview additions include intents, topics, citation share, and compare.
Those observations should be recorded as webmaster_export evidence rather
than reconstructed by scraping result pages.
Repository evidence location: https://github.com/MSKazemi/idkmesh/tree/main/evidence/search-visibility
Technical crawl eligibility cannot manufacture authority. Competitive visibility still depends on useful content, independent references/backlinks, real users, repository adoption, reproducible research, and observed engagement. Track those outcomes in issue #665 rather than asserting them from configuration.