# susanhub.com — Sustainability Academic Network # # The public directories (datasets, jobs, conferences, events, grants, # institutes, researchers, research-papers) are all crawlable. Everything # disallowed below is either signed-in-only, a redirect target, or a legacy # duplicate, so crawling it spends crawl budget and returns nothing indexable. # # One group for every crawler, AI assistants' included (OAI-SearchBot, # ChatGPT-User, PerplexityBot, ClaudeBot). Do not add a group that names one: # a crawler obeys only the group that names it, so it would drop every # Disallow below. scripts/robots_txt.test.mjs checks this. User-agent: * Allow: / Disallow: /admin Disallow: /profile Disallow: /settings Disallow: /notifications Disallow: /onboarding Disallow: /action Disallow: /logout Disallow: /unsubscribe Disallow: /open-data-access Disallow: /session Disallow: /403 # /auth (the login page) is deliberately NOT disallowed. While it was, Google # could not read the page, so it listed the bare URL from the "Log in" link on # every page, and for searches of the site's name it showed the login page # (Nov 2026: 330 of 430 impressions). It is kept out of the index instead with # an `X-Robots-Tag: noindex` header in firebase.json, which a crawler can only # see if it may fetch the page. # Superseded by /discussions; indexing it would compete with the live surface. Disallow: /discussions_old # /share/** is deliberately NOT disallowed. That route exists so link-preview # crawlers can read Open Graph tags before the visitor is bounced to the real # page, and the in-app Share dialog hands those URLs straight to Facebook, # X and LinkedIn. facebookexternalhit, Twitterbot and LinkedInBot all honour # robots.txt, so a Disallow here would stop the fetch — not the indexing — and # every share card would render as a bare link. Keeping /share/** out of the # search index is done with an `X-Robots-Tag: noindex` response header in # firebase.json, which the previewers ignore and search engines obey. Sitemap: https://susanhub.com/sitemap.xml