Crawl restrictions
Control which GitHub content Glean crawls using repository inclusion and exclusion lists. Use these rules minimally to preserve the search experience: most end users expect to find all content, and most customers apply no rules or use exclusions sparingly for sensitive repositories.
- Use Greenlist restrictions to allow crawling only for content that you specify.
- Use Redlist restrictions to prevent Glean from crawling content that you specify.
| Restriction Type | Greenlist | Redlist | Details |
|---|---|---|---|
| Time-based Restrictions | ❌ | ❌ | Restrict crawling to include/exclude content created/modified/viewed after a certain date. |
| Identity-based Restrictions | ❌ | ❌ | Restrict crawling to include/exclude content created/modified/viewed by specific users or a specific group (plus public content). |
| Content-based Restrictions | ✅ | ✅ | Restrict crawling to include/exclude specific content, documents, messages, or objects. |
How repository lists work
- Inclusion (greenlist): Enter specific repository names and Glean crawls only those repositories. Any repository not listed is skipped.
- Exclusion (redlist): Enter specific repository names and Glean skips those repositories. All other repositories are crawled.
If both rules apply to the same repository, the content is not indexed: the exclusion rule takes priority.
The Glean GitHub App is always installed against all repositories in the organization; the
installation itself cannot be scoped to a subset. Repository inclusion and exclusion lists
(allowedRepositories and blockedRepositories) are how you control what gets indexed.
How to apply restrictions
| Method | Supported | Details |
|---|---|---|
| Admin UI | ❌ | Restrictions can be applied in the Admin UI under the connector settings. |
| Glean Support | ✅ | Restrictions can be applied by Glean support on request. |
Contact your Glean representative to configure repository inclusion or exclusion lists, including excluding GitHub Pages sites.