Crawl restrictions
Control which GitHub content Glean crawls using repository inclusion and exclusion lists and content-type toggles. Use these rules minimally to preserve the search experience: most end users expect to find all content, and most customers apply no rules or use exclusions sparingly for sensitive repositories.
- Use Greenlist restrictions to allow crawling only for content that you specify.
- Use Redlist restrictions to prevent Glean from crawling content that you specify.
| Restriction Type | Greenlist | Redlist | Details |
|---|---|---|---|
| Time-based Restrictions | ❌ | ❌ | Restrict crawling to include/exclude content created/modified/viewed after a certain date. |
| Identity-based Restrictions | ❌ | ❌ | Restrict crawling to include/exclude content created/modified/viewed by specific users or a specific group (plus public content). |
| Content-based Restrictions | ✅ | ✅ | Restrict crawling to include/exclude specific content, documents, messages, or objects. |
Supported restrictions
| Restriction | Greenlist | Redlist | Details |
|---|---|---|---|
| Repository | ✅ | ✅ | Restrict crawling to include or exclude specific repositories by name. |
| GitHub Pages repository | ✅ | ✅ | Restrict GitHub Pages crawling to include or exclude specific repositories by name. |
How repository lists work
- Inclusion (greenlist): Enter specific repository names and Glean crawls only those repositories. Any repository not listed is skipped.
- Exclusion (redlist): Enter specific repository names and Glean skips those repositories. All other repositories are crawled.
If both rules apply to the same repository, the content is not indexed: the exclusion rule takes priority.
Glean indexes only the repositories granted to the GitHub App installation. If you installed the app on a subset of repositories, only those repositories are eligible for crawling. Repository inclusion and exclusion lists further refine what gets indexed within that granted set.
Content-type controls
In addition to repository-level restrictions, you can toggle crawling for the following content types from the Manage Data tab in the admin console. To use the code file filters, enable Index Code File Body.
| Content type | Default | Description |
|---|---|---|
| GitHub Pages | Off | Crawl HTML and Markdown files from repositories that use the legacy gh-pages branch. If you do not add repository names, Glean crawls legacy Pages for all repositories by default. |
| Wikis | Off | Crawl wiki pages attached to repositories. |
| Issues | On | Crawl issues and issue comments. |
| Pull request diffs | On | Crawl diff content from pull requests. |
| Code file types to index body | Varies | Select or add file extensions, such as .py or .md, whose full file bodies Glean should index. |
| Code files to index body | Varies | Enter exact filenames to index, such as Dockerfile, skill.md, or .gitignore. Glean matches each value with the repository file name after removing its directory path. Enter multiple filenames as comma-separated values. Filename matching is case-insensitive. |
Use Code files to index body for one exact file name, such as .dockerfile. To match every filename ending in .dockerfile, add .dockerfile under Code file types to index body.
Apply restrictions
| Method | Supported | Details |
|---|---|---|
| Admin UI | ✅ | Restrictions can be applied in the Admin UI under the connector settings. |
| Glean Support | ✅ | Restrictions can be applied by Glean support on request. |
Navigate to Admin console → Connectors → GitHub and select the Manage Data tab to configure repository lists and content-type toggles.