Skip to main content

Crawl restrictions

Control which GitHub content Glean crawls using repository inclusion and exclusion lists and content-type toggles. Use these rules minimally to preserve the search experience: most end users expect to find all content, and most customers apply no rules or use exclusions sparingly for sensitive repositories.

Limit access in GitHub first

The most reliable way to control what Glean crawls is to limit what the Glean GitHub App can access. Install the app only in the organizations you want indexed and choose Only select repositories during installation. GitHub enforces that boundary, and you can change the selection anytime from the app's installation settings. See Install the Glean GitHub App.

Use the repository inclusion and exclusion lists below only to fine-tune within the repositories the app can already access.

  • Use Greenlist restrictions to allow crawling only for content that you specify.
  • Use Redlist restrictions to prevent Glean from crawling content that you specify.
Restriction TypeGreenlistRedlistDetails
Time-based Restrictions❌❌Restrict crawling to include/exclude content created/modified/viewed after a certain date.
Identity-based Restrictions❌❌Restrict crawling to include/exclude content created/modified/viewed by specific users or a specific group (plus public content).
Content-based Restrictions✅✅Restrict crawling to include/exclude specific content, documents, messages, or objects.

Supported restrictions​

RestrictionGreenlistRedlistDetails
Repository✅✅Restrict crawling to include or exclude specific repositories by name.
GitHub Pages repository✅✅Restrict GitHub Pages crawling to include or exclude specific repositories by name.

How repository lists work​

  • Inclusion list: Enter specific repository names and Glean crawls only those repositories. Any repository not listed is skipped.
  • Exclusion list: Enter specific repository names and Glean skips those repositories. All other repositories are crawled.

If both rules apply to the same repository, the content is not indexed: the exclusion rule takes priority.

Glean indexes only the repositories granted to the GitHub App installation, so the installation scope is the outer boundary. Inclusion and exclusion lists refine what gets indexed within that granted set and can't add repositories the app wasn't granted.

Cross-organization behavior​

When you connect multiple GitHub organizations to Glean — either through separate connector instances or through a multi-organization connector (GitHub Enterprise Server, GitHub Enterprise Cloud with 50+ organizations) — the repository inclusion list applies globally, not per organization.

Example: Suppose you connect two GitHub organizations, acme-engineering and acme-research. If you add acme-engineering/api-server and acme-engineering/web-app to the repository inclusion list, Glean crawls only those two repositories. Repositories in acme-research are not crawled, even though that organization is connected, because they are not on the inclusion list.

To crawl all repositories from one organization while limiting another:

  1. For separate connector instances (GitHub Cloud): Limit the GitHub App installation in each organization, granting All repositories in the organization you want fully crawled and Only select repositories in the one you want limited. If lists are still needed, configure them independently on each connector instance.
  2. For multi-organization connectors: Add all repositories you want indexed to the inclusion list, including every repository from organizations you want fully crawled. Alternatively, use the exclusion list to block only the specific repositories you want to skip, which allows all other repositories across all connected organizations. On GitHub Enterprise Server, you can also use the organization allow and block lists to drop entire organizations without listing their repositories.

Enforcement for webhook-driven updates​

Glean enforces repository inclusion and exclusion lists when processing webhook events. Content from repositories on the exclusion list (or not on the inclusion list, when one is configured) is not indexed, even when GitHub sends a webhook event for that repository.

Content-type controls​

In addition to repository-level restrictions, you can toggle crawling for the following content types from the Manage Data tab in the Admin console. The issue, wiki, pull request diff, and code file settings are in the tab's Additional Crawl Settings section. To use the code file filters, enable Index Code File Body.

Content typeDefaultDescription
GitHub PagesOffCrawl HTML and Markdown files from repositories that use the legacy gh-pages branch. If you do not add repository names, Glean crawls legacy Pages for all repositories by default.
WikisOffCrawl wiki pages attached to repositories.
IssuesOnCrawl issues and issue comments.
Pull request diffsOnCrawl diff content from pull requests.
Code file types to index bodyVariesSelect or add file extensions, such as .py or .md, whose full file bodies Glean should index.
Code files to index bodyVariesEnter exact filenames to index, such as Dockerfile, skill.md, or .gitignore. Glean matches each value with the repository file name after removing its directory path. Enter multiple filenames as comma-separated values. Filename matching is case-insensitive.

Use Code files to index body for one exact file name, such as .dockerfile. To match every filename ending in .dockerfile, add .dockerfile under Code file types to index body.

Apply restrictions​

MethodSupportedDetails
Admin UI✅Restrictions can be applied in the Admin UI under the connector settings.
Glean Support✅Restrictions can be applied by Glean support on request.

Navigate to Admin console → Connectors → GitHub and select the Manage Data tab to configure repository lists and content-type toggles.