Crawler and indexing size limits

note

This page covers indexing size constraints, not crawl freshness or timing. To monitor crawl activity in your deployment, see Managing connectors. For default crawl frequencies, see Crawling strategy.

Introduction

When Glean is crawling an organization’s corpus using the configured connectors, the crawler pulls down the following information that is then fed to the indexer:

• Item content (for example, title, body, comments, media)
• Metadata (for example, created by, created time, updated time, type, facets, folder)
• Permissions (for example, who is allowed to view the item)

Of the three types of data indexed, the first one (Item content) must take into consideration file size.

Item content indexing limit

Before the crawler downloads the item content to be fed to the indexer, it will first ascertain the content item’s size. If the size is GREATER than 64 MB, the crawler WILL NOT download the item contents and will only download the metadata and permissions information for indexing. If the file size is LESS than 64 MB, the item content will also be downloaded and fed to the indexer.

Q&A

Yes, the item is still searchable with Glean based upon the metadata and it will still be ranked and secured based upon permissions and the personalization features of the Glean Platform.

Yes, the crawler applies this logic for all connectors.

Not by default. By default the crawler will do the following:

1. Download the content
2. Convert the content from its RAW state (PDF, Word, Google Doc, etc…) to a text based indexable format. During this process multimedia content will be removed (Images, Videos, etc…). This process has been proven to drastically reduce document size while preserving the quality of the indexed content. (95%+ reduction in PDFs!).
3. If the content (converted or not) is less than 16.875 MB, the content is stored and fed to the indexer. If the content is greater than 16.875 MB, then the first 16.875 MB of content will be stored and fed to the indexer. The rest of the content will be cut from indexing.

It is possible to turn on OCR (Optical Character Recognition) for certain connectors. If OCR is enabled, then the crawler WILL NOT strip the multimedia content and will attempt to extract text from it. Once this process is complete, then the default crawling/indexing process continues as described above (content conversion, total text indexable size of 16.875 MB, etc…)

note

Even with OCR enabled, the initial size limit of 64 MB still applies as described in the Item content indexing limit section.

Yes. Because the content of the document(s) in question will have not been indexed, Glean is unable to utilize their content as part of its standard functionality. This also means that you will not see the “Summarize” button in the Search Engine Result Page.

The burden is on the Glean Platform to provide a performant, secure, and scalable platform for organizations to quickly find and understand their internal knowledge. Limits had to be applied in order for the platform to achieve these goals. This may mean that some larger content will not be indexed, but the value tradeoffs for the rest of the content far outweigh the potential value lost in these large documents.

Customers have the opportunity to break up large documents into a logical series and/or downsample large multimedia content within them for crucial content whenever necessary.

note

Google.com’s upper bound limit for content size is 16 MB. Also as a point of reference the complete works of Shakespeare contain approximately 5 MB of text.

No. The Glean Product Team is willing to revisit these limits as technology and content changes occur. Feel free to reach out to Glean Support or your Glean Account Team if you feel these limits are limiting for your business.

Yes. Spreadsheet files (such as Excel and Google Sheets) have a character-based indexing limit. By default, approximately 250,000 characters of spreadsheet content are indexed. Content beyond this limit is truncated and won't appear in search results or be available to Assistant.

If you have large spreadsheets that exceed this limit, consider breaking them into smaller files or using Data Analysis to upload and analyze the full file directly.

Essentially all content is affected by these limits. However, in practicality it would be highly unlikely that certain content sources would produce enough text data to ever hit these high limits. (Teams, Slack, Salesforce, ServiceNow, etc…)

It would be considered a safe assumption that most customers would only run into these limits for unstructured Office (Powerpoint, Word, etc..), Google Docs (Slides, Docs, etc…), and PDFs.

For organizations with very large corpuses, consider setting a crawl restriction to limit the scope of indexed content. Depending on your connector, available restriction types include date-based (for example, crawl only the last 2 years of content), greenlist or redlist by site or folder, and owner-based restrictions. For many connectors, the majority of search queries are satisfied by recently created or updated documents.

For connectors that support it, Glean's data fetching feature performs a federated live search alongside indexed corpus search. This means content outside the indexed window can still be surfaced at query time, providing coverage even when the crawler has not indexed the full corpus.

You can configure crawl restrictions on your connector setup page in the Admin console.