To enable Glean’s Generative AI features, you need to select a LLM provider. We recommend using Glean’s Azure OpenAI key. We also provide the option of using your own LLM key with Azure OpenAI, Google Vertex AI (if your deployment is on GCP), and Amazon Bedrock (if your deployment is on AWS). This can can be setup in Admin console > Platform > Assistant. To use your own LLM provider, please work with your Glean sales or technical services person.

​
Option 1: Glean’s Azure OpenAI key (Recommended)

Glean has a signed agreement with Azure OpenAI promising:
  • 0-day retention: Your data will not be stored by Azure.
  • Data will not be used to train any custom large language models.
  • Data encryption: All data is encrypted in transit.
  • Compliance: Azure is compliant with a variety of industry standards. See details here
The advantages of this approach are
  • Guaranteed capacity: Scale to all your users
  • Performance: Low latency
  • Access to all the necessary models out of the box.
  • 0-day retention isn’t the default on Azure Open AI.

​
Option 2: Your own OpenAI or Azure OpenAI key

We currently support using either GPT-4o or GPT-4-Turbo as the primary model for query planning and answer generation. Fill out the Azure OpenAI Service form and request access to the following models:
Model nameHow Glean uses the model
GPT-4o or GPT-4-TurboQuery planning and answer generation
GPT-4o-mini or GPT-3.5-TurboTool selection and followup question generation
Ada EmbeddingsMatch sentences in the generated answer with the retrieved search results to generate citations.
Please see Azure OpenAI Service quotas and limits for the default quotas and instructions for requesting additional quota.

​
Option 3: Anthropic Claude models on Amazon Bedrock (AWS only)

Log into the AWS Console with a user who has permissions to subscribe to Bedrock models within the AWS account. If your AWS instance is in one of the following regions, we will send the LLM traffic to Amazon Bedrock in the same region as your instance. If it is not in one of these regions, please contact the Glean team to configure your instance to send the LLM traffic to the nearest region where the models are supported.
  • us-east-1 (N. Virginia)
  • us-west-2 (Oregon)
  • ap-northeast-1 (Tokyo)
  • eu-central-1 (Frankfurt)
Go to Amazon Bedrock > Model access, select the same region as your Glean AWS instance (or the nearest supported region), and request access to these models:
Model nameHow Glean uses the model
Anthropic > Claude 3.5 Sonnet
anthropic.claude-3-5-sonnet-20240620-v1:0		Query planning and answer generation
Anthropic > Claude 3 Haiku
anthropic.claude-3-haiku-20240307-v1:0		Tool selection and followup question generation
Amazon > Titan Embeddings G1 - Text
amazon.titan-embed-text-v1		Match sentences in the generated answer with the retrieved search results to generate citations.
If you are asked about the use case for the models, you can enter: Generate answers to questions about internal company documents Please see Amazon Bedrock Quotas for the default quotas for these models on pay-as-you-go. If you require additional quota, you will need to reach out to your AWS account manager. Bedrock does not provide a self-service method to increase quota at this time. Glean does not currently support Provisioned Throughput on Bedrock, but we can work with you and the AWS team to enable this in the future.

​
Option 4: Anthropic Claude models on Google Vertex AI (GCP only)

Go to the Vertex AI Model Garden and make sure you have enabled access to the following foundation models from the GCP project that Glean is running in:
Model nameHow Glean uses the model
Claude 3.5 Sonnet
claude-3-5-sonnet@20240620		Query planning and answer generation
Claude 3 Haiku
claude-3-haiku@20240307		Tool selection and followup question generation
Embeddings for Text
textembedding-gecko@002		Match sentences in the generated answer with the retrieved search results to generate citations.
You will need to file a standard GCP quota request, which is expressed in Requests Per Minute (RPM) and Tokens Per Minute (TPM). Filter for “base_model:” on the model name (anthropic-claude-3-5-sonnet, anthropic-claude-3-haiku, textembedding-gecko) and “region:” for the region that your GCP project is running in. Please note that the quota is not a guarantee of capacity, but is intended by Google to ensure fair use of the shared capacity, and your requests may not be served during peak periods. To obtain guaranteed capacity, please speak with your Google account team about purchasing Provisioned Throughput.

​
Option 5: Gemini models on Google Vertex AI (GCP only)

Go to the Vertex AI Model Garden and make sure you have enabled access to the following foundation models from the GCP project that Glean is running in:
Model nameHow Glean uses the model
Gemini 1.5 Pro
gemini-1.5-pro-002		Query planning and answer generation
Gemini 1.5 Flash
gemini-1.5-flash-002		Tool selection and followup question generation
Embeddings for Text
textembedding-gecko@002		Match sentences in the generated answer with the retrieved search results to generate citations.
You will need to file a standard GCP quota request, which is expressed in Requests Per Minute (RPM) and Tokens Per Minute (TPM). Filter for “base_model:” on the model names (gemini-1.5-pro, gemini-1.5-flash, textembedding-gecko) and “region:” for the region that your GCP project is running in. Please note that the quota is not a guarantee of capacity, but is intended by Google to ensure fair use of the shared capacity, and your requests may not be served during peak periods. To obtain guaranteed capacity, please speak with your Google account team about purchasing Provisioned Throughput.

​
Capacity Requirements

​
Query planning and answer generation

Here are the capacity requirements for GPT-4o, GPT-4 Turbo, Claude 3.5 Sonnet, or Gemini 1.5 Pro:
UsersRPMTPM
5002070,000
1,00040135,000
2,500100335,000
5,000200665,000
10,0003501,165,000
20,0005001,665,000

​
Tool selection and followup question generation

Here are the capacity requirements for GPT-4o mini, GPT-3.5-Turbo, Claude 3 Haiku, or Gemini 1.5 Flash:
UsersRPMTPM
5001015,000
1,0002025,000
2,5005060,000
5,000100115,000
10,000175205,000
20,000250290,000

​
Citation generation

Here are the capacity requirements for Open AI Ada Embeddings or Google Embeddings for Text:
UsersRPMTPM
5001513,000
1,0003025,000
2,5007563,000
5,000150125,000
10,000260219,000
20,000375313,000
Here are the capacity requirements for Amazon > Titan Embeddings G1 - Text:
UsersRPMTPM
50050013,000
1,0001,00025,000
2,5002,50063,000
5,0005,000125,000
10,0008,750219,000
20,00012,500313,000