GCP Deployment Guide
Deploying Glean within your own Google Cloud Platform (GCP) environment can ensure compliance with your organization’s data residency requirements while maintaining Glean’s managed service model.
Overview
Glean provides our customers the ability to deploy Glean software inside their own Google Cloud Platform (GCP) project. This deployment requires your GCP admin to:
- Create a new GCP project.
- Associate a valid billing account.
- Enable applicable GCP APIs.
- Request the required quota increases from GCP.
- Create a Service Account with Project Owner role and associate a JSON account key.
- Notify Glean of the GCP zone selected, the Project Name, Project ID, Project Number, and the service account JSON key.
After completing the above, Glean’s systems will automatically build and deploy the required compute, workflows, and software into your GCP project.
At this stage, Glean will advise you that your tenant is ready; allowing your admins to proceed with the setup process in our Getting Started guide.
This document will cover the steps required by your GCP admins to prepare a GCP project that is ready for your Glean build.
1. Select a GCP Region
You must first select a supported GCP region for Glean to build your environment in.
- More information: Supported GCP Regions
You must notify Glean of the GCP zone selected, e.g. asia-northeast1-a
The region selected cannot be changed once your tenant has been built. Changing region will require a complete rebuild of your tenant.
2. Create the GCP Project
-
Go to the Manage resources page in the GCP console and click Create Project.
-
In the New Project window that appears, add a project name, organization, and location.
- For the project name, the preferred format is
glean-{customer name}
orglean-{customer name}-{prod/sandbox}
- E.g.
glean-company
orglean-company-prod
- For the project name, the preferred format is
-
Make sure that your project is created under the same organization as your Google Workplace account, and not “No Organization”.
Glean is not able to proceed with the build if the project is created under “No Organization”. If you are unsure of how to resolve this, please contact your GCP account team or GCP support.
-
Save the Project ID (which is directly below the Project name) and Project Number.
-
Click Create.
-
Notify Glean of the following information:
a. Project name, eg
glean-company
→ This was set in Step 2 above.b. Project ID, eg
glean-company
→ This was saved in Step 4 above.c. Project number, eg
715000000000
→ This was saved in Step 4 above.d. Region and Zone where you want to deploy Glean, e.g.
us-central1-a
3. Configure Billing
-
Go to Billing in the GCP console.
-
Click Link a billing account to set up billing for this project.
Ensure that the billing account has a corporate credit card attached to it. Using the “free trial billing tier” will not work.
4. Enable APIs
Glean requires that the following GCP APIs are enabled for the deployment to succeed. Substitute your Project ID to the end of the URL for each API below to enable the API on the project.
API Name | URL |
---|---|
Cloud Resource Manager API (cloudresourcemanager.googleapis.com) | https://console.cloud.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=[PROJECT_ID] |
Service Usage API (serviceusage.googleapis.com) | https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=[PROJECT_ID] |
Compute Engine API (compute.googleapis.com) | https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=[PROJECT_ID] |
Cloud SQL Admin API (sqladmin.googleapis.com) | https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=[PROJECT_ID] |
5. Request Quota Changes
Search for [Quotas]
in the search box of the GCP Console and navigate to All Quotas, under IAM & Admin.
For each of the quotas in the table below, request a quota change by completing the following:
- Click on the required quota.
- Select Edit Quotas
- Enter the value specified by Glean for the quota.
- Click Submit Request.
Please note that some quota requests will require filing a ticket with GCP support. Response time is typically within 2 days.
You must ensure that the region/location specified in your quota request(s) match the GCP Region and Zone that you wish to deploy in. For more information, see Supported GCP Regions.
Quota Type | Service | Metric | Location | New Value | Justification |
---|---|---|---|---|---|
All Quotas | Compute Engine API | CPUs | us-central1 (or primary deployment region) | 110 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | N2 CPUs | us-central1 (or primary deployment region) | 110 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | N2D CPUs | us-central1 (or primary deployment region) | 110 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | T2D CPUs | us-central1 (or primary deployment region) | 128 | The Glean search system runs batch Dataflow pipelines to generate training data, compute statistics, and perform model inference. Without this quota, these pipelines cannot efficiently run. |
All Quotas | Compute Engine API | VM Instances | us-central1 (or primary deployment region) | 240 | The Glean search system deploys Dataflow jobs and Kubernetes cluster, which create VM instances when jobs are launched. |
All Quotas | Compute Engine API | NVIDIA T4 GPUs | us-central1 (or primary deployment region) | 4 | The Glean search system runs batch Dataflow pipelines to generate training data, compute statistics, and perform model inference. Without this quota, these pipelines cannot efficiently run. |
All Quotas | Vertex AI API | Custom model training TPU V2 Cores | us-central1 (or primary deployment region) | 8 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. We use these TPU accelerators to power the training. |
All Quotas | Vertex AI API | Custom model training Nvidia V100 GPUs per region | us-central1 (or primary deployment region) | 8 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. When there are no TPUs available, we use these GPU accelerators to power the training. |
All Quotas | Vertex AI API | Custom model training Nvidia T4 GPUs per region | us-central1 (or primary deployment region) | 4 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. When there are no TPUs or V100’s available, we use these GPU accelerators to power the training. |
All Quotas | Compute Engine API | Persistent Disk Standard | us-central1 (or primary deployment region) | 10TB | The Glean search system stores millions of enterprise documents in Cloud SQL and in a search index with persistent storage. Due to the number and size of documents stored we need the quota to be increased. |
All Quotas | Compute Engine API | In-use IP addresses | us-central1 (or primary deployment region) | 20 | The Glean search system deploys 20-25 flex instances of crawler services on Kubernetes Engine, and each flex instance requires its own IP address. |
Quota Type | Service | Metric | Location | New Value | Justification |
---|---|---|---|---|---|
All Quotas | Compute Engine API | CPUs | us-central1 (or primary deployment region) | 110 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | N2 CPUs | us-central1 (or primary deployment region) | 110 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | N2D CPUs | us-central1 (or primary deployment region) | 110 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | T2D CPUs | us-central1 (or primary deployment region) | 128 | The Glean search system runs batch Dataflow pipelines to generate training data, compute statistics, and perform model inference. Without this quota, these pipelines cannot efficiently run. |
All Quotas | Compute Engine API | VM Instances | us-central1 (or primary deployment region) | 240 | The Glean search system deploys Dataflow jobs and Kubernetes cluster, which create VM instances when jobs are launched. |
All Quotas | Compute Engine API | NVIDIA T4 GPUs | us-central1 (or primary deployment region) | 4 | The Glean search system runs batch Dataflow pipelines to generate training data, compute statistics, and perform model inference. Without this quota, these pipelines cannot efficiently run. |
All Quotas | Vertex AI API | Custom model training TPU V2 Cores | us-central1 (or primary deployment region) | 8 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. We use these TPU accelerators to power the training. |
All Quotas | Vertex AI API | Custom model training Nvidia V100 GPUs per region | us-central1 (or primary deployment region) | 8 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. When there are no TPUs available, we use these GPU accelerators to power the training. |
All Quotas | Vertex AI API | Custom model training Nvidia T4 GPUs per region | us-central1 (or primary deployment region) | 4 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. When there are no TPUs or V100’s available, we use these GPU accelerators to power the training. |
All Quotas | Compute Engine API | Persistent Disk Standard | us-central1 (or primary deployment region) | 10TB | The Glean search system stores millions of enterprise documents in Cloud SQL and in a search index with persistent storage. Due to the number and size of documents stored we need the quota to be increased. |
All Quotas | Compute Engine API | In-use IP addresses | us-central1 (or primary deployment region) | 20 | The Glean search system deploys 20-25 flex instances of crawler services on Kubernetes Engine, and each flex instance requires its own IP address. |
Quota Type | Service | Metric | Location | New Value | Justification |
---|---|---|---|---|---|
All Quotas | Compute Engine API | CPUs | us-central1 (or primary deployment region) | 324 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | CPUs | All Regions | 324 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | N2 CPUs | us-central1 (or primary deployment region) | 324 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | N2 CPUs | All Regions | 324 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | N2D CPUs | us-central1 (or primary deployment region) | 324 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | N2D CPUs | All Regions | 324 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. |
All Quotas | Compute Engine API | T2D CPUs | us-central1 (or primary deployment region) | 256 | The Glean search system runs batch Dataflow pipelines to generate training data, compute statistics, and perform model inference. Without this quota, these pipelines cannot efficiently run. |
All Quotas | Compute Engine API | NVIDIA T4 GPUs | us-central1 (or primary deployment region) | 8 | The Glean search system runs batch Dataflow pipelines to generate training data, compute statistics, and perform model inference. Without this quota, these pipelines cannot efficiently run. |
All Quotas | Vertex AI API | Custom model training TPU V2 Cores | us-central1 (or primary deployment region) | 8 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. We use these TPU accelerators to power the training. |
All Quotas | Vertex AI API | Custom model training Nvidia V100 GPUs per region | us-central1 (or primary deployment region) | 8 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. When there are no TPUs available, we use these GPU accelerators to power the training. |
All Quotas | Vertex AI API | Custom model training Nvidia T4 GPUs per region | us-central1 (or primary deployment region) | 4 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. When there are no TPUs or V100’s available, we use these GPU accelerators to power the training. |
All Quotas | Compute Engine API | VM Instances | us-central1 (or primary deployment region) | 324 | The Glean search system deploys, Dataflow jobs and Kubernetes cluster, which create VM instances when jobs are launched. |
All Quotas | Compute Engine API | Persistent Disk Standard | us-central1 (or primary deployment region) | 20TB | The Glean search system stores millions of enterprise documents in Cloud SQL and in a search index with persistent storage. Due to the number and size of documents stored we need the quota to be increased. |
All Quotas | Compute Engine API | In-use IP addresses | us-central1 (or primary deployment region) | 40 | The Glean search system deploys 20-25 flex instances of crawler services on Kubernetes Engine, and each flex instance requires its own IP address. |
Quota Type | Service | Metric | Location | New Value | Justification | Resource Examples |
---|---|---|---|---|---|---|
All Quotas | Compute Engine API | CPUs | us-central1 (or primary deployment region) | 512 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. | |
All Quotas | Compute Engine API | CPUs | All Regions | 512 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. | |
All Quotas | Compute Engine API | N2 CPUs | us-central1 (or primary deployment region) | 512 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. | |
All Quotas | Compute Engine API | N2 CPUs | All Regions | 512 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. | |
All Quotas | Compute Engine API | N2D CPUs | us-central1 (or primary deployment region) | 512 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. | |
All Quotas | Compute Engine API | N2D CPUs | All Regions | 512 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. | |
All Quotas | Compute Engine API | T2D CPUs | us-central1 (or primary deployment region) | 256 | The Glean search system runs batch Dataflow pipelines to generate training data, compute statistics, and perform model inference. Without this quota, these pipelines cannot efficiently run. | |
All Quotas | Compute Engine API | M1 CPUs | us-central1 (or primary deployment region) | 192 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. | We plan to use the “m1-megamem-96”, which has 96 CPUs. We occasionally run “Switchovers” where we temporarily run a blue-green update to Elasticsearch. This means that our quota usage at the peak would be 192 (96*2), which is why we’re requesting for this quota |
All Quotas | Compute Engine API | M1 CPUs | All Regions | 192 | The Glean search system deploys 20+ instances of crawler services on nodes, as well as multiple nodes of the Elastic index service in Kubernetes cluster and so this quota is needed. Without this quota the system cannot be deployed to the project. We generally run on less than 50% of this quota, and go beyond 50% during Elastic index rolling deployments. | We plan to use the “m1-megamem-96”, which has 96 CPUs. We occasionally run “Switchovers” where we temporarily run a blue-green update to Elasticsearch. This means that our quota usage at the peak would be 192 (96*2), which is why we’re requesting for this quota |
All Quotas | Compute Engine API | NVIDIA T4 GPUs | us-central1 (or primary deployment region) | 8 | The Glean search system runs batch Dataflow pipelines to generate training data, compute statistics, and perform model inference. Without this quota, these pipelines cannot efficiently run. | |
All Quotas | Vertex AI API | Custom model training TPU V2 Cores | us-central1 (or primary deployment region) | 8 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. We use these TPU accelerators to power the training. | |
All Quotas | Vertex AI API | Custom model training Nvidia V100 GPUs per region | us-central1 (or primary deployment region) | 8 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. When there are no TPUs available, we use these GPU accelerators to power the training. | |
All Quotas | Vertex AI API | Custom model training Nvidia T4 GPUs per region | us-central1 (or primary deployment region) | 4 | The Glean search system trains a custom AI language model on the corpus, enabling features such as semantic search, synonyms, and more. When there are no TPUs or V100’s available, we use these GPU accelerators to power the training. | |
All Quotas | Compute Engine API | VM Instances | us-central1 (or primary deployment region) | 512 | The Glean search system deploys Dataflow jobs and Kubernetes cluster, which create VM instances when jobs are launched. | |
All Quotas | Compute Engine API | Persistent Disk Standard | us-central1 (or primary deployment region) | 40TB | The Glean search system stores millions of enterprise documents in Cloud SQL and in a search index with persistent storage. Due to the number and size of documents stored we need the quota to be increased. | |
All Quotas | Compute Engine API | In-use IP addresses | us-central1 (or primary deployment region) | 50 | The Glean search system deploys 20-25 flex instances of crawler services and each flex instance requires its own IP address. | |
All Quotas | Cloud Key Management Service (KMS) API | Cryptographic requests per minute | 120,000 | We use the KMS to secury keep values encrypted in the project to ensure the highest level of security | ||
All Quotas | Cloud Tasks API | Number of queues per region | us-central1 (or primary deployment region) | 9,800 | The Glean search system deploys Cloud Tasks queues to manage the crawler tasks. As the deployment scales its connected datasources, more task queues must be created to handle the increasing scale and complexity of tasks. |
6. Create a Service Account
The service account is used to allow Glean’s systems to access the project and perform the build. You will create the service account and provide Glean with the private JSON key required to use it.
-
Go to the Service Accounts page in the GCP console and click Select a Project.
-
Click Create Service Account. Enter the service account name (
glean-admin
), ID, and description (optional), then click Create. -
Click the Select a role dropdown to make your service account an Owner of the project. Click Continue.
-
Ignore the Grant users access to this service account option. It is not required.
-
Click Create Key. In the panel that appears, select the key type JSON, then Create. This will save a private JSON key to your computer.
7. Upload the Service Account Key to the Glean Admin UI
-
If you haven’t already, follow the instructions from the Access the Admin UI section of the Getting Started guide.
- Browse to https://app.glean.com/admin
- Enter your email to receive a link via email to sign in.
-
On the page titled Create a Google Cloud Platform project, click the box under Step 2 to upload the private JSON key to Glean.
-
Click Save. Glean will now use the JSON key to validate that all the steps above have been performed correctly.
If the save fails, you will be presented with a red error message detailing the issues to correct. The key must be saved correctly before the build of your Glean tenant can proceed.
Troubleshooting
For Error Codes and troubleshooting steps, please see the Troubleshooting section.
FAQ
Was this page helpful?