Service restoration access
Glean provides a secure, audited, and time limited service restoration process. This mechanism allows engineers on call to restore services during rare, serious incidents when normal administrative paths are unavailable. The process maximizes safety and business continuity, without granting persistent access or exposing customer data.
Infrastructure and security
Administrators approve infrastructure commands in advance, and an isolated backend service automatically executes them using service account credentials with limited privileges.
- Restricted access: Engineers submit commands through an internal portal accessible only to approved production personnel.
- Mandatory approval: Every command requires peer review and approval before execution.
- Zero direct access: Employees don't have direct permissions to execute commands through this service account.
- Framework isolation: This process operates independently from Glean’s Debug Operation request handling framework.
Key advantages
Business continuity: During rare outages, the service mitigates incidents even when those outages impair standard paths, such as vault keys or administrator controls. Security by design: Enforces security through zero persistent entitlements, a narrow tool scope, and strict traceability. Operational accountability: Provides transparent auditing for every request, approval, and command.
Architectural and security specifications
- Isolated execution environment: A lightweight, fully isolated service that runs within the same Kubernetes cluster but operates in a separate namespace from the main application. This separation reduces the attack surface and maintains operational independence.
- Short-lived credentials with limited privileges: The system generates credentials with a short lifetime scoped exclusively to infrastructure commands, for example,
kubectl,gcloud,aws, andaz. These credentials explicitly deny access to customer data stores. - Zero standing privileges: Glean doesn't maintain persistent service restoration access. The system creates access when needed, limits usage to a minimum duration, and ensures automatic expiration.
- Auditable approval workflow: Requests require justification and approval by an authorized second party. This separate approval follows the
two person ruleand prevents self approval. - Immutable auditing: The system logs every tool, including the requester, approver, execution details, timestamps, and justification, to a cloud provider audit log designed to detect tampering. Logs are available to customers upon request.
Scope and restrictions
| Area | Not included |
|---|---|
| Data access | Prohibits direct access to customer data stores. |
| Persistent access | No long-lived credentials, backdoors, or standing entitlements. |
| Routine ops | Excludes routine maintenance and convenience tasks. |
| One person control | Prevents any single actor from both requesting and approving access. |
How it works
-
Request: An on-call engineer submits a service restoration request through Glean’s internal admin UI, including a justification and an intended timeout.
-
Approval: A separate authorized production engineer or manager (Glean employee) reviews and approves or rejects the request. Approval is required before any command can be executed.
-
Execution in isolation: Upon receiving an approved command payload, the isolated service automatically executes the vetted infrastructure command by using a service account with limited privileges. Access to customer data is explicitly denied by policy.
-
Audit everywhere: The system logs the unique request identifier, requester, approver, timestamps, commands, and outcomes in an immutable audit store.
Customer safeguards by design
-
Strong separation of duties with the two person rule.
-
Explicitly deny data store access for querying or viewing customer data.
-
Short-lived execution credentials that automatically expire.
-
Full transparency through immutable audit logs ordered by time.
-
Principle of least privilege applied to all service restoration access tools.
-
No shared infrastructure between the service restoration access service and the application runtime to minimize the impact of a security incident.
Permissions granted to the service
The following roles support troubleshooting and administration beyond the access that on-call engineers have by default.
Cloud service account “service-restoration-gke” with roles:
- Kubernetes Engine Developer
- Dataflow Developer
- Custom Cloud Build Developer
- App Engine Service Admin
- Cloud SQL Admin
- Service Usage Admin
- Cloud Run Admin
- Cloud Scheduler Admin
- Cloud Functions Admin
- Cloud Tasks Admin
- Redis Admin
- IAM Role Viewer
- Service Account Viewer
- Compute Viewer
- Compute Network User
- Compute Load Balancer Viewer
- DNS Reader
Kubernetes service account service-restoration-gke on glean-cluster only with role:
- Kubernetes Cluster Admin
Blocked operations
The service restoration blocks all commands that could expose customer data:
Database access
- Direct database connections (mysql, psql, redis-cli)
- Database dumps and backups (mysqldump, pg_dump)
- Data export and import operations
Object storage
- Google Cloud Storage (gsutil, gcloud storage)
- AWS S3 (aws s3, s3api, s3control)
- Azure Blob Storage (az storage)
- Third-party storage tools (rclone, MinIO)
File operations
- Reading sensitive file types (.sql, .db, .json, .csv, .parquet, .log)
- File transfers (ssh, scp, rsync)
Secrets and credentials
- Viewing secrets (kubectl get secret, gcloud secrets, aws secretsmanager, az keyvault)
Command chaining
- The system blocks shell operators, including pipes, semicolons, and backgrounding, to prevent bypass attempts
This capability is part of Glean’s standard reliability and safety controls to ensure service is restored under adverse conditions. Learn more about Glean’s access to customer deployments in the Trust Center resources.