Run the Admin Toolkit

Overview

The Admin Toolkit is an always-on tool deployed alongside Domino in the Kubernetes cluster that allows Domino administrators to run a system scan on the Domino deployment to do the following:

Check for unpatched known issues
Validate the health of core services
Get information to assist troubleshooting, run remediation actions, or provide context about issues affecting the deployment
Provide the toolkit report to Domino Support as part of an incident

How it works

The toolkit has a front-end web interface provided at https://<your-domino-url>/toolkit/ where you can view scan reports, initiate new scans, and export platform logs. You can run scans as many times as necessary because the checks performed by the toolkit are only reading information and not making any modifications. You can run a scan during any of these stages:

After a new Domino install
At the start of an incident
In the midst of an incident
After incident resolution to verify health
Before and after a Domino upgrade
Before and after a Domino migration

You can find details about the specific tests that are included in the scan in the Admin Toolkit documentation. You can find check runbooks in the Support site knowledge base (requires sign-in).

Use the Admin Toolkit

The Admin Toolkit is available:

at the URL: https://<your-domino-url>/toolkit/
through the Domino Admin UI under Admin Panel > Reports > Diagnostic toolkit.

Out of the box, Admin Toolkit should be configured with SSO so that users are automatically logged in to the Toolkit UI if they are logged in to Domino. If they are not currently logged in to Domino or their session has expired, then they see the Admin Toolkit login screen. Clicking on Continue with Domino SSO takes the user to the normal Domino login screen where they can log in using their Domino credentials as normal. Once logged in, Domino redirects back to the Toolkit UI. All Domino SysAdmin users are automatically granted the Toolkit Admin role.

Log in when SSO is unavailable

This is a break-glass procedure. If the toolkit pod itself is unavailable, resolve the pod issue before attempting this login. When SSO is not working (for example, Keycloak is unavailable for some reason), then there is a backup method to access Toolkit with Admin privileges.

Get the Admin Toolkit password

The Admin Toolkit password is stored in a secret called domino-admin-toolkit-http in the platform namespace. Retrieve it with kubectl:

kubectl -n <platform-namespace> get secrets domino-admin-toolkit-http \
  -o jsonpath='{.data.webui-password}' | base64 -d

Navigate to the Toolkit login page and do the following:

Enter admin-toolkit as the username.
Enter the password you retrieved in the preceding step.
Click Login.

Run a system scan

Navigate to https://<your-domino-url>/toolkit/ and log in.
Click the Run Report Now button at the top of the Reports page.

When the scan has finished, the report appears in the Recent Reports list.

View the reports

Use the Admin Toolkit’s Reports tab to review completed scans. The Recent Reports section displays a list of the 10 most recent reports with the following information:

Status: COMPLETED, RUNNING, PENDING, or FAIL
Summary: counts of failed, passed, and skipped checks
Started: timestamp and date of the scan
Started by: the user who initiated the scan

Expanding a report row shows additional details:

Failure groups: checks that failed, grouped by error type, with specific test names listed
Time breakdown: total run time and duration of the slowest individual checks

Download or delete a report

Each report in the list has action buttons:

View: open the full HTML report in the browser
Download HTML: save the report as a standalone HTML file
Download ZIP: download an archive containing the HTML report and a machine-readable JSON results file
Delete: remove the report

Reports are stored in the toolkit database and persist across container restarts within the same pod. If the toolkit pod is rescheduled or deleted, stored reports are lost. With kubectl access you can bulk-copy all stored reports:

TOOLKIT_POD=$(kubectl get pod -n <platform-namespace> \
  -l app.kubernetes.io/name=domino-admin-toolkit \
  -o jsonpath='{.items[0].metadata.name}')

kubectl cp -n <platform-namespace> -c domino-admin-toolkit \
  "${TOOLKIT_POD}:/app/domino_admin_toolkit/ui/reports/" ./reports/

Read a report

Each check in a toolkit report produces one of the following results:

Result	Meaning
PASS	The check ran and found no issue.
WARN	The check ran and found a non-critical anomaly. Investigate, but the platform is functional.
FAIL	The check ran and found a critical issue. Action required.
ERROR	The check encountered a technical failure (for example, missing data or an unexpected exception) and could not produce a result. Investigate the root cause.
SKIP	The check did not apply to this deployment (for example, a cloud-specific check on a different provider, or a Model Monitoring check on a deployment without DMM).

For every check the report shows a description of what was checked, the result, the evidence collected (tables, counts, thresholds), and troubleshooting and resolution steps where applicable.

Filter checks in the report

Each scan runs the full toolkit check suite: general health checks, informational checks, pre-upgrade checks, and pre-migration checks. You cannot limit a scan to a single category from the web UI or API. After a scan completes, open the HTML report and use the Show Checks toggle to filter what is displayed:

Category	What it contains
General	Day-to-day health and platform validation checks (files matching `test_*.py` at the checks root). Shown by default.
Info	Deployment state snapshots: versions, feature flags, configuration tables. These checks never fail; they collect context for support tickets. Shown by default.
Pre-upgrade	Checks that should pass before a Domino upgrade. Hidden by default; enable the toggle when reviewing upgrade readiness.
Pre-migration	Checks that gate a data-plane or deployment migration. Hidden by default; enable when planning a migration.

For upgrade or migration workflows, run a normal scan and filter the report to the relevant category. Resolve any failures before proceeding. For deeper guidance on a specific check, see the Support site knowledge base or the embedded docs at https://<your-domino-url>/toolkit/docs/.

What the toolkit checks

At a high level the checks cover the following areas:

Core services: MongoDB, Postgres, Redis, RabbitMQ, Vault, Elasticsearch, Keycloak, Nexus, nucleus-frontend health endpoint.
Kubernetes and the cluster: node health, control-plane / Kubelet version drift, etcd, OOM kills, time drift, flapping nodes, ephemeral storage, kernel vulnerabilities (all platforms), CNI, Istio, CoreDNS, NGINX ingress, cluster-autoscaler, Karpenter capacity.
Domino platform: deployment compatibility matrix, HelmRelease drift, central config, image builder, image cache, container image versions, mutating-webhook patches, Flows pods, workload-service log analysis.
Sizing and capacity: platform sizing, hardware-tier sizing, replica counts, file-task queue status, run queue.
Networking: service DNS, egress, network stats, Docker Hub rate limits.
Certificates: TLS secret expiration, Domino URL certificate, cert-manager health.
Storage: platform PVCs, Domino filesystem, dataset lifecycle, backup cronjobs.
Observability: Prometheus, Grafana, Grafana alerting, Dynatrace, New Relic.
Cloud-specific: AWS cloud config, AWS-specific kernel vulnerability checks, ulimit configuration.
Model Monitoring (DMM): ingestion job status, Redis queues, pod list.

Export platform logs

The Log Export tab provides a web interface for collecting and bundling logs from Domino platform components. Use it to gather diagnostic logs for troubleshooting or to share with Domino Support.

Collection options

Before starting a collection, configure the options at the top of the Log Export page:

Option	Description
Log line limit	Maximum number of log lines to collect per pod (default: 5000).
Time range	Collect logs from a rolling window (for example, Last 2 hours). Select All available to collect without a time window, limited only by the log line limit and node log rotation.
Container logs	Which container inside each pod to collect: Default container only, Specific container, or All containers.
Include previous container logs	Also collect logs from previously terminated containers. Useful for diagnosing crash-loop or OOMKilled restarts.
Include per-component analysis summaries	Run pattern analysis on collected logs and include summaries in the bundle.

Bundle profiles

Bundle profiles are one-click presets for the most common troubleshooting scenarios. Each profile collects logs from a predefined set of components. Click Collect next to the profile that matches your issue:

Profile	Use for
Authorization & Authentication	Login failures, token/SSO/OIDC issues, 401/403 errors, roles and groups. Collects Keycloak, Nucleus frontend, ingress, Postgres, IDSM audit trail.
Core Domino Platform	Jobs or workspaces stuck or failing, scheduling, quotas, node scale-up, and execution path problems. Collects Nucleus dispatcher, frontend, develop, train, data-plane-agent, workload-service, cluster-autoscaler, and RabbitMQ.
Environment Build Issues	Workspace or job environment image builds failing, stuck, or slow. Registry push/pull errors, BuildKit failures, hephaestus controller errors. Collects hephaestus-manager, BuildKit, Nucleus frontend/dispatcher, workload-service, and ingress.

Click the ▸ Components (N) link under each profile to see the full list of components included.

Workspace and user log collection

The table below maps common workspace troubleshooting scenarios to the appropriate collection method:

Scenario	Recommended approach	What is collected
Workspace stuck at startup, failing to launch, or running slowly	Core Domino Platform profile	`nucleus-dispatcher` and `nucleus-develop` capture the scheduling and workspace container startup path. `data-plane-agent` captures node-side provisioning errors.
Batch job (train run) stuck or failing	Core Domino Platform profile	`nucleus-train` captures batch job container logs alongside `nucleus-dispatcher` and `workload-service`.
Workspace environment image failing to build	Environment Build Issues profile	`hephaestus-manager`, `hephaestus-buildkit`, `nucleus-dispatcher`, and `workload-service` cover the full image build and push path.
User code logs from inside a running workspace (stdout / stderr from notebooks or scripts)	Advanced: select the compute namespace and executor pods	User code runs in executor pods in the `domino-compute` namespace, not the platform namespace. In the Advanced section, switch the namespace to `domino-compute` and select the relevant executor pod by prefix or label to collect its logs.

Advanced custom component selection

Expand the Advanced section to manually select individual components for collection. Use this when no bundle profile matches your scenario or when you need logs from a specific component only.

Recent log bundles

The Recent log bundles section shows the history of past export jobs and auto-refreshes while jobs are running. Expand a row to see live per-component progress and to download the completed bundle as a ZIP archive.

Send reports to Domino

Regardless of whether a scan runs from the web interface or API, the toolkit can automatically upload the report to a centralized S3 bucket owned by Domino within Domino’s internal AWS account. Access is restricted to authorized Domino engineers via single sign-on (SSO). This behavior is on by default, but you can turn it off by selecting the opt-out option in the web interface or by applying a ConfigMap. There are many benefits to uploading reports to Domino:

Quicker resolution of support tickets.
Validation of deployment configuration.
Receive more proactive, rather than reactive, support.

Domino can also use the data collected from multiple customers to better understand how customer-specific configuration and infrastructure affect the overall performance and stability of the Domino platform and to improve the product with each release.

Admin Toolkit reports do contain potentially sensitive information such as IP addresses, hostnames, partial log files and usernames.Passwords are not included.If you have concerns about any of the information collected, contact support@dominodatalab.com or your CSM or TAM to discuss your concerns.

Turn report sending to Domino on or off

By default, the feature to have reports uploaded to Domino is on. Each newly generated toolkit report is securely sent to Domino Support and Engineering. This allows Domino engineers to gain access to the reports faster during an outage and get historical context to previous outages. You can opt out of this service (or opt in again) via the web interface. To opt out, navigate to the Settings tab, find the Report Sending card, and click the Opt Out of Report Sending button. To opt back in, click the Opt Into Report Sending button.

Reports do not contain passwords or secrets. They may include hostnames, IP addresses, usernames, and partial log data.

Scheduled reports

Scheduled reporting lets you generate Domino health reports at regular intervals. Having a “healthy state” report handy provides a comparative baseline during system disruptions. This can be invaluable in swiftly identifying system anomalies, thus accelerating the issue resolution process.

Set up scheduled reporting

Follow the steps below to set up scheduled reporting:

Open the toolkit web interface and navigate to the Settings tab.
In the Schedule Reports card, enter a cron expression in the Cron Expression field to specify the frequency and timing for the reports.
Click Update Schedule.

Now the toolkit generates and stores system health reports according to the schedule you’ve set. These reports can be accessed and reviewed anytime, providing you with a consistent and regular overview of your system’s health.

By default:

The schedule is set to once a day.
A maximum number of 10 reports are saved at any one time.

Use scheduled reports during system disruptions

In case of a system disruption or outage:

Access the most recent “healthy state” report from the stored scheduled reports.
Generate an immediate system health report (if possible).
Compare the two reports to identify any deviations or unusual activities.

Sharing this information with the Domino support team can expedite the troubleshooting process and help identify the root cause of the problem more efficiently. By leveraging scheduled reporting, you can manage system health more effectively, contributing to smoother, more reliable operations in your Domino environment.

Check timeout configuration

By default, each individual check in a toolkit scan is allowed 60 seconds to complete before it times out with an error. You can increase this limit for environments where checks take longer due to cluster size or load. Set the CHECK_TIMEOUT_SECONDS environment variable on the toolkit deployment (for example via Helm podEnv on the domino-admin-toolkit container):

- name: CHECK_TIMEOUT_SECONDS
  value: "120"

Per-check timeout overrides are also available in the check source code using the @pytest.mark.check_timeout(seconds) marker, but these are set by the toolkit developers and cannot be changed without modifying the toolkit itself.

Understand and resolve failures

The Admin Toolkit documentation provides comprehensive information about the checks, their meanings, and any details on resolving issues. Use the toolkit and its documentation to quickly identify and resolve issues, and ensure optimal performance of the Domino platform. You can access the documentation in two ways:

Publicly accessible online documentation: https://docs.toolkit.re.domino.tech/ (always updated to the latest version of the toolkit).
Documentation included within the Domino deployment: https://<your-domino-url>/toolkit/docs/ (specific to the deployed version of the toolkit).

Architecture & deployment

Domino installation

Identity & access

Infrastructure & compute

Platform configuration

Data administration

Operations, monitoring & cost

Security & compliance

Troubleshooting

Run the Admin Toolkit

Overview

How it works

Use the Admin Toolkit

Log in when SSO is unavailable

Get the Admin Toolkit password

Run a system scan

View the reports

Download or delete a report

Read a report

Filter checks in the report

What the toolkit checks

Export platform logs

Collection options

Bundle profiles

Workspace and user log collection

Advanced custom component selection

Recent log bundles

Send reports to Domino

Turn report sending to Domino on or off

Scheduled reports

Set up scheduled reporting

Use scheduled reports during system disruptions

Check timeout configuration

Understand and resolve failures

​Overview

​How it works

​Use the Admin Toolkit

​Log in when SSO is unavailable

​Get the Admin Toolkit password

​Run a system scan

​View the reports

​Download or delete a report

​Read a report

​Filter checks in the report

​What the toolkit checks

​Export platform logs

​Collection options

​Bundle profiles

​Workspace and user log collection

​Advanced custom component selection

​Recent log bundles

​Send reports to Domino

​Turn report sending to Domino on or off

​Scheduled reports

​Set up scheduled reporting

​Use scheduled reports during system disruptions

​Check timeout configuration

​Understand and resolve failures

Overview

How it works

Use the Admin Toolkit

Log in when SSO is unavailable

Get the Admin Toolkit password

Run a system scan

View the reports

Download or delete a report

Read a report

Filter checks in the report

What the toolkit checks

Export platform logs

Collection options

Bundle profiles

Workspace and user log collection

Advanced custom component selection

Recent log bundles

Send reports to Domino

Turn report sending to Domino on or off

Scheduled reports

Set up scheduled reporting

Use scheduled reports during system disruptions

Check timeout configuration

Understand and resolve failures