Skip to content

Commit f61300b

Browse files
potatoqualiteepotatoqualitee
committed
blog drafs
1 parent 6c3d9b8 commit f61300b

2 files changed

Lines changed: 362 additions & 0 deletions

File tree

‎content/post/dbatools-psu-performance.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
---
2+
title: "Optimizing dbatools Performance in PowerShell Universal"
3+
date: 2026-01-27
4+
author: "Chrissy LeMaire"
5+
slug: "psu"
6+
categories: [announcements]
7+
tags: [dbatools, powershell-universal, performance, docker]
8+
draft: true
9+
---
10+
11+
You ever notice how dbatools can be a bit... chatty? Tab completion suggestions, colorful console output, detailed logging — all fantastic when you're working interactively in your terminal. Not so great when you're running dbatools inside a PowerShell Universal API endpoint.
12+
13+
I've been running dbatools inside PSU containers for a while now, and I finally sat down to figure out all the ways we could trim the fat. The result? A two-layer optimization approach that puts each setting in its most elegant place.
14+
15+
## The Problem
16+
17+
dbatools was designed for interactive use. When you import the module, it spins up background runspaces for:
18+
19+
- **TEPP (Tab Expansion Plus Plus)**: Powers intelligent tab completion
20+
- **Logging**: Maintains message queues for troubleshooting
21+
- **Maintenance**: Background housekeeping tasks
22+
23+
These features are amazing at the console. But in an API context? They're overhead. Your API doesn't need tab completion. Those 1,024 messages sitting in a memory queue? Nobody's reading them. Console output formatting? Goes straight to /dev/null.
24+
25+
## The Architecture
26+
27+
Here's the approach: each setting goes where it's architecturally appropriate.
28+
29+
```
30+
Layer 1: Container Start (docker-compose.yml)
31+
├── DBATOOLS_DISABLE_TEPP=1 # Must be set BEFORE Import-Module
32+
└── DBATOOLS_DISABLE_LOGGING=1 # Must be set BEFORE Import-Module
33+
34+
Layer 2: PSU Startup (initialize.ps1)
35+
├── Set-DbatoolsConfig (logging, console output)
36+
└── $PSDefaultParameterValues (EnableException, Verbose, Debug, WhatIf)
37+
```
38+
39+
Why two layers? Because timing matters. The environment variables must exist *before* `Import-Module dbatools` runs — dbatools checks them at import time to decide whether to spin up those background runspaces. Everything else can be configured after the module loads.
40+
41+
## Layer 1: docker-compose.yml
42+
43+
Add these environment variables to your PSU service:
44+
45+
```yaml
46+
environment:
47+
# DBATOOLS PERFORMANCE: Disable background runspaces (must be set before module import)
48+
- DBATOOLS_DISABLE_TEPP=1
49+
- DBATOOLS_DISABLE_LOGGING=1
50+
```
51+
52+
That's it. Two lines. But they have to be here — if you try to set these in PowerShell after the module imports, you've already missed the window.
53+
54+
## Layer 2: initialize.ps1
55+
56+
This is where the bulk of the optimization lives. Here's the configuration block I use:
57+
58+
```powershell
59+
# =============================================================================
60+
# GLOBAL PSU SETTINGS FOR dbatools
61+
# =============================================================================
62+
# These settings optimize dbatools for server/API use where interactive
63+
# features and logging overhead are unnecessary.
64+
#
65+
# IMPORTANT: DBATOOLS_DISABLE_TEPP and DBATOOLS_DISABLE_LOGGING environment
66+
# variables are set in docker-compose.yml because they must be present
67+
# BEFORE Import-Module dbatools runs.
68+
69+
# --- Pipeline Pollution Prevention ---
70+
# dbatools writes progress bars that corrupt JSON output
71+
$global:ProgressPreference = 'SilentlyContinue'
72+
73+
# --- PSDefaultParameterValues: Centralized dbatools defaults ---
74+
# Set common parameters once here instead of on every command call
75+
$global:PSDefaultParameterValues = @{
76+
# Make dbatools throw terminating errors for proper try/catch handling
77+
'*-Dba*:EnableException' = $true
78+
# Disable verbose/debug output (adds overhead, pollutes API responses)
79+
'*-Dba*:Verbose' = $false
80+
'*-Dba*:Debug' = $false
81+
# Ensure commands execute (prevent accidental WhatIf propagation)
82+
'*-Dba*:WhatIf' = $false
83+
# Suppress confirmation prompts (APIs are non-interactive)
84+
'*-Dba*:Confirm' = $false
85+
}
86+
87+
# --- Strict Error Handling ---
88+
$global:ErrorActionPreference = 'Stop'
89+
$global:ConfirmPreference = 'None'
90+
91+
# --- dbatools Connection Settings ---
92+
# Trust self-signed certs in dev/test environments (Docker SQL instances)
93+
Set-DbatoolsConfig -FullName sql.connection.trustcert -Value $true -PassThru | Register-DbatoolsConfig
94+
95+
# --- dbatools Logging: Disable In-Memory Queues ---
96+
# Even with DBATOOLS_DISABLE_LOGGING=1, these provide defense-in-depth.
97+
# Prevents accumulation of 1,024 messages + 128 errors in memory.
98+
Set-DbatoolsConfig -FullName logging.messagelogenabled -Value $false
99+
Set-DbatoolsConfig -FullName logging.messagelogfileenabled -Value $false
100+
Set-DbatoolsConfig -FullName logging.errorlogenabled -Value $false
101+
Set-DbatoolsConfig -FullName logging.errorlogfileenabled -Value $false
102+
103+
# --- dbatools Console Output: Disable ---
104+
# In API context, console output is wasted - it goes nowhere useful
105+
Set-DbatoolsConfig -FullName message.consoleoutput.disable -Value $true
106+
```
107+
108+
A few notes on this configuration:
109+
110+
**`$PSDefaultParameterValues`** is cleaner than setting parameters on every command. Define it once, and every `*-Dba*` command picks up your defaults automatically.
111+
112+
**`EnableException = $true`** is critical for APIs. By default, dbatools writes errors to the error stream but doesn't throw. That's fine interactively — you see the red text and know something went wrong. In an API? You need proper try/catch handling, which means you need terminating errors.
113+
114+
**Defense in depth on logging**: Even with `DBATOOLS_DISABLE_LOGGING=1` set, I still explicitly disable the in-memory queues. Belt and suspenders.
115+
116+
## Performance Impact
117+
118+
Here's what changes:
119+
120+
| Before | After |
121+
|--------|-------|
122+
| 3 background runspaces (TEPP, logging, maintenance) | 1 background runspace (maintenance only) |
123+
| 1,024 message queue + 128 error queue in memory | Queues disabled |
124+
| Console output formatting overhead | Console output disabled |
125+
126+
The estimated improvement is 10-30% memory reduction per runspace, plus eliminated logging I/O. Your mileage may vary depending on how heavily you're using dbatools, but in my testing, API response times became noticeably more consistent.
127+
128+
## What About environments.ps1?
129+
130+
If you're using PSU's environment configuration, you'll want to make sure you have:
131+
132+
- **Integrated environment** sharing process state (required for dbatools connection caching)
133+
- **PersistentRunspace = $true** to keep runspaces warm (80ms vs 500ms latency)
134+
- **Modules pre-loaded** via the environment config
135+
136+
The environment sharing is what lets dbatools cache your SQL connections across API calls. Without it, you'd be reconnecting on every request.
137+
138+
## Wrapping Up
139+
140+
The key insight here is that dbatools isn't slow — it's just optimized for a different use case. Interactive console sessions need tab completion and logging. API endpoints need lean, predictable execution. Configure accordingly.
141+
142+
If you're running dbatools in PSU or any other server/API context, give these optimizations a try. I'd love to hear how they work for you.
143+
144+
\- Chrissy

‎content/post/testing-locally.md‎

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
---
2+
title: "Running dbatools Tests Locally"
3+
date: 2026-01-24
4+
author: "Chrissy LeMaire"
5+
slug: "testing-locally"
6+
aliases:
7+
- /testing-locally/
8+
- /testing-locally/index.html
9+
categories: [announcements]
10+
tags: [testing, pester, contributing]
11+
draft: true
12+
---
13+
14+
If you're contributing to dbatools or just want to make sure your changes work before submitting a PR, running tests locally quite helpful. I've wanted to write this guide for a while because our testing infrastructure has evolved significantly, and the old docs were getting stale.
15+
16+
Initially, our testing suite was developed with Appveyor in mind, but over the years, we've evolved to include local labs as well. Most recently, with Andreas Jordan refactored nearly 400 test files to do better with local setups and to even use scenario-based testing.
17+
18+
Until I get VMSS + local GitHub runners working, Appveyor test runs takes about an hour and a half to run the entire testing suite, so testing locally is even more attractive now.
19+
20+
If you're interested in testing dbatools locally, this post will get you up and running with local testing in about 15 minutes.
21+
22+
## What Changed
23+
24+
Instead of generic `instance1`, `instance2`, `instance3` naming, tests now use purpose-specific instance references:
25+
26+
| Old Pattern | New Pattern | Purpose |
27+
|-------------|-------------|---------|
28+
| `$TestConfig.instance1` | `$TestConfig.InstanceSingle` | Tests needing one instance |
29+
| `$TestConfig.instance1/2` | `$TestConfig.InstanceMulti1/2` | Tests needing multiple instances |
30+
| - | `$TestConfig.InstanceCopy1/2` | Tests that copy between instances |
31+
| - | `$TestConfig.InstanceHadr` | HA/DR tests (AGs, mirroring, log shipping) |
32+
| - | `$TestConfig.InstanceRestart` | Tests that restart SQL Server |
33+
34+
This matters because the CI can now run tests in parallel based on infrastructure requirements, and your local tests can use the same pattern.
35+
36+
## Prerequisites
37+
38+
Before you start, you'll need:
39+
40+
- **PowerShell 5.1+** (Windows PowerShell) or **PowerShell 7+**
41+
- **Git** for cloning the repository
42+
- **At least one SQL Server instance** for basic tests (two for most integration tests)
43+
- **Administrator access** to your SQL instances
44+
45+
### Run PowerShell as Administrator
46+
47+
This is important! Many dbatools commands require local Administrator privileges. Commands like `Copy-DbaLinkedServer`, `Get-DbaService`, and anything that touches WMI will fail without it. Just right-click PowerShell and "Run as Administrator" to save yourself from cryptic permission errors 😊
48+
49+
## Quick Setup
50+
51+
Navigate to your dbatools repository and run these commands:
52+
53+
```powershell
54+
cd c:\github\dbatools
55+
56+
# Install dbatools.library (REQUIRED - contains SMO assemblies)
57+
Install-Module dbatools.library
58+
59+
# Install Pester and PSScriptAnalyzer
60+
Install-Module Pester -RequiredVersion 5.7.1 -Force -SkipPublisherCheck
61+
Install-Module PSScriptAnalyzer -RequiredVersion 1.18.2 -Force
62+
63+
# Copy the config template
64+
Copy-Item .\tests\constants.local.ps1.example .\tests\constants.local.ps1
65+
```
66+
67+
### Configure Your Instances
68+
69+
Edit `tests\constants.local.ps1` with your SQL Server details. Here's a simple two-instance setup:
70+
71+
```powershell
72+
# constants.local.ps1
73+
74+
# Your SQL Server instances
75+
$config['InstanceSingle'] = "YourServer\Instance1"
76+
$config['InstanceMulti1'] = "YourServer\Instance1"
77+
$config['InstanceMulti2'] = "YourServer\Instance2"
78+
$config['InstanceCopy1'] = "YourServer\Instance1"
79+
$config['InstanceCopy2'] = "YourServer\Instance2"
80+
81+
# SQL Authentication (or leave as $null for Windows Auth)
82+
$securePassword = ConvertTo-SecureString "YourPassword!" -AsPlainText -Force
83+
$config['SqlCred'] = New-Object PSCredential ("sa", $securePassword)
84+
85+
# Set defaults for all dbatools commands
86+
$config['Defaults']['*:SqlCredential'] = $config['SqlCred']
87+
$config['Defaults']['*:SourceSqlCredential'] = $config['SqlCred']
88+
$config['Defaults']['*:DestinationSqlCredential'] = $config['SqlCred']
89+
```
90+
91+
> **Tip:** If you only have one SQL Server, just point everything to the same instance. Some multi-instance tests will skip, but you'll still be able to run most tests.
92+
93+
> **Remote instances?** The `$config['Temp']` path must be accessible from both your PowerShell session AND the SQL Server service accounts. Use a network share like `\\FileServer\Share\dbatools-tests`.
94+
95+
## Verify and Run Tests
96+
97+
First, verify your setup:
98+
99+
```powershell
100+
# Import the module
101+
Import-Module .\dbatools.psd1 -Force
102+
Import-Module .\dbatools.psm1 -Force
103+
104+
# Get test config and set defaults
105+
$TestConfig = Get-TestConfig
106+
$PSDefaultParameterValues["*:SqlCredential"] = $TestConfig.SqlCred
107+
108+
# Test connectivity
109+
$TestConfig.InstanceSingle | Connect-DbaInstance | Select-Object Name, Version
110+
```
111+
112+
If you see certificate errors, run:
113+
114+
```powershell
115+
Set-DbatoolsConfig -FullName sql.connection.trustcert -Value $true -Register
116+
```
117+
118+
### Running Tests with Invoke-ManualPester
119+
120+
The easiest way to run tests locally:
121+
122+
```powershell
123+
# Run unit tests only (no SQL Server required)
124+
Invoke-ManualPester -Path Get-DbaDatabase
125+
126+
# Run integration tests (requires SQL Server)
127+
Invoke-ManualPester -Path Get-DbaDatabase -TestIntegration
128+
129+
# Run with code coverage
130+
Invoke-ManualPester -Path Get-DbaDatabase -TestIntegration -Coverage
131+
132+
# Run multiple tests matching a pattern
133+
Invoke-ManualPester -Path "*Backup*" -TestIntegration
134+
```
135+
136+
### Alternative: Direct Pester
137+
138+
If you prefer running Pester directly:
139+
140+
```powershell
141+
# Setup first
142+
Import-Module .\dbatools.psd1 -Force
143+
Import-Module .\dbatools.psm1 -Force
144+
$TestConfig = Get-TestConfig
145+
$PSDefaultParameterValues = $TestConfig.Defaults
146+
147+
# Run tests
148+
Invoke-Pester .\tests\Get-DbaDatabase.Tests.ps1 -Output Detailed
149+
```
150+
151+
## Test Scenarios
152+
153+
Tests are organized into scenarios based on infrastructure requirements (defined in `tests\pester.groups.ps1`):
154+
155+
| Scenario | Description |
156+
|----------|-------------|
157+
| **SINGLE** | Tests needing one instance |
158+
| **MULTI** | Tests needing multiple instances |
159+
| **COPY** | Tests that copy between instances |
160+
| **HADR** | HA/DR tests (AGs, mirroring) |
161+
| **RESTART** | Tests that restart SQL Server |
162+
163+
The CI automatically detects which scenario each test belongs to based on which `$TestConfig` properties it uses.
164+
165+
## CI Integration
166+
167+
When you submit a PR, you can control which tests run using commit message patterns:
168+
169+
```bash
170+
# Run only Get-DbaDatabase tests
171+
git commit -m "Fix database enumeration (do Get-DbaDatabase)"
172+
173+
# Run all backup-related tests
174+
git commit -m "Update backup logic (do *Backup*)"
175+
```
176+
177+
For the curious, our AppVeyor builds run scenarios in parallel across multiple VMs. Check out [our AppVeyor project](https://ci.appveyor.com/project/dataplat/dbatools) to watch tests run in real-time.
178+
179+
## Troubleshooting
180+
181+
**"dbatools.library not found"**
182+
183+
Reinstall it: `.\.github\scripts\install-dbatools-library.ps1 -Force`
184+
185+
**"Cannot connect to SQL Server"**
186+
187+
Check your instance name with `Test-DbaConnection -SqlInstance $TestConfig.InstanceSingle` and verify your credentials.
188+
189+
**"Access denied" errors**
190+
191+
Make sure you're running PowerShell as Administrator. This is required for WMI operations, service management, and cross-server operations.
192+
193+
**"Access denied to temp path"**
194+
195+
For remote instances, use a network share: `$config['Temp'] = "\\FileServer\Share\dbatools-tests"`. The SQL Server service accounts need write access.
196+
197+
## Quick Reference
198+
199+
```powershell
200+
# === INITIAL SETUP (one time) ===
201+
.\.github\scripts\install-dbatools-library.ps1
202+
Install-Module Pester -RequiredVersion 5.7.1 -Force -SkipPublisherCheck
203+
Copy-Item .\tests\constants.local.ps1.example .\tests\constants.local.ps1
204+
# Edit constants.local.ps1 with your instances
205+
206+
# === BEFORE EACH TEST SESSION ===
207+
Import-Module .\dbatools.psd1 -Force
208+
Import-Module .\dbatools.psm1 -Force
209+
$TestConfig = Get-TestConfig
210+
$PSDefaultParameterValues["*:SqlCredential"] = $TestConfig.SqlCred
211+
212+
# === RUN TESTS ===
213+
Invoke-ManualPester -Path Get-DbaDatabase -TestIntegration
214+
```
215+
216+
Got questions? Hit up #dbatools on the [SQL Server Community Slack](https://aka.ms/sqlslack) and we'll help you out.
217+
218+
\- Chrissy

0 commit comments

Comments
 (0)