Test Maintenance Guide¶
Guide for maintaining a healthy test suite over time.
Table of Contents¶
- Coverage Monitoring
- When Coverage Drops
- Updating Tests
- Test Performance
- CI/CD Integration
- Troubleshooting Flaky Tests
- Test Isolation Issues
- Deprecating Tests
Coverage Monitoring¶
Current Coverage Requirements¶
The project maintains 35% minimum code coverage across all source code (configured in pyproject.toml).
Checking Current Coverage¶
Check coverage locally:
# Quick check
make test-coverage
# Detailed report
uv run pytest --cov=registry --cov-report=term-missing
# HTML report for detailed analysis
uv run pytest --cov=registry --cov-report=html
open htmlcov/index.html
Coverage Reports¶
Coverage reports show:
- Overall coverage percentage
- Coverage per module
- Missing lines (not covered by tests)
- Branch coverage (conditional paths)
Example output:
Name Stmts Miss Cover Missing
---------------------------------------------------------------
registry/services/server.py 45 3 93% 12, 45-47
registry/api/routes.py 120 15 88% 78-82, 156-162
registry/core/config.py 25 0 100%
---------------------------------------------------------------
TOTAL 450 35 92%
Monitoring Coverage in CI/CD¶
Coverage is automatically checked in CI/CD:
- GitHub Actions: Every PR and commit
- Codecov: Tracks coverage over time
- PR Comments: Shows coverage changes
Coverage Badges¶
Add coverage badge to README:
[](https://codecov.io/gh/username/repo)
When Coverage Drops¶
Identifying Uncovered Code¶
- Run coverage report:
- Check the HTML report:
- Red lines: Not covered
- Yellow lines: Partially covered (some branches)
-
Green lines: Fully covered
-
Focus on critical paths first:
- API endpoints
- Business logic
- Error handling
- Data validation
Adding Tests for Uncovered Code¶
Example: Adding tests for uncovered function
# Original uncovered function
def calculate_score(metrics: Dict[str, float]) -> float:
"""Calculate composite score from metrics."""
if not metrics:
return 0.0
total = sum(metrics.values())
count = len(metrics)
return total / count
# Add tests to cover this function
@pytest.mark.unit
class TestScoreCalculation:
"""Tests for calculate_score function."""
def test_calculate_score_with_valid_metrics(self):
"""Test score calculation with valid metrics."""
metrics = {"metric1": 0.8, "metric2": 0.9, "metric3": 0.7}
score = calculate_score(metrics)
assert score == pytest.approx(0.8, rel=0.01)
def test_calculate_score_with_empty_metrics(self):
"""Test score calculation with no metrics."""
score = calculate_score({})
assert score == 0.0
def test_calculate_score_with_single_metric(self):
"""Test score calculation with single metric."""
score = calculate_score({"metric1": 0.5})
assert score == 0.5
Strategies for Improving Coverage¶
- Start with low-hanging fruit: Test simple functions first
- Focus on new code: Ensure new features have tests
- Test error paths: Add tests for exception handling
- Test edge cases: Boundary conditions, empty inputs, etc.
- Add integration tests: Cover component interactions
Updating Tests¶
When Code Changes¶
Update tests when code changes:
- API changes: Update API tests
- Function signatures: Update unit tests
- New features: Add new tests
- Bug fixes: Add regression tests
- Refactoring: Update mocks and fixtures
Test Update Checklist¶
When updating code:
- Update affected unit tests
- Update integration tests if needed
- Add tests for new functionality
- Verify all tests still pass
- Check coverage hasn't dropped
- Update test documentation
Example: Updating Tests After Code Change¶
Code change: Add pagination to list_servers endpoint
# Old code
def list_servers():
return server_service.list_servers()
# New code
def list_servers(page: int = 1, page_size: int = 10):
return server_service.list_servers_paginated(page, page_size)
Update tests:
# Old test
def test_list_servers(server_service):
servers = server_service.list_servers()
assert isinstance(servers, list)
# Updated test
def test_list_servers_default_pagination(server_service):
"""Test list servers with default pagination."""
result = server_service.list_servers_paginated(page=1, page_size=10)
assert isinstance(result["items"], list)
assert result["page"] == 1
assert result["page_size"] == 10
def test_list_servers_custom_pagination(server_service):
"""Test list servers with custom pagination."""
result = server_service.list_servers_paginated(page=2, page_size=5)
assert result["page"] == 2
assert result["page_size"] == 5
def test_list_servers_invalid_page_raises_error(server_service):
"""Test invalid page number raises error."""
with pytest.raises(ValueError):
server_service.list_servers_paginated(page=0, page_size=10)
Test Performance¶
Identifying Slow Tests¶
Find slow tests:
# Show test durations
uv run pytest --durations=10
# Show all durations
uv run pytest --durations=0
# Run only slow tests
uv run pytest -m slow
Optimizing Test Performance¶
- Use appropriate fixtures:
# Good - Function-scoped for isolation
@pytest.fixture
def temp_database():
db = create_database()
yield db
db.cleanup()
# Better - Module-scoped for performance
@pytest.fixture(scope="module")
def shared_database():
db = create_database()
yield db
db.cleanup()
- Mock expensive operations:
# Slow - Real API calls
def test_fetch_data():
data = external_api.fetch()
assert data is not None
# Fast - Mocked API
@patch('module.external_api.fetch')
def test_fetch_data(mock_fetch):
mock_fetch.return_value = {"data": "test"}
data = external_api.fetch()
assert data == {"data": "test"}
- Run tests in parallel:
- Skip slow tests during development:
# Mark slow tests
@pytest.mark.slow
def test_expensive_operation():
pass
# Skip in development
pytest -m "not slow"
Test Performance Goals¶
- Unit tests: < 1 second each
- Integration tests: < 5 seconds each
- E2E tests: < 30 seconds each
- Total test suite: < 5 minutes
CI/CD Integration¶
GitHub Actions Configuration¶
Example test workflow:
name: Tests
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.12'
- name: Install uv
run: pip install uv
- name: Install dependencies
run: uv sync
- name: Run tests
run: uv run pytest --cov=registry --cov-report=xml
- name: Upload coverage
uses: codecov/codecov-action@v3
with:
files: ./coverage.xml
Handling CI/CD Test Failures¶
When tests fail in CI/CD:
- Check the logs:
- Look for error messages
- Check which test failed
-
Review stack traces
-
Reproduce locally:
# Run the same test
uv run pytest tests/path/to/test.py::test_name
# Run with same markers
uv run pytest -m integration
- Common CI/CD issues:
- Missing environment variables
- Service dependencies not running
- File permissions
-
Timing-sensitive tests
-
Fix and verify:
- Make necessary changes
- Run tests locally
- Push fix
- Verify CI/CD passes
Troubleshooting Flaky Tests¶
Identifying Flaky Tests¶
Flaky tests pass/fail intermittently. Signs:
- Tests fail randomly in CI/CD
- Tests pass when run individually
- Tests fail when run with others
- Different results on different machines
Finding Flaky Tests¶
Run tests multiple times:
# Run tests 10 times
for i in {1..10}; do
uv run pytest tests/test_file.py || echo "Failed on iteration $i"
done
# Use pytest-repeat
uv add --dev pytest-repeat
uv run pytest --count=10 tests/test_file.py
Common Causes of Flaky Tests¶
- Timing issues:
# Flaky - Depends on timing
def test_async_operation():
start_background_task()
time.sleep(0.1) # May not be enough
assert task_complete()
# Fixed - Wait for condition
def test_async_operation():
start_background_task()
wait_for_condition(lambda: task_complete(), timeout=5)
assert task_complete()
- Shared state:
# Flaky - Modifies global state
def test_with_global_state():
global_config.update({"key": "value"})
assert process_data() == expected
# Fixed - Isolated state
def test_with_isolated_state(monkeypatch):
test_config = {"key": "value"}
monkeypatch.setattr('module.global_config', test_config)
assert process_data() == expected
- Order dependencies:
# Flaky - Depends on test order
def test_first():
create_resource("test")
def test_second():
resource = get_resource("test") # Assumes test_first ran
assert resource is not None
# Fixed - Independent tests
def test_first():
create_resource("test1")
assert get_resource("test1") is not None
def test_second():
create_resource("test2")
assert get_resource("test2") is not None
- Non-deterministic data:
# Flaky - Random data
def test_with_random_data():
data = generate_random_data()
assert process(data) > 0 # May fail with certain random values
# Fixed - Deterministic data
def test_with_fixed_data():
data = [1, 2, 3, 4, 5]
assert process(data) == 15
Test Isolation Issues¶
Ensuring Test Isolation¶
Tests should not affect each other:
# Bad - Tests share state
class TestSharedState:
shared_list = []
def test_append(self):
self.shared_list.append(1)
assert len(self.shared_list) == 1 # Fails on second run
def test_length(self):
assert len(self.shared_list) == 0 # Fails if test_append ran first
# Good - Tests are isolated
class TestIsolatedState:
def test_append(self):
test_list = []
test_list.append(1)
assert len(test_list) == 1
def test_length(self):
test_list = []
assert len(test_list) == 0
Using Fixtures for Isolation¶
@pytest.fixture
def isolated_list():
"""Provide a fresh list for each test."""
return []
def test_append(isolated_list):
isolated_list.append(1)
assert len(isolated_list) == 1
def test_length(isolated_list):
assert len(isolated_list) == 0
Cleanup After Tests¶
Always cleanup:
@pytest.fixture
def temp_file():
"""Create and cleanup temporary file."""
path = Path("temp.txt")
path.write_text("test")
yield path
# Cleanup
if path.exists():
path.unlink()
@pytest.fixture
def database():
"""Create and cleanup test database."""
db = create_database()
yield db
# Cleanup
db.drop_all_tables()
db.close()
Deprecating Tests¶
When to Deprecate Tests¶
Deprecate tests when:
- Feature is removed
- API is changed significantly
- Test is replaced by better test
- Test is no longer relevant
How to Deprecate Tests¶
- Mark as deprecated:
- Add deprecation warning:
import warnings
def test_legacy_feature():
warnings.warn(
"This test is deprecated and will be removed in v2.0",
DeprecationWarning
)
# Test code...
- Document migration path:
# DEPRECATED: This test is deprecated as of v1.5.0
# Use test_new_implementation in test_new_feature.py instead
# Will be removed in v2.0.0
@pytest.mark.skip(reason="Deprecated - see test_new_implementation")
def test_old_implementation():
pass
Best Practices¶
- Monitor coverage regularly: Check coverage on every PR
- Keep tests fast: Optimize slow tests
- Fix flaky tests immediately: Don't ignore them
- Update tests with code: Tests are part of the codebase
- Document test patterns: Help others write good tests
- Review test code: Tests deserve code review too
- Refactor tests: Keep test code clean
- Delete obsolete tests: Remove tests for removed features
Maintenance Checklist¶
Weekly¶
- Review test failures in CI/CD
- Check for slow tests
- Monitor coverage trends
Monthly¶
- Review and fix flaky tests
- Update test dependencies
- Refactor duplicate test code
- Update test documentation
Quarterly¶
- Audit test coverage
- Remove obsolete tests
- Review test performance
- Update testing guidelines
Summary¶
Key maintenance tasks:
- Monitor and maintain 35% coverage (minimum threshold)
- Keep tests fast and reliable
- Fix flaky tests immediately
- Ensure test isolation
- Update tests with code changes
- Optimize test performance
- Clean up obsolete tests
For more information, see: