|
| 1 | +# DefectDojo Development Guide |
| 2 | + |
| 3 | +## Project Overview |
| 4 | + |
| 5 | +DefectDojo is a Django application (`dojo` app) for vulnerability management. The codebase is undergoing a modular reorganization to move from monolithic files toward self-contained domain modules. |
| 6 | + |
| 7 | +## Module Reorganization |
| 8 | + |
| 9 | +### Reference Pattern: `dojo/url/` |
| 10 | + |
| 11 | +All domain modules should match the structure of `dojo/url/`. This is the canonical example of a fully reorganized module. |
| 12 | + |
| 13 | +``` |
| 14 | +dojo/{module}/ |
| 15 | +├── __init__.py # import dojo.{module}.admin # noqa: F401 |
| 16 | +├── models.py # Domain models, constants, factory methods |
| 17 | +├── admin.py # @admin.register() for the module's models |
| 18 | +├── services.py # Business logic (no HTTP concerns) |
| 19 | +├── queries.py # Complex DB aggregations/annotations |
| 20 | +├── signals.py # Django signal handlers |
| 21 | +├── [manager.py] # Custom QuerySet/Manager if needed |
| 22 | +├── [validators.py] # Field-level validators if needed |
| 23 | +├── [helpers.py] # Async task wrappers, tag propagation, etc. |
| 24 | +├── ui/ |
| 25 | +│ ├── __init__.py # Empty |
| 26 | +│ ├── forms.py # Django ModelForms |
| 27 | +│ ├── filters.py # UI-layer django-filter classes |
| 28 | +│ ├── views.py # Thin view functions — delegates to services.py |
| 29 | +│ └── urls.py # URL routing |
| 30 | +└── api/ |
| 31 | + ├── __init__.py # path = "{module}" |
| 32 | + ├── serializer.py # DRF serializers |
| 33 | + ├── views.py # API ViewSets — delegates to services.py |
| 34 | + ├── filters.py # API-layer filters |
| 35 | + └── urls.py # add_{module}_urls(router) registration |
| 36 | +``` |
| 37 | + |
| 38 | +### Architecture Principles |
| 39 | + |
| 40 | + |
| 41 | +**services.py is the critical layer**: Both `ui/views.py` and `api/views.py` call `services.py` for business logic. Services accept domain objects and primitives — never request/response objects, forms, or serializers. |
| 42 | + |
| 43 | +**Backward-compatible re-exports**: When moving code out of monolithic files (`dojo/models.py`, `dojo/forms.py`, `dojo/filters.py`, `dojo/api_v2/serializers.py`, `dojo/api_v2/views.py`), always leave a re-export at the original location: |
| 44 | +```python |
| 45 | +from dojo.{module}.models import {Model} # noqa: F401 -- backward compat |
| 46 | +``` |
| 47 | +Never remove re-exports until all consumers are updated in a dedicated cleanup pass. |
| 48 | + |
| 49 | +### Current State |
| 50 | + |
| 51 | +Modules in various stages of reorganization: |
| 52 | + |
| 53 | +| Module | models.py | services.py | ui/ | api/ | Status | |
| 54 | +|--------|-----------|-------------|-----|------|--------| |
| 55 | +| **url** | In module | N/A | Done | Done | **Complete** | |
| 56 | +| **location** | In module | N/A | N/A | Done | **Complete** | |
| 57 | +| **product_type** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work | |
| 58 | +| **test** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work | |
| 59 | +| **engagement** | In dojo/models.py | Partial (32 lines) | Partial (views at root) | In dojo/api_v2/ | Needs work | |
| 60 | +| **product** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work | |
| 61 | +| **finding** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work | |
| 62 | + |
| 63 | +### Monolithic Files Being Decomposed |
| 64 | + |
| 65 | +These files still contain code for multiple modules. Extract code to the target module's subdirectory and leave a re-export stub. |
| 66 | + |
| 67 | +- `dojo/models.py` (4,973 lines) — All model definitions |
| 68 | +- `dojo/forms.py` (4,127 lines) — All Django forms |
| 69 | +- `dojo/filters.py` (4,016 lines) — All UI and API filter classes |
| 70 | +- `dojo/api_v2/serializers.py` (3,387 lines) — All DRF serializers |
| 71 | +- `dojo/api_v2/views.py` (3,519 lines) — All API viewsets |
| 72 | + |
| 73 | +--- |
| 74 | + |
| 75 | +## Reorganization Playbook |
| 76 | + |
| 77 | +When asked to reorganize a module, follow these phases in order. Each phase should be independently verifiable. |
| 78 | + |
| 79 | +### Phase 0: Pre-Flight (Read-Only) |
| 80 | + |
| 81 | +Before any changes, identify all code to extract: |
| 82 | + |
| 83 | +```bash |
| 84 | +# 1. Model classes and line ranges in dojo/models.py |
| 85 | +grep -n "class {Model}" dojo/models.py |
| 86 | + |
| 87 | +# 2. Form classes in dojo/forms.py |
| 88 | +grep -n "class.*{Module}" dojo/forms.py |
| 89 | +grep -n "model = {Model}" dojo/forms.py |
| 90 | + |
| 91 | +# 3. Filter classes in dojo/filters.py |
| 92 | +grep -n "class.*{Module}\|class.*{Model}" dojo/filters.py |
| 93 | + |
| 94 | +# 4. Serializer classes |
| 95 | +grep -n "class.*{Model}" dojo/api_v2/serializers.py |
| 96 | + |
| 97 | +# 5. ViewSet classes |
| 98 | +grep -n "class.*{Model}\|class.*{Module}" dojo/api_v2/views.py |
| 99 | + |
| 100 | +# 6. Admin registrations |
| 101 | +grep -n "admin.site.register({Model}" dojo/models.py |
| 102 | + |
| 103 | +# 7. All import sites (to verify backward compat) |
| 104 | +grep -rn "from dojo.models import.*{Model}" dojo/ unittests/ |
| 105 | + |
| 106 | +# 8. Business logic in current views |
| 107 | +# Scan dojo/{module}/views.py for: .save(), .delete(), create_notification(), |
| 108 | +# jira_helper.*, dojo_dispatch_task(), multi-model workflows |
| 109 | +``` |
| 110 | + |
| 111 | +### Phase 1: Extract Models |
| 112 | + |
| 113 | +1. Create `dojo/{module}/models.py` with the model class(es) and associated constants |
| 114 | +2. Create `dojo/{module}/admin.py` with `admin.site.register()` calls (remove from `dojo/models.py`) |
| 115 | +3. Update `dojo/{module}/__init__.py` to `import dojo.{module}.admin # noqa: F401` |
| 116 | +4. Add re-exports in `dojo/models.py` |
| 117 | +5. Remove original model code (keep re-export line) |
| 118 | + |
| 119 | +**Import rules for models.py:** |
| 120 | +- Upward FKs (e.g., Test -> Engagement): import from `dojo.models` if not yet extracted, or `dojo.{module}.models` if already extracted |
| 121 | +- Downward references (e.g., Product_Type querying Finding): use lazy imports inside method bodies |
| 122 | +- Shared utilities (`copy_model_util`, `_manage_inherited_tags`, `get_current_date`, etc.): import from `dojo.models` |
| 123 | +- Do NOT set `app_label` in Meta — all models inherit `dojo` app_label automatically |
| 124 | + |
| 125 | +**Verify:** |
| 126 | +```bash |
| 127 | +python manage.py check |
| 128 | +python manage.py makemigrations --check |
| 129 | +python -c "from dojo.{module}.models import {Model}" |
| 130 | +python -c "from dojo.models import {Model}" |
| 131 | +``` |
| 132 | + |
| 133 | +### Phase 2: Extract Services |
| 134 | + |
| 135 | +Create `dojo/{module}/services.py` with business logic extracted from UI views. |
| 136 | + |
| 137 | +**What belongs in services.py:** |
| 138 | +- State transitions (close, reopen, status changes) |
| 139 | +- Multi-step creation/update workflows |
| 140 | +- External integration calls (JIRA, GitHub) |
| 141 | +- Notification dispatching |
| 142 | +- Copy/clone operations |
| 143 | +- Bulk operations |
| 144 | +- Merge operations |
| 145 | + |
| 146 | +**What stays in views:** |
| 147 | +- HTTP request/response handling |
| 148 | +- Form instantiation and validation |
| 149 | +- Serialization/deserialization |
| 150 | +- Authorization checks (`@user_is_authorized`, `user_has_permission_or_403`) |
| 151 | +- Template rendering, redirects |
| 152 | +- Pagination, breadcrumbs |
| 153 | + |
| 154 | +**Service function pattern:** |
| 155 | +```python |
| 156 | +def close_engagement(engagement: Engagement, user: User) -> Engagement: |
| 157 | + engagement.active = False |
| 158 | + engagement.status = "Completed" |
| 159 | + engagement.save() |
| 160 | + if jira_helper.get_jira_project(engagement): |
| 161 | + dojo_dispatch_task(jira_helper.close_epic, engagement.id, push_to_jira=True) |
| 162 | + return engagement |
| 163 | +``` |
| 164 | + |
| 165 | +Update UI views and API viewsets to call the service instead of containing logic inline. |
| 166 | + |
| 167 | +### Phase 3: Extract Forms to `ui/forms.py` |
| 168 | + |
| 169 | +1. Create `dojo/{module}/ui/__init__.py` (empty) |
| 170 | +2. Create `dojo/{module}/ui/forms.py` — move form classes from `dojo/forms.py` |
| 171 | +3. Add re-exports in `dojo/forms.py` |
| 172 | + |
| 173 | +### Phase 4: Extract UI Filters to `ui/filters.py` |
| 174 | + |
| 175 | +1. Create `dojo/{module}/ui/filters.py` — move module-specific filters from `dojo/filters.py` |
| 176 | +2. Shared base classes (`DojoFilter`, `DateRangeFilter`, `ReportBooleanFilter`) stay in `dojo/filters.py` |
| 177 | +3. Add re-exports in `dojo/filters.py` |
| 178 | + |
| 179 | +### Phase 5: Move UI Views/URLs into `ui/` |
| 180 | + |
| 181 | +1. Move `dojo/{module}/views.py` -> `dojo/{module}/ui/views.py` |
| 182 | +2. Move `dojo/{module}/urls.py` -> `dojo/{module}/ui/urls.py` |
| 183 | +3. Update URL imports: |
| 184 | + - product: update `dojo/asset/urls.py` |
| 185 | + - product_type: update `dojo/organization/urls.py` |
| 186 | + - others: update the include in `dojo/urls.py` |
| 187 | + |
| 188 | +### Phase 6: Extract API Serializers to `api/serializer.py` |
| 189 | + |
| 190 | +1. Create `dojo/{module}/api/__init__.py` with `path = "{module}"` |
| 191 | +2. Create `dojo/{module}/api/serializer.py` — move from `dojo/api_v2/serializers.py` |
| 192 | +3. Add re-exports in `dojo/api_v2/serializers.py` |
| 193 | + |
| 194 | +### Phase 7: Extract API Filters to `api/filters.py` |
| 195 | + |
| 196 | +1. Create `dojo/{module}/api/filters.py` — move `Api{Model}Filter` from `dojo/filters.py` |
| 197 | +2. Add re-exports |
| 198 | + |
| 199 | +### Phase 8: Extract API ViewSets to `api/views.py` |
| 200 | + |
| 201 | +1. Create `dojo/{module}/api/views.py` — move from `dojo/api_v2/views.py` |
| 202 | +2. Add re-exports in `dojo/api_v2/views.py` |
| 203 | + |
| 204 | +### Phase 9: Extract API URL Registration |
| 205 | + |
| 206 | +1. Create `dojo/{module}/api/urls.py`: |
| 207 | + ```python |
| 208 | + from dojo.{module}.api import path |
| 209 | + from dojo.{module}.api.views import {ViewSet} |
| 210 | + |
| 211 | + def add_{module}_urls(router): |
| 212 | + router.register(path, {ViewSet}, path) |
| 213 | + return router |
| 214 | + ``` |
| 215 | +2. Update `dojo/urls.py` — replace `v2_api.register(...)` with `add_{module}_urls(v2_api)` |
| 216 | + |
| 217 | +### After Each Phase: Verify |
| 218 | + |
| 219 | +```bash |
| 220 | +python manage.py check |
| 221 | +python manage.py makemigrations --check |
| 222 | +python -m pytest unittests/ -x --timeout=120 |
| 223 | +``` |
| 224 | + |
| 225 | +--- |
| 226 | + |
| 227 | +## Cross-Module Dependencies |
| 228 | + |
| 229 | +The model hierarchy is: Product_Type -> Product -> Engagement -> Test -> Finding |
| 230 | + |
| 231 | +Extract in this order (top to bottom) so that upward FKs can import from already-extracted modules. The recommended order is: product_type, test, engagement, product, finding. |
| 232 | + |
| 233 | +For downward references (e.g., Product_Type's cached properties querying Finding), always use lazy imports: |
| 234 | +```python |
| 235 | +@cached_property |
| 236 | +def critical_present(self): |
| 237 | + from dojo.models import Finding # lazy import |
| 238 | + return Finding.objects.filter(test__engagement__product__prod_type=self, severity="Critical").exists() |
| 239 | +``` |
| 240 | + |
| 241 | +--- |
| 242 | + |
| 243 | +## Key Technical Details |
| 244 | + |
| 245 | +- **Single Django app**: Everything is under `app_label = "dojo"`. Moving models to subdirectories does NOT require migration changes. |
| 246 | +- **Model discovery**: Triggered by `__init__.py` importing `admin.py`, which imports `models.py`. This is the same chain `dojo/url/` uses. |
| 247 | +- **Signal registration**: Handled in `dojo/apps.py` via `import dojo.{module}.signals`. Already set up for test, engagement, product, product_type. |
| 248 | +- **Watson search**: Uses `self.get_model("Product")` in `apps.py` — works via Django's model registry regardless of file location. |
| 249 | +- **Admin registration**: Currently at the bottom of `dojo/models.py` (lines 4888-4973). Must be moved to `{module}/admin.py` and removed from `dojo/models.py` to avoid `AlreadyRegistered` errors. |
0 commit comments