Skip to content

[ctm360] Add CyberBlindSpot external-import connector - #6161

Merged
Samuel Hassine (SamuelHassine) merged 17 commits into
OpenCTI-Platform:masterfrom
Khidr6G:feature/ctm360-cyberblindspot-feed
Jun 4, 2026
Merged

[ctm360] Add CyberBlindSpot external-import connector#6161
Samuel Hassine (SamuelHassine) merged 17 commits into
OpenCTI-Platform:masterfrom
Khidr6G:feature/ctm360-cyberblindspot-feed

Conversation

@Khidr6G

@Khidr6G Khidr6G commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

External import connector for CTM360 CyberBlindSpot Digital Risk Protection. Imports incidents, malware logs, breached credentials, card leaks, and domain protection findings. Includes a background status tracker that syncs CBS incident status changes to OpenCTI CaseIncident labels.

Closes #6157

Proposed changes

  • Add the CTM360 CyberBlindSpot Feed external import connector under external-import/ctm360-cyberblindspot-feed/
  • Imports 5 data categories: incidents, malware logs, breached credentials, card leaks, domain protection
  • Background status tracker daemon syncs CBS incident status changes to OpenCTI CaseIncident labels
  • Pydantic configuration via connectors-sdk, deterministic STIX IDs via pycti generators
  • Includes manifest, config schema, logo, mermaid graph, and a full pytest suite

Related issues

Checklist

  • I consider the submitted work as finished
  • I have signed my commits using GPG key.
  • I tested the code for its functionality using different use cases
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

Further comments

Built and tested against OpenCTI 7.260401.0 in a local development environment. Pylint with linter_stix_id_generator plugin passes (no W9101 errors). Black, isort, and flake8 pass clean. A valid CTM360 CyberBlindSpot API key is required for end-to-end testing - happy to coordinate via Slack for verified-status review.

Maintainer review updates

The following changes were applied during review to make the PR merge-ready.

First pass:

  • Fixed a broken import in test_settings.py (ConfigValidationError was imported from connectors_sdk.exceptions, which does not export it) - this was the root cause of the failing test job.
  • Added a pyproject.toml (pythonpath/testpaths) and dropped pytest-cov from test-requirements.txt so run_test.sh actually collects source coverage, matching the cofense-threathq/spycloud convention. Codecov patch coverage was previously 0%.
  • Added a unit test suite (utils, STIX converter, API client with mocked HTTP + retry/pagination, status tracker, connector orchestration). Source coverage is now ~96-100% and codecov/patch passes the 80% gate.
  • Removed a dead normalize_timestamp() call in malware_logs_to_stix and corrected the README rate-limiting section to match the client (honours Retry-After, linear backoff, retries 5xx).
  • Merged master into the branch to bring connectors-sdk/pycti up to date (the stale branch caused uv pip check to fail in CI), repinning connectors-sdk to @master#subdirectory=connectors-sdk and pycti to 7.260529.0.

Second pass (commit fc12bc2) - resolved the remaining inline review threads:

  • normalize_timestamp() now converts offset-aware ISO-8601 timestamps to UTC before formatting the trailing Z (e.g. 18:00+02:00 -> 16:00Z, previously 18:00Z), and treats a numeric 0 as a valid epoch (1970-01-01T00:00:00Z) rather than a missing value. Both behaviours are now covered by exact-value tests.
  • Structured-logging metadata is now passed via the meta= keyword across the connector, status tracker, API client, and converter, matching the pycti logger signature and the rest of the repo.
  • The status tracker is seeded with the incident's normalised status (exposed by the converter) instead of a hard-coded "unknown", so the first poll cycle no longer treats the initial status as a change or re-applies an already-present status:<value> label.

Third pass (commit 89b8f69) - independent full-file review, resolved the last open threads:

  • API client: Retry-After on HTTP 429 is now parsed by a dedicated _parse_retry_after() helper that accepts an integer/float number of seconds and falls back to the configured linear backoff for anything else (a missing header or an HTTP-date, both allowed by RFC 9110). Previously int(...) raised ValueError on a non-integer value and aborted the request loop.
  • Converter: every cyber observable (SCO) - the malware-log IPv4Address/DomainName/EmailAddress, the breached-credential EmailAddress/DomainName/UserAccount, and the domain-protection DomainName/IPv4Address - now carries x_opencti_created_by_ref so OpenCTI attributes the observable to the CyberBlindSpot identity instead of leaving it author-less.
  • Tests: tests/test-requirements.txt now installs -r ../src/requirements.txt so the test env mirrors the connector's pinned runtime deps (repo convention). Added tests for Retry-After parsing (int/float/HTTP-date/missing/negative) and for observable author attribution across all three converters. Local suite: 145 passed; black/isort/flake8 clean.

Fourth pass (commit dbc2edf) - independent full-file re-review plus the remaining Copilot findings, keeping this connector consistent with the sibling CYNA connector (#6162):

  • API client: _extract_items guards incident_list with isinstance(..., list) so a non-list payload can't break extend(); pagination only applies the count/total stop condition when the API reports a positive count (a missing/zero count, e.g. a bare-list response, no longer stops after the first full page); and the 5xx retry guard is widened from (500, 502, 503) to status >= 500 (incl. 504), matching the documented behaviour.
  • normalize_timestamp(): datetime.fromtimestamp() is now wrapped for both numeric and numeric-string epochs, so an out-of-range value falls back to UTC now instead of raising OverflowError/OSError and crashing the import.
  • Connector: the fragile "All" not in str(e) work-error check is replaced with an explicit work_marked_in_error flag, so an unrelated exception whose text contains "All" can no longer skip to_processed(..., in_error=True).
  • Settings: api_key is modelled as SecretStr (read via get_secret_value(), schema format: password / writeOnly); import_interval and status_poll_interval are validated as positive integers (gt=0, schema exclusiveMinimum: 0).
  • README: corrected the troubleshooting note - per-category (partial) and full-cycle failures are both logged at ERROR, not WARNING.
  • Added regression tests (non-list incident_list, count-absent pagination, 504 retry, out-of-range epochs, positive-interval validation, secret API key). Local suite: 155 passed; black/isort/flake8 clean.

Fifth pass (commit 1dd4dd5a) - independent full-file re-review; resolved the final 7 Copilot threads:

  • Converter: records without an API id (malware logs, breached credentials, card leaks, domain protection) no longer fall back to a random uuid.uuid4(). A new _stable_fallback_id() helper derives a deterministic id from stable content fields, so re-imports reuse the same Note/Indicator/external-reference ids instead of creating duplicates on every run.
  • docker-compose.yml: dropped the depends_on: condition: service_healthy block so the sample stays portable when OpenCTI runs in a separate stack or on an external host, and switched to the canonical ${OPENCTI_TOKEN} / ${CONNECTOR_ID} env var names.
  • README: aligned the .env example with the documented env vars, corrected the rate-limiting note (any HTTP 5xx incl. 504, not just 500/502/503), and documented the status-tracking variables.
  • config.yml.sample: added enable_status_tracking / status_poll_interval.
  • Added regression tests asserting the fallback ids are deterministic. Local suite: 162 passed; black/isort/flake8 clean.

Sixth pass (commit 89f3e17) - independent full-file re-review; resolved the last open Copilot thread:

  • README: corrected the Connector Framework Variables table to match the generated connector_config_schema.json - CONNECTOR_SCOPE and CONNECTOR_DURATION_PERIOD are now documented as required with no model default (the docker-compose.yml sample still supplies CTM360-CyberBlindSpot / PT24H), and CONNECTOR_LOG_LEVEL now shows its real default of error (not info). Also clarified that runtime scheduling is driven by CTM360_CBS_IMPORT_INTERVAL while CONNECTOR_DURATION_PERIOD is only required by the SDK base config.
  • Re-reviewed the full changed files (client, converter, connector, tracker, settings, utils): deterministic STIX ids, Retry-After/5xx backoff, STIX value escaping, SecretStr API key, positive-interval validation, and state preservation under the shared lock all verified correct. pycti remains pinned to the latest released 7.260529.0.

Seventh pass (commit 87676905e3) - independent full-file re-review; resolved the last 4 Copilot threads:

  • generate_deterministic_id() now joins its UUID5 seed args with | instead of -. STIX IDs already contain -, so the previous separator let different arg tuples collapse to the same seed (e.g. ('a-b', 'c') and ('a', 'b-c') both yielded a-b-c) and could mint identical relationship ids for distinct objects. | never appears in STIX IDs, so the seed is now unambiguous.
  • Converter: malware_logs_to_stix() de-duplicates the stix2.Malware SDO per family within a conversion run (malware_by_family). Previously one Malware object was emitted per log with a family-derived id, so several logs of the same family produced multiple malware objects sharing an id but with differing external_references/timestamps - a source of conflicting updates on ingestion. The shared Malware now carries a family-stable external reference (malware:<family>) so it stays byte-identical across logs and across runs, while every per-log observable and uses relationship is still emitted and points at it.
  • Added regression tests for the | separator collision, the family-stable Malware external reference, and per-family Malware de-duplication. Local suite: 164 passed; black/isort/flake8 clean.

Eighth pass (commit 3ec13b3792) - independent full-file re-review; resolved the last 3 Copilot threads:

  • Connector: _import_data() now de-duplicates all_objects by STIX id before building the bundle. Every category converter prepends the shared author Identity, so concatenating the per-category outputs shipped that Identity (and any observable surfacing in more than one category) up to five times in a single bundle; it is now sent once.
  • API client: get_incident() validates that the first incident_list element is a dict before returning it (returning {} otherwise), so a non-dict element can no longer crash the CaseStatusTracker / converter, which both call .get(...) on the result.
  • Converter: malware_logs_to_stix() now emits uses relationships from the Malware family to the observed domain and email observables as well as the IPv4-Addr (previously only the IP was linked), and the README's Malware Logs Relationship row was tightened to describe the per-observable uses links exactly.
  • Added regression tests for the bundle de-duplication, the non-dict get_incident element, and the malware->domain/email relationships. Local suite: 168 passed; black/isort/flake8 clean.

Ninth pass (commit b3d8a10) - independent full-file re-review; resolved the last 2 Copilot threads:

  • Converter: a missing/blank/whitespace-only incident type is normalised to Unknown before slugification (inc_type = str(inc.get("type") or "").strip() or "Unknown"), and the type label is only added when slugification yields a non-empty value. Previously an explicit "" (or punctuation-only) type produced an empty label, which later triggered an add_label(label_name="") call during CaseIncident creation.
  • Connector: because every category converter prepends the shared author Identity, an otherwise-empty cycle still produced an author-only list, making the "No new data to import" branch unreachable and shipping an identity-only bundle (plus a work item) every cycle. The bundle is now sent only when there is at least one non-author object; otherwise the cycle is reported as "No new data to import". CaseIncident creation and last_run advancement still run unconditionally.
  • Added regression tests for the blank/punctuation-only type labels and the author-only no-data cycle. Local suite: 171 passed; black/isort/flake8 clean.

Tenth pass (commit dd55ae0) - independent full-file re-review; resolved the last Copilot thread (STIX id generation):

  • Converter: the deterministic SDO/SRO ids now come from the canonical pycti generators instead of connector-local uuid5 seeds, so objects de-duplicate across connectors rather than only within CyberBlindSpot - stix2.Malware uses Malware.generate_id(family), both stix2.Indicators use Indicator.generate_id(pattern) (the pattern is computed before the id), and every stix2.Relationship uses StixCoreRelationship.generate_id(type, source_ref, target_ref). The now-unused generate_deterministic_id helper and its tests were removed.
  • The two stix2.Note ids are the one deliberate exception and keep a content-stable uuid5 seed (documented inline): Note.generate_id(created, content) keys on created, which falls back to import time when the CBS record has no date, so a generator-derived id would mint a fresh Note on every run and duplicate the same finding.
  • Added regression tests asserting the Malware/Indicator/Relationship ids match the pycti generators. Local suite: 170 passed; black/isort/flake8 and the STIX-id pylint plugin clean.

Eleventh pass (commits 88797332e1, 94ca456e1f) - independent full-file re-review; resolved the last open Copilot thread and fixed a CI dependency break:

  • Converter: in breached_credentials_to_stix, indicator_value = email or username is truthy when an email is present but invalid (no @) and the username is empty. The else branch then built the user-account pattern/name from the empty username, producing [user-account:account_login = ''] � an empty, constant pattern that collapses unrelated records onto a single Indicator.generate_id(pattern) id. The branch now falls back to username or email (mirroring the UserAccount.account_login built just above), so the pattern is never empty when an Indicator is created.
  • Dependencies: connectors-sdk on master now requires pycti==7.260604.0, so the test job could no longer resolve against the previous pycti==7.260529.0 pin (unsatisfiable-requirements error). Bumped the pinned pycti to the latest released 7.260604.0 in src/requirements.txt and synced support_version in the manifest and the version references in the README; connectors-sdk stays installed from @master.
  • Added regression tests asserting the non-empty user-account pattern and that two distinct invalid emails map to two distinct Indicator ids. Local suite: 172 passed; black/isort/flake8 clean.

All GitHub Actions checks and Codecov patch/project are green; 0 unresolved review threads.

Notes for the final merge

  • The manifest ships verified: false; end-to-end validation against a live CyberBlindSpot API key is still pending (the contributor offered to coordinate via Slack).
  • Non-blocking follow-up: scheduling still uses CTM360_CBS_IMPORT_INTERVAL while the SDK-required CONNECTOR_DURATION_PERIOD is not used at runtime - kept consistent with the sibling CYNA connector ([ctm360] Add CYNA news external-import connector #6162). Consolidating both CTM360 connectors onto duration_period-based scheduling is a reasonable dedicated follow-up.
  • Resolved in the tenth pass: the Malware/Indicator SDO ids and all relationship ids now use the pycti generators (Malware.generate_id, Indicator.generate_id, StixCoreRelationship.generate_id) for cross-source de-duplication. The two Note ids deliberately keep a content-stable uuid5 seed because Note.generate_id keys on created, which falls back to import time when the record has no date (documented inline).

External import connector for CTM360 CyberBlindSpot Digital Risk Protection. Imports incidents, malware logs, breached credentials, card leaks, and domain protection findings. Includes a background status tracker that syncs CBS incident status changes to OpenCTI CaseIncident labels.

Closes OpenCTI-Platform#6157
@filigran-cla-bot filigran-cla-bot Bot added the cla:pending CLA signature required. label Apr 6, 2026
@filigran-cla-bot

filigran-cla-bot Bot commented Apr 6, 2026

Copy link
Copy Markdown

Contributor License Agreement

CLA signed 💚

Thank you Khidr6G for signing the Contributor License Agreement! Your pull request can now be reviewed and merged.

We appreciate your contribution to Filigran's open source projects! ❤️

This is an automated message from the Filigran CLA Bot.

The settings test imported `ConfigValidationError` from `connectors_sdk.exceptions`, which does not export it (it lives in `connectors_sdk.settings.exceptions` and is re-exported at the `connectors_sdk` top level). This broke pytest collection and failed the `ci/circleci: test` job. Import it from the `connectors_sdk` package public API instead.

Also remove a dead `normalize_timestamp()` call in `malware_logs_to_stix` whose result was discarded, and correct the README rate-limiting section to match the client behaviour (honours `Retry-After`, linear backoff, retries 5xx up to 3 attempts).
@codecov

codecov Bot commented May 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.50146% with 24 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...blindspot-feed/src/ctm360_cbs_client/api_client.py 93.93% 8 Missing ⚠️
...rnal-import/ctm360-cyberblindspot-feed/src/main.py 38.46% 8 Missing ⚠️
...360-cyberblindspot-feed/src/connector/connector.py 96.33% 7 Missing ⚠️
...lindspot-feed/src/connector/case_status_tracker.py 98.76% 1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (a6409cf) and HEAD (94ca456). Click for more details.

HEAD has 103 uploads less than BASE
Flag BASE (a6409cf) HEAD (94ca456)
connectors 105 2
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #6161       +/-   ##
===========================================
- Coverage   27.56%    0.80%   -26.77%     
===========================================
  Files        1875     1835       -40     
  Lines      117302   118104      +802     
===========================================
- Hits        32338      946    -31392     
- Misses      84964   117158    +32194     
Files with missing lines Coverage Δ
...m360-cyberblindspot-feed/src/connector/__init__.py 100.00% <100.00%> (ø)
...rblindspot-feed/src/connector/converter_to_stix.py 100.00% <100.00%> (ø)
...m360-cyberblindspot-feed/src/connector/settings.py 100.00% <100.00%> (ø)
.../ctm360-cyberblindspot-feed/src/connector/utils.py 100.00% <100.00%> (ø)
...erblindspot-feed/src/ctm360_cbs_client/__init__.py 100.00% <100.00%> (ø)
...lindspot-feed/src/connector/case_status_tracker.py 98.76% <98.76%> (ø)
...360-cyberblindspot-feed/src/connector/connector.py 96.33% <96.33%> (ø)
...blindspot-feed/src/ctm360_cbs_client/api_client.py 93.93% <93.93%> (ø)
...rnal-import/ctm360-cyberblindspot-feed/src/main.py 38.46% <38.46%> (ø)

... and 1021 files with indirect coverage changes

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new CTM360 CyberBlindSpot Feed external-import connector to ingest DRP findings (incidents, malware logs, breached credentials, card leaks, domain protection) into OpenCTI, including a background status-tracking daemon for incident label synchronization.

Changes:

  • Introduces the CTM360 CyberBlindSpot connector implementation (API client, STIX conversion, import loop, CaseIncident creation, status tracker).
  • Adds Docker/Docker Compose packaging plus sample configuration and connector metadata (manifest + config schema).
  • Adds basic pytest coverage for importability and settings validation.

Reviewed changes

Copilot reviewed 22 out of 26 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
external-import/ctm360-cyberblindspot-feed/src/ctm360_cbs_client/api_client.py CBS API client with pagination and retry/backoff logic
external-import/ctm360-cyberblindspot-feed/src/connector/connector.py Main connector loop: fetch → convert → send bundle; creates CaseIncidents and starts status tracker
external-import/ctm360-cyberblindspot-feed/src/connector/converter_to_stix.py Converts CBS records into STIX objects/relationships and collects CaseIncident metadata
external-import/ctm360-cyberblindspot-feed/src/connector/case_status_tracker.py Background daemon that polls CBS and updates CaseIncident status labels
external-import/ctm360-cyberblindspot-feed/src/connector/settings.py Pydantic settings models using connectors-sdk
external-import/ctm360-cyberblindspot-feed/src/connector/utils.py Timestamp normalization + deterministic ID helper
external-import/ctm360-cyberblindspot-feed/src/main.py Entry point wiring settings → helper → connector
external-import/ctm360-cyberblindspot-feed/src/config.yml.sample Example YAML configuration for local runs
external-import/ctm360-cyberblindspot-feed/src/requirements.txt Connector runtime dependencies
external-import/ctm360-cyberblindspot-feed/src/connector/init.py Exposes connector + settings for imports
external-import/ctm360-cyberblindspot-feed/src/ctm360_cbs_client/init.py Exposes API client
external-import/ctm360-cyberblindspot-feed/tests/conftest.py Test bootstrap: PYTHONPATH + required env vars
external-import/ctm360-cyberblindspot-feed/tests/test_main.py Smoke test that main imports resolve
external-import/ctm360-cyberblindspot-feed/tests/test_connector/test_settings.py Settings validation + defaults/overrides tests
external-import/ctm360-cyberblindspot-feed/tests/test-requirements.txt Test dependencies
external-import/ctm360-cyberblindspot-feed/Dockerfile Container build for connector
external-import/ctm360-cyberblindspot-feed/entrypoint.sh Container entrypoint
external-import/ctm360-cyberblindspot-feed/docker-compose.yml Example Compose service definition
external-import/ctm360-cyberblindspot-feed/.dockerignore Docker build context exclusions
external-import/ctm360-cyberblindspot-feed/README.md Connector documentation and usage guide
external-import/ctm360-cyberblindspot-feed/docs/stix-mapping.svg Mermaid-generated STIX mapping diagram
external-import/ctm360-cyberblindspot-feed/metadata/connector_manifest.json Connector manifest metadata
external-import/ctm360-cyberblindspot-feed/metadata/connector_config_schema.json Generated config schema for env vars

Comment thread external-import/ctm360-cyberblindspot-feed/src/connector/connector.py Outdated
Comment thread external-import/ctm360-cyberblindspot-feed/src/connector/converter_to_stix.py Outdated
Comment thread external-import/ctm360-cyberblindspot-feed/Dockerfile Outdated
Comment thread external-import/ctm360-cyberblindspot-feed/README.md Outdated
Codecov patch coverage (target 80%) was failing at 0% because the connector shipped no pytest/coverage configuration. `run_test.sh` relies on a `pyproject.toml` `[tool.pytest.ini_options]` block to collect source coverage, so without it no connector lines were ever measured.

- Add `pyproject.toml` (`pythonpath = ["src"]`, `testpaths = ["tests"]`) so `run_test.sh` collects coverage for the connector source packages.
- Drop `pytest-cov` from `test-requirements.txt` so `run_test.sh` measures the whole connector source, matching the cofense-threathq / spycloud convention.
- Add unit tests for `utils`, the STIX converter, the API client (with mocked HTTP and retry/pagination paths), the background status tracker, and the connector orchestration, raising source coverage to ~96-100% (patch well above the 80% gate).
@SamuelHassine Samuel Hassine (SamuelHassine) changed the title [CTM360] Add CTM360 CyberBlindSpot feed connector [CTM360] Add CyberBlindSpot external-import connector May 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full changed files end-to-end (not just the diff) and applied the review fixes directly to the branch. The connector follows the connectors-sdk settings convention, deterministic STIX IDs are used throughout (no linter_stix_id_generator W9101 issues), and the new pytest suite brings source coverage to ~96–100%. All GitHub Actions checks and Codecov patch/project are green. LGTM.

@SamuelHassine

Copy link
Copy Markdown
Member

Khidr6G thanks for the contribution! I pushed a few review fixes to make this merge-ready:

  • Fixed the broken ConfigValidationError import in test_settings.py (it was imported from connectors_sdk.exceptions, which doesn't export it) — this was the root cause of the failing test job.
  • Added a pyproject.toml (pythonpath/testpaths) and dropped pytest-cov from test-requirements.txt so run_test.sh actually collects source coverage (Codecov patch was reporting 0%), matching the cofense-threathq/spycloud convention.
  • Added a unit-test suite (utils, the STIX converter, the API client with mocked HTTP + retry/pagination, the status tracker, and the connector orchestration). Source coverage is now ~96–100% and codecov/patch clears the 80% gate.
  • Removed a dead normalize_timestamp() call in malware_logs_to_stix and corrected the README rate-limiting section to match the client behaviour.
  • Merged master to bring connectors-sdk/pycti up to date — the stale branch was failing uv pip check in the CircleCI test job.

All GitHub Actions checks and Codecov patch/project are green. Heads-up for the final merge: the manifest ships verified: false, and end-to-end validation against a live CyberBlindSpot API key is still pending (the contributor offered to coordinate via Slack).

Apply review feedback:

- Preserve existing connector state (e.g. `tracked_cases` written by the status tracker) when updating `last_run`, and do it under the shared lock to avoid races and dropped tracking state.
- Escape backslashes and single quotes before embedding `email`/`username`/`domain` values into STIX patterns so values with quotes cannot produce invalid (un-ingestable) patterns.
- Use `pycti.Identity.generate_id` for the author Identity so the STIX author ID matches OpenCTI's deterministic ID generation instead of a hard-coded UUID.
- Remove the no-op Docker `HEALTHCHECK` that always exited 0 and provided false health signals.
- Add tests for STIX value escaping and for state-key preservation during import.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving after addressing all review feedback (including the Copilot suggestions): connector state is now preserved/locked when updating last_run, STIX pattern values are escaped, the author Identity uses pycti.Identity.generate_id, and the no-op Docker healthcheck was removed. All seven review threads are resolved. Tests pass (134) with ~96–100% source coverage, and all GitHub Actions checks plus Codecov patch/project are green. LGTM.

@SamuelHassine Samuel Hassine (SamuelHassine) changed the title [CTM360] Add CyberBlindSpot external-import connector [ctm360] Add CyberBlindSpot external-import connector May 30, 2026
Bump `pycti` to the latest release (`7.260529.0`) and install `connectors-sdk` from the repo `master` subdirectory instead of the `7.260401.0` tag, so the Docker build resolves a consistent set (the pinned tag pulled `pycti==7.260401.0` and would conflict). Sync the README and manifest `support_version` references accordingly.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 28 out of 33 changed files in this pull request and generated 8 comments.

Comment thread external-import/ctm360-cyberblindspot-feed/src/connector/utils.py
Comment thread external-import/ctm360-cyberblindspot-feed/src/requirements.txt
Comment thread external-import/ctm360-cyberblindspot-feed/src/connector/utils.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving after the dependency bump (pycti pinned to the latest 7.260529.0 + connectors-sdk from master, README/manifest synced). CI is green and all review threads remain resolved.

Apply the outstanding reviewer feedback to make the connector merge-ready:

- normalize_timestamp(): convert offset-aware ISO-8601 timestamps to UTC
  before formatting the trailing "Z" so values are no longer silently
  shifted (e.g. 18:00+02:00 now yields 16:00Z), and treat numeric 0 as a
  valid epoch (1970-01-01T00:00:00Z) instead of a missing value.
- Tighten the timestamp tests to assert exact UTC output for positive and
  negative offsets and to cover the epoch-zero case.
- Pass structured logging metadata via the meta= keyword across the
  connector, status tracker, API client, and converter, matching the pycti
  logger signature and the rest of the repo.
- Seed the status tracker with the incident's normalised status instead of a
  hard-coded "unknown", so the first poll cycle does not re-apply an
  already-present status:<value> label.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving after the second review pass. All inline review threads are addressed and resolved, the new behaviour is covered by tests, and every GitHub Actions check plus Codecov patch/project is green.

Independent review of the full changed files (not just the diff) confirmed the connector is sound: partial-import error handling, state preservation under the shared lock, STIX pattern escaping, and the status-tracker lifecycle all look correct. The one timezone correctness bug surfaced by the reviewer is now fixed and tested.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eighth review pass (commit 3ec13b3792) — independent full-file re-review; resolved the last 3 open Copilot threads.

  • Bundle de-duplication: _import_data() now drops already-seen STIX ids before building the bundle, so the shared author Identity (prepended by every category converter) — and any observable surfacing in more than one category — is sent once instead of up to five times per cycle.
  • get_incident() now validates that the extracted incident_list element is a dict before returning it, so a non-dict element can no longer crash the CaseStatusTracker / converter.
  • malware_logs_to_stix() now links the Malware family to every observed observable (IP, domain, email) via uses, matching the README (whose Malware Logs Relationship row was tightened accordingly).

Re-verified the full changed files (client, converter, connector, tracker, settings, utils): deterministic STIX ids, Retry-After/5xx backoff, STIX value escaping, SecretStr API key, positive-interval validation, and last_run/tracked_cases state preservation under the shared lock all check out; pycti stays pinned to the latest released 7.260529.0. Local suite: 168 passed; black/isort/flake8 clean. All GitHub Actions checks and Codecov patch/project are green; 0 unresolved review threads. LGTM.

@SamuelHassine

Copy link
Copy Markdown
Member

Eighth review pass (3ec13b3792) — resolved the final 3 Copilot threads and re-reviewed the full changed files:

  • Bundle de-duplication (fix): _import_data() de-duplicates STIX objects by id before bundling, so the shared author Identity (prepended by every category converter) is sent once rather than up to five times per cycle.
  • get_incident() hardening (fix): the first incident_list element is now validated as a dict before being returned, so a non-dict element can no longer crash the status tracker / converter.
  • Malware relationships (fix): malware_logs_to_stix() now emits uses relationships from the Malware family to the observed domain and email as well as the IPv4-Addr, matching the README.

Added regression tests for all three. Local suite: 168 passed; black/isort/flake8 clean. All GitHub Actions checks and Codecov patch/project are green; 0 unresolved review threads.

Non-blocking for the final merge: the manifest still ships verified: false — end-to-end validation against a live CyberBlindSpot API key is still pending (the contributor offered to coordinate via Slack). Scheduling stays on CTM360_CBS_IMPORT_INTERVAL for parity with the sibling CYNA connector (#6162), and a few converter SDO/relationship ids still use raw uuid5 rather than the pycti generators — both tracked as follow-ups.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 28 out of 33 changed files in this pull request and generated 2 comments.

Comment thread external-import/ctm360-cyberblindspot-feed/src/connector/connector.py Outdated
Resolve the two remaining review findings on the CyberBlindSpot connector.

Converter: a missing/blank/whitespace-only incident "type" is now normalised to "Unknown" before slugification, and the type label is only added when slugification yields a non-empty value. Previously an empty (or punctuation-only) type produced an empty label, which triggered an add_label(label_name="") call during CaseIncident creation.

Connector: every category converter prepends the shared author Identity, so a cycle where all endpoints returned no data still produced an author-only list and shipped an identity-only bundle (plus a work item) every cycle. The bundle is now sent only when there is at least one non-author object; otherwise the cycle is reported as "No new data to import". CaseIncident creation and state advancement are unchanged.

Added regression tests for both behaviours. Local suite: 171 passed; black/isort/flake8 clean.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent full-file re-review of the CyberBlindSpot connector (client, converter, connector, status tracker, settings, utils, tests). The two remaining Copilot findings are addressed in b3d8a10: blank/punctuation-only incident types no longer emit empty labels, and an author-only cycle is now treated as "no data" so the connector stops shipping an identity-only bundle (plus a work item) every cycle. Deterministic STIX ids, Retry-After / 5xx backoff, pagination guards, STIX value escaping, the SecretStr API key, positive-interval validation, and state preservation under the shared lock all verified correct. pycti is pinned to the latest released 7.260529.0 and connectors-sdk is installed from @master. All checks are green, 0 unresolved threads. LGTM.

@SamuelHassine

Copy link
Copy Markdown
Member

Ninth review pass (commit b3d8a10) — resolved the final 2 Copilot threads after an independent full-file re-review:

  • Empty incident-type label: a missing/blank/whitespace-only type is normalised to Unknown before slugification, and the type label is only added when slugification yields a non-empty value, so no empty label can reach add_label(label_name="").
  • Author-only bundle: every converter prepends the shared author Identity, so empty cycles were shipping an identity-only bundle and a work item each run. The bundle is now sent only when there is at least one non-author object; otherwise the cycle reports "No new data to import". CaseIncident creation and last_run advancement are unchanged.
  • Added regression tests for both. Local suite: 171 passed; black/isort/flake8 clean.

Status: all GitHub Actions checks + Codecov are green, 0 unresolved review threads, MERGEABLE. Pre-merge note carried over from the description — the manifest still ships verified: false; end-to-end validation against a live CyberBlindSpot API key remains the only outstanding item.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 28 out of 33 changed files in this pull request and generated 1 comment.

Replace the connector-local uuid5 seeds for the Malware/Indicator SDOs and
the custom generate_deterministic_id relationship helper with the canonical
pycti generators (Malware.generate_id, Indicator.generate_id,
StixCoreRelationship.generate_id) so these objects de-duplicate across
connectors, not just within CyberBlindSpot.

Notes intentionally keep their content-stable uuid5 seed: Note.generate_id
keys on `created`, which falls back to import time when the CBS record has no
date, so a generator-derived id would mint a fresh Note on every run and
duplicate the same breach. Drop the now-unused generate_deterministic_id
helper and its tests, and add regression tests asserting the generated ids.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent full-file re-review of the connector (client, converter, connector orchestration, status tracker, settings, utils) plus the test suite. Logic is sound: deterministic STIX ids, Retry-After/5xx backoff, pagination stop conditions, STIX value escaping, SecretStr API key, positive-interval validation, bundle de-duplication, and state preservation under the shared lock all check out.

The one open thread (raw uuid5 ids vs pycti generators) is now addressed: Malware/Indicator SDOs and all relationships use the canonical pycti generators for cross-connector de-duplication, with the two Notes kept on a content-stable seed (documented) because Note.generate_id keys on a created that falls back to import time. Local suite 170 passed; black/isort/flake8 and the STIX-id pylint plugin clean; all GitHub Actions checks and Codecov patch/project green.

Approving. Only remaining non-CI item is end-to-end validation against a live CyberBlindSpot API key (verified: false in the manifest).

@SamuelHassine

Copy link
Copy Markdown
Member

Review & fix summary

  • Independent full-file re-review of the connector + tests — no new correctness issues found.
  • Resolved the last open Copilot thread (STIX id generation) in dd55ae0:
    • MalwareMalware.generate_id(family), both Indicators → Indicator.generate_id(pattern), every RelationshipStixCoreRelationship.generate_id(...) so objects de-duplicate across connectors; dropped the now-unused generate_deterministic_id helper + its tests.
    • The two Note ids deliberately keep a content-stable uuid5 seed (documented inline): Note.generate_id(created, content) keys on created, which falls back to import time when the record has no date and would otherwise duplicate the Note on every run.
    • Added regression tests asserting the generated ids match the pycti generators.
  • Local suite: 170 passed; black/isort/flake8 and the STIX-id pylint plugin clean; coverage 100% on the changed files.
  • All GitHub Actions checks + Codecov patch (96.49% of diff) / project are green; 0 unresolved review threads.

Non-CI item for the maintainer: the manifest still ships verified: false — end-to-end validation against a live CyberBlindSpot API key is pending.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 28 out of 33 changed files in this pull request and generated 1 comment.

Comment thread external-import/ctm360-cyberblindspot-feed/src/connector/converter_to_stix.py Outdated
In breached_credentials_to_stix, `indicator_value = email or username` is truthy when an email is present but invalid (no "@") and the username is empty. The else-branch then built the user-account pattern/name from the empty username, producing `[user-account:account_login = '']` — an empty, constant pattern that collapses unrelated records onto a single `Indicator.generate_id(pattern)` id and breaks ingestion expectations.

Fall back to `username or email` for the account-login pattern and name (mirroring the UserAccount `account_login` already built above), so the pattern is never empty when an Indicator is created. Add regression tests asserting the non-empty pattern and that distinct invalid emails map to distinct Indicators.
connectors-sdk on master now requires pycti==7.260604.0, so the test environment (which installs the local master connectors-sdk alongside this connector's pinned runtime deps) could no longer resolve against the previous pycti==7.260529.0 pin, failing the connector test job with an unsatisfiable-requirements error.

Bump the pinned pycti to the latest released 7.260604.0 in src/requirements.txt and sync the support_version in the manifest and the version references in the README. connectors-sdk is already installed from @master, so it stays compatible.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent full-file re-review of the connector (client, converter, connector orchestration, status tracker, settings, utils) plus the test suite. Logic is sound: deterministic STIX ids via the pycti generators, Retry-After/5xx backoff, pagination stop conditions, STIX value escaping, SecretStr API key, positive-interval validation, bundle de-duplication, author-only "no data" handling, and last_run/tracked_cases state preservation under the shared lock all check out. The status-tracker status:<value> label naming is consistent with the converter, so no label drift on the first status change.

The last open Copilot thread is now resolved: breached_credentials_to_stix no longer builds an empty [user-account:account_login = ''] Indicator pattern when an email is present-but-invalid and the username is empty — it falls back to username or email, matching the UserAccount.account_login. Also bumped the pinned pycti to the latest released 7.260604.0 (the master connectors-sdk now requires it; the previous 7.260529.0 pin made the test env unresolvable).

Local suite: 172 passed; black/isort/flake8 clean. All GitHub Actions checks and Codecov patch/project are green; 0 unresolved review threads. Approving. The only remaining non-CI item is end-to-end validation against a live CyberBlindSpot API key (verified: false in the manifest).

@SamuelHassine

Copy link
Copy Markdown
Member

Review & fix summary

  • Independent full-file re-review of the connector + tests — no new correctness issues found.
  • Resolved the last open Copilot thread (88797332e1): breached_credentials_to_stix no longer mints an empty [user-account:account_login = ''] Indicator pattern when an email is present-but-invalid (no @) and the username is empty. The else branch now falls back to username or email, mirroring the UserAccount.account_login, so unrelated records can't collapse onto a single Indicator.generate_id(pattern) id. Added regression tests for the non-empty pattern and for distinct invalid emails mapping to distinct Indicator ids.
  • CI dependency fix (94ca456e1f): connectors-sdk on master now requires pycti==7.260604.0, which made the test env unresolvable against the previous 7.260529.0 pin. Bumped the pinned pycti to 7.260604.0 in src/requirements.txt and synced support_version (manifest) and the README version references; connectors-sdk stays installed from @master.
  • Local suite: 172 passed; black/isort/flake8 clean. All GitHub Actions checks + Codecov patch/project are green; 0 unresolved review threads; MERGEABLE.

Non-CI item for the maintainer: the manifest still ships verified: false — end-to-end validation against a live CyberBlindSpot API key is the only outstanding item.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 28 out of 33 changed files in this pull request and generated no new comments.

@SamuelHassine
Samuel Hassine (SamuelHassine) merged commit caed556 into OpenCTI-Platform:master Jun 4, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

filigran team Item from the Filigran team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[New Connector] CTM360 CyberBlindSpot Feed (External Import)

5 participants