fix: only heartbeats received in live stream; checkpoint collection missing or not created

[socsocsoc-lab]

Hello,

I am trying to integrate the Splunk TA-OpenCTI add-on with OpenCTI via the live stream API. However, when I'm trying to look for some data in the index (named opencti) i have empty index.

Additional details:

• The live stream seems to be configured correctly, with public access enabled.
  

• The input configuration in Splunk seems fine (inputs.conf is set and active).
  

• Logs show a recurring 404 error when the add-on tries to access the KV Store collection TA_opencti_add_on_checkpointer for checkpoint data:
  

GET /servicesNS/nobody/TA-opencti-add-on/storage/collections/data/TA_opencti_add_on_checkpointer/opencti HTTP/1.1" 404

Additionally, some log entries contain:

200 None "GET //stream/4a03ebca-b497-44bb-bd78-f1d4edbceed0?recover=2025-07-24T10:12:19Z HTTP/1.1"

• There is no collections.conf file in the add-on directory to define the KV Store collection.
  

• The collection TA_opencti_add_on_checkpointer does not appear in Splunk's Lookup Table Files.
  

• MongoDB (KV Store backend) is running and accessible on port 8191.
  

• Splunkd REST API is running on port 8089.
  

• Restarting Splunk or reloading the app does not create the missing collection or checkpoint.
  

• I suspect that because the collection does not exist or is not properly defined, the add-on cannot save or read checkpoints, resulting in it always trying to stream from the beginning or only receiving heartbeats.
  

• No relevant checkpoint saved logs appear in ta_opencti_add_on.log.
  

• SSL certificate errors appeared earlier but have been addressed (or are under investigation).
  

Could you please advise:

• Does the add-on require a manually created collections.conf file to define the checkpoint collection?
  

• Are there any additional configurations needed for the KV Store collections?
  

• Is the 404 error expected behavior on first run, and should the collection be auto-created later?
  

• Could missing checkpoint saving cause the stream to only deliver heartbeats?
  

• Any tips on how to troubleshoot and confirm checkpoint creation and usage?
  

Here is full log from initialization:

2025-07-25 11:09:04,548 INFO pid=2402735 tid=MainThread file=base_modinput.py:log_info:295 | OpenCTI data input module start
2025-07-25 11:09:04,549 INFO pid=2402735 tid=MainThread file=setup_util.py:log_info:117 | Proxy is not enabled!
2025-07-25 11:09:04,549 DEBUG pid=2402735 tid=MainThread file=base_modinput.py:log_debug:288 | proxy configuration: None
2025-07-25 11:09:04,549 INFO pid=2402735 tid=MainThread file=base_modinput.py:log_info:295 | going to fetch data of OpenCTI stream.id: 4a03ebca-b497-44bb-bd78-f1d4edbceed0
2025-07-25 11:09:04,549 DEBUG pid=2402735 tid=MainThread file=binding.py:get:735 | GET request to https://localhost:8089/servicesNS/nobody/TA-opencti-add-on/storage/collections/config/opencti_indicators (body: {})
2025-07-25 11:09:04,557 DEBUG pid=2402735 tid=MainThread file=binding.py:new_f:78 | Operation took 0:00:00.007618
2025-07-25 11:09:04,559 INFO pid=2402735 tid=MainThread file=splunk_rest_client.py:_request_handler:99 | Use HTTP connection pooling
2025-07-25 11:09:04,560 DEBUG pid=2402735 tid=MainThread file=binding.py:get:735 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-opencti-add-on/storage/collections/config/TA_opencti_add_on_checkpointer (body: {})
2025-07-25 11:09:04,561 DEBUG pid=2402735 tid=MainThread file=connectionpool.py:_new_conn:1019 | Starting new HTTPS connection (1): 127.0.0.1:8089
2025-07-25 11:09:04,573 DEBUG pid=2402735 tid=MainThread file=connectionpool.py:_make_request:474 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-opencti-add-on/storage/collections/config/TA_opencti_add_on_checkpointer HTTP/1.1" 200 5607
2025-07-25 11:09:04,573 DEBUG pid=2402735 tid=MainThread file=binding.py:new_f:78 | Operation took 0:00:00.013395
2025-07-25 11:09:04,573 DEBUG pid=2402735 tid=MainThread file=binding.py:get:735 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-opencti-add-on/storage/collections/config/ (body: {'count': -1, 'offset': 0, 'search': 'TA_opencti_add_on_checkpointer'})
2025-07-25 11:09:04,578 DEBUG pid=2402735 tid=MainThread file=connectionpool.py:_make_request:474 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-opencti-add-on/storage/collections/config/?count=-1&offset=0&search=TA_opencti_add_on_checkpointer HTTP/1.1" 200 4805
2025-07-25 11:09:04,578 DEBUG pid=2402735 tid=MainThread file=binding.py:new_f:78 | Operation took 0:00:00.005023
2025-07-25 11:09:04,579 DEBUG pid=2402735 tid=MainThread file=binding.py:get:735 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-opencti-add-on/storage/collections/data/TA_opencti_add_on_checkpointer/opencti (body: {})
2025-07-25 11:09:04,583 DEBUG pid=2402735 tid=MainThread file=connectionpool.py:_make_request:474 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-opencti-add-on/storage/collections/data/TA_opencti_add_on_checkpointer/opencti HTTP/1.1" 404 140
2025-07-25 11:09:04,583 INFO pid=2402735 tid=MainThread file=base_modinput.py:log_info:295 | checkpoint State: None
2025-07-25 11:09:04,583 INFO pid=2402735 tid=MainThread file=base_modinput.py:log_info:295 | No state, going to initialize it
2025-07-25 11:09:04,583 INFO pid=2402735 tid=MainThread file=base_modinput.py:log_info:295 | Initialized state: {'start_from': '1748243344000-0', 'recover_until': '2025-07-25T09:09:04Z'}
2025-07-25 11:09:04,584 DEBUG pid=2402735 tid=MainThread file=connectionpool.py:_new_conn:1019 | Starting new HTTPS connection (1): opencti.soc.domain.pl:443
2025-07-25 11:09:04,596 DEBUG pid=2402735 tid=MainThread file=connectionpool.py:_make_request:474 | https://opencti.soc.domain.pl:443 "GET //stream/4a03ebca-b497-44bb-bd78-f1d4edbceed0?recover=2025-07-25T09:09:04Z HTTP/1.1" 200 None
2025-07-25 11:09:07,601 DEBUG pid=2402735 tid=MainThread file=connectionpool.py:_new_conn:1019 | Starting new HTTPS connection (1): opencti.soc.domain.pl:443
2025-07-25 11:09:07,614 DEBUG pid=2402735 tid=MainThread file=connectionpool.py:_make_request:474 | https://opencti.soc.domain.pl:443 "GET //stream/4a03ebca-b497-44bb-bd78-f1d4edbceed0?recover=2025-07-25T09:09:04Z HTTP/1.1" 200 None

Thank you for your support!

[CTIBurn0ut]

socsocsoc-lab if you are using version 1.1.5, then no data is ever written to the index, the KV store is directly managed by the Modular Input script.

[Imsaga28]

socsocsoc-lab Did you managed to fix this ? I have the same issue. Any input appreciated.
Thanks!