[GSoC] Add more timeseries database clients to OpenWISP Monitoring

[nemesifier]

Official thread for this GSoC idea, ask questions and announce your interest here.

[pushpitkamboj]

Hi I am interested for this project, already started researching on this.
I have a question. Could you share the reasoning behind choosing ElasticSearch as the additional TSDB? I reviewed the 2020 PR and related discussions, and I understand the context from that time. However, during my research I also came across TimescaleDB as a potential option worth considering. Am I missing some context that led to Elasticsearch being preferred over alternatives? (Because ElasticSearch is fundamentally a search engine, not a native TSDB)

[pushpitkamboj]

sorry, i mean influxDB 1.7 and 2.0

[nemesifier]

sorry, i mean influxDB 1.7 and 2.0

Influxdb 1.8 is the version we're using now and I don't understand why you're mentioning it, can you expand?

[pushpitkamboj]

Ya sure,
I built a local machine monitoring prototype supporting InfluxDB 1.8 and 2.0 (Elasticsearch in progress), implementing the same abstraction layer as OpenWISP Monitoring like batch_write(), query(), delete_metric_data() and more. The goal was to get hands on experience with the codebase structure before writing the proposal, rather than approaching it purely theoretically.

[nemesifier]

Sounds like a good approach, if you have more questions let me know.

[Kapingacoder]

Hi, thank you for your question, that's a very good point.

From my understanding based on the previous discussions and documentation, Elasticsearch was considered mainly because of its strong capabilities in indexing, searching, and analyzing large volumes of log and monitoring data, which can be useful for querying device metrics and command execution results efficiently.

I agree that TimescaleDB is also a very interesting alternative since it is built on PostgreSQL and is specifically optimized for time-series data, which could make it a strong candidate depending on the performance and integration requirements.

At the moment, I am still reviewing the historical design decisions and trying to better understand the trade-offs between flexibility, scalability, and integration with the existing OpenWISP stack. It is possible that Elasticsearch was preferred due to its ecosystem and existing integrations, but I am also open to learning whether TimescaleDB or other TSDB solutions could be more suitable.

I would also be very interested to hear maintainers’ opinions on whether newer TSDB options should be reconsidered given current requirements and technologies.

Thanks again for bringing up this important discussion.

[pushpitkamboj]

I have completed the prototype implementation. The repository is at https://github.com/pushpitkamboj/hostwatch
It validates the core approach for the GSoC project. Feedback is welcome

[nemesifier]

@pushpitkamboj would be good if you base your prototype on the actual code of openwisp-monitoring. That'll make it easier for us to review and give feedback.

[pushpitkamboj]

yes, actually it was a misunderstanding from my side, wasn't able to understand the prototype motive.
So sorry for that prototype I made. Now I am doing exactly this you stated, will also edit the proposal i have created. Will ping here when i am done.

[pushpitkamboj]

thanks to @stktyagi He helped me in understanding that how I can build prototype with openwisp-monitoring

[cbeaujoin-stellar]

I took a look, it was promising. Now as @nemesifier suggest let's build on top of openwisp-monitoring.

[pushpitkamboj]

built the prototype on top of openwisp-monitoring here https://github.com/pushpitkamboj/openwisp-monitoring/tree/gsoc_prototype
Also submitted the new proposal with detailed explanation over gsoc website.
@cbeaujoin-stellar @nemesifier

[Baboux98]

hi, I'm seeing that there's work already going here, I'm interested by the this idea.
I've downloaded the monitoring code base, I've tried to run the tests suite. Most of the selenium related does work. I'm getting an error related to the firefox browser. didn't start investigated yet.
Now I'm also looking at this pr: openwisp/openwisp-monitoring#584 to understand what has been done
I also think that there is some test related to celery

[855princekumar]

Hi, I've been exploring openwisp-monitoring for this idea and recently worked on a PR around duplicate check creation and async handling:
openwisp/openwisp-monitoring#769

While doing that I got a bit familiar with the monitoring module and the Celery side of things. I'm now going through the old Elasticsearch PR (#584) to understand what's already been done.

Just had one question, for Elasticsearch, should the focus be mostly on completing/stabilizing that work, or are there parts that are expected to be reworked?

I'll keep digging into the code and try to contribute more in the meantime.

[cbeaujoin-stellar]

Hi, for Elasticsearch, a first step would be to rebase to current branch, update (mainly the dependencies) and fix the existing PR.

I've also added an additional list of rework/refactor in openwisp/openwisp-monitoring#164
That could be discussed and splited in smaller more detailed tickets.

[855princekumar]

Thanks for the direction, this helps.

I’ll start by checking PR #164, try to rebase it on the current codebase, and see what breaks (especially tests and dependencies). Also, go through the issues mentioned there step by step and try to get a minimal working state first before thinking about further improvements. Rest, I'll share updates once I have something concrete.

[nemesifier]

Hi, for Elasticsearch, a first step would be to rebase to current branch, update (mainly the dependencies) and fix the existing PR.

I've also added an additional list of rework/refactor in openwisp/openwisp-monitoring#164 That could be discussed and splited in smaller more detailed tickets.

@cbeaujoin-stellar I recommend moving those to a dedicate separate issue to avoid confusion.

[cbeaujoin-stellar]

Before proceeding with implementation with TimescalDB, we must verify that the Timescale License (TSL) aligns with OpenWISP distribution model, as its 'Source Available' restrictions differ from standard open-source agreements.

[pushpitkamboj]

You mean to understand this TSL. The core features we need fall under the Apache 2.0 portion. The TSL restrictions primarily target cloud DBaaS providers, which does not apply to OpenWISP's self hosted model. I understand your point, I think we can discuss this in later stages after the primary goal of adding influxDB2.0 and elasticsearch is acheived.