Nexus

Nexus is a Data Distribution System designed for event streaming, dataset access, and value storage. It allows publishers to inform the Nexus server about event streams and datasets, while consumers can request specific data paths.

Features

1. Event Streaming (Kafka)

   • Publishers can register event streams with the Nexus server, providing details such as the server address, data structure of events, and the path to publish.

2. Dataset Access

   • Users can run scripts to inform the Nexus server about static datasets, including how to access them and their data structure.

3. Value Store/Publishing (Redis)

   • Small values can be sent directly to the Nexus server, allowing consumers to retrieve data directly or act as a cache for event streaming.

4. Go and Python Clients

   • Go client is available here
   • Python client in this repo

Installation

• Install Golang:
  • sudo apt install golang-go
• Install protoc:
  • sudo apt install -y protobuf-compiler
• Install go-protoc:
  • go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
  • go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest Run make all

Add and start as systemd service:

sudo cp ./nexus-server.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable nexus-server.service
sudo systemctl start nexus-server.service

Monitor service:

sudo systemctl status nexus-server.service
sudo journalctl -u nexus-server.service -f

Usage

Running the Server

To run the Nexus server, use the following command:

Usage: 'nexus-server <load_file_path> <save_file_path>' or 'nexus-server <save_file_path>'
Options:
  <load_file_path>   Path to the file to load.
  <save_file_path>   Path to the file to save.
  --help             Show this help message.

Running the Client

To run the Nexus client, use the following command:

make run-client

Publishing Data

You can publish values and datasets using the Nexus client. For example:

./nexus-client publish value /testing/a 123
./nexus-client publish dataset /testing/dataset/a ./tests/example_a.csv

Example Use Cases

• Single location for all data sharing. Can organize by app, team, user, etc.
• Monitoring
  • Build system progress
  • Git branches
  • Docker container status
  • Kubernetes
• Inspect app internal state
• Share data between apps (pub-sub)
• Save queries
• Data versioning
• Personal scripting
  • e.g. a script that searches a directory for "TODO:_" statements and then publishes these to /users/$USER/todo/path/of/project
• JIRA integration?

Roadmap

General

• Save server state to disk and load from disk on startup
• Implement user authentication and authorization
• Improve error handling and logging
• Watch and auto-publish files updates, directories (new files, subdirs), and DBs (new tables)
• Symbolic path links
  • i.e. /status/my_app/ -> /my_app/prod/bin/status
• Support functions
  • i.e. publish a bash command or binary to a path. When that path is accessed, the command is executed and the output is returned.
• Remote hosting
  • Local data storage (Redis)
  • Remote data storage (S3, GCS, etc.)
• Shard Nexus server
  • Path can point to a specific path on a different server
  • i.e. americas:my/main/path/alt -> europe:my/european/path
• Documentation for API endpoints
• Implement data filtering
• Implement data transformation
• Implement data compression
• Implement data encryption
• Implement data deduplication
• Seperate out registering and publishing

Values

• Publishing Values
  • Int32
  • Float64
  • String
• Accessing Values
  • Int32
  • Float64
  • String

Datasets

• Registering Datasets
  • Registering individual files
  • Registering directories
  • Registering datasets from a remote source
  • Registering DB tables
  • Registering table queries
• Accessing Datasets
  • Accessing individual files
  • Accessing directories
  • Accessing datasets from a remote source
  • Accessing DB tables

Event Streaming

• Register __ streams
  • kafka
  • Websocket
  • file
  • DB
  • direct tcp
• Access __ streams
  • kafka
  • Websocket
  • file
  • DB
  • direct tcp
• Add support for stream filtering

Yukon

• Minimal navigation and info display
• Type out path
• Command line path argument
• Browse data
  • Browse leaf node
    • Browse values
    • Browse datasets
      • Browse individual files
      • Browse directories
      • Browse DB tables
    • Browse streams
  • Display data of non-leaf node
• Display data
  • Display leaf node
    • Display values
    • Display datasets
    • Display streams
  • Display data of non-leaf node
• Add method to __ data
  • add
  • delete
  • update
  • query
• Add help message with keybindings
• Tab completion if only one child
• Create a web interface for easier interaction

Contributing

Contributions are welcome! Please open an issue or submit a pull request for any improvements or features you'd like to add.