Network Data Transfer

Introduction to Data Handling

This section details the protocols and methods used for transmitting data within the MS ecosystem. We support various data formats and provide robust mechanisms for efficient and secure data exchange.

Key concepts include:

• Serialization/Deserialization: Converting data structures into a format suitable for transmission and back.
• Data Formats: Supported formats like JSON, Protocol Buffers, and custom binary formats.
• Transmission Protocols: Leveraging HTTP/2, WebSockets, and direct TCP connections.

Supported Data Formats

We strive to be flexible and support the most common and efficient data formats. Choose the format that best suits your needs for performance, readability, and interoperability.

JSON (JavaScript Object Notation)

A widely used, human-readable format. Ideal for configuration files and general-purpose data exchange.

{
    "user_id": 12345,
    "username": "example_user",
    "active": true,
    "roles": ["admin", "editor"]
}

Protocol Buffers (Protobuf)

A language-neutral, platform-neutral, extensible mechanism for serializing structured data. Offers excellent performance and smaller message sizes.

Example `.proto` definition:

syntax = "proto3";

message User {
  int32 user_id = 1;
  string username = 2;
  bool active = 3;
  repeated string roles = 4;
}

Custom Binary Formats

For highly specialized use cases requiring extreme performance or minimal overhead, custom binary formats can be defined and implemented.

API Endpoints for Data Transfer

The following API endpoints facilitate data operations. Each endpoint specifies the supported HTTP methods, request, and response details.

POST /ms/api/v1/data/send

POST /ms/api/v1/data/send

Submits data for processing or storage.

Request Body (JSON example)

{
    "payload": {
        "type": "log",
        "timestamp": "2023-10-27T10:30:00Z",
        "message": "User logged in successfully."
    },
    "format": "json"
}

Response (JSON example)

{
    "status": "success",
    "message_id": "abc123xyz789",
    "received_at": "2023-10-27T10:30:01Z"
}

GET /ms/api/v1/data/{id}

GET /ms/api/v1/data/{id}

Retrieves data associated with a specific ID.

Parameters

Name	Type	Description
id	string	The unique identifier of the data to retrieve.

Response (JSON example)

{
    "status": "success",
    "data": {
        "original_payload": {
            "type": "log",
            "timestamp": "2023-10-27T10:30:00Z",
            "message": "User logged in successfully."
        },
        "format": "json",
        "retrieved_at": "2023-10-27T10:31:05Z"
    }
}

Error Response (JSON example)

{
    "status": "error",
    "message": "Data not found for the provided ID."
}

PUT /ms/api/v1/data/{id}

PUT /ms/api/v1/data/{id}

Updates existing data.

Request Body (JSON example)

{
    "payload": {
        "message": "User logged out."
    },
    "format": "json"
}

Response (JSON example)

{
    "status": "success",
    "message": "Data updated successfully.",
    "updated_at": "2023-10-27T10:35:00Z"
}

WebSockets for Real-time Data

For applications requiring real-time data streams, MS provides WebSocket support. This allows for bidirectional communication between the client and server.

Connecting to the WebSocket Server

The WebSocket endpoint is typically:

ws://your-ms-domain.com/ms/ws/data

Example Usage (JavaScript)

const ws = new WebSocket('ws://your-ms-domain.com/ms/ws/data');

ws.onopen = (event) => {
  console.log('WebSocket connection opened:', event);
  ws.send(JSON.stringify({ message: 'Hello Server!' }));
};

ws.onmessage = (event) => {
  console.log('Message from server:', JSON.parse(event.data));
};

ws.onerror = (event) => {
  console.error('WebSocket error observed:', event);
};

ws.onclose = (event) => {
  console.log('WebSocket connection closed:', event);
};

The WebSocket protocol supports sending and receiving messages in various formats. Ensure your client and server are configured to use compatible formats, such as JSON or Protobuf.