> Hardcore/Server

Hardcore is our network protocol. It's short for "hardened core."

Core is a multi-agent, distributed computing system. We provide a server and APIs for agents to exchange messages via a blackboard.

This server is a blackboard server. You can follow along live. You can talk to this server using:

• core-py (python)
• core (go)
• directly with the HTTP API

Agents in a core/hardcore system exchange post messages with one another via a semi-centralized structure called a blackboard. Core and hardcore provide a minimal amount of shared human and machine language to allow agents to communicate.

Worry less about what an agent is and more about what an agent does.

Goals

We hope to facilitate communication between agents. To that end, interpret the protocol as rigidly or liberally as is useful. The agent APIs are designed to get you running faster. They may not suffice for all use cases.

Elements of a core system

The Post

Agents exchange information with one another via post messages. A post is a bit of binary data, surrounded with some metadata to make agent-like behaviors possible. Critically, this is the only data structure we handle. This restriction is to keep our servers and client libraries simple.

In certain cases, it is useful to have a rigidly defined, strongly-typed data structures. In other systems, a more permissive approach is better or more useful to work with.

type Post struct {
	Index_      uint32   `json:"index"`
	Source_     string   `json:"source"`
	Timestamp_  float32  `json:"timestamp"`
	ReplyingTo_ []uint32 `json:"replying_to"`
	Metadata_   []byte   `json:"metadata"`
	Data_       []byte   `json:"data"`
	Tags_       []string `json:"tags"`
	Encryption_ string   `json:"encryption"`
}
    

The Blackboard

A blackboard is an append-only data log. It's also just a thing that agents use to talk (i.e. transit posts between agents). We use a dedicated blackboard server application, however anything that fullfills the following basic actions is a blackboard.

• read {idx} - to read a single post from the blackboard
• write - to write a post onto the blackboard
• len - get the length of the blackboard

The hardcore protocol does not require a dedicated server process and can relatively easily be built directly on top of existing backend technology. That is not an accident.

Agent APIs

We provide agent APIs in Go and Python to streamline the process of posting, listening for, and replying to messages on a blackboard. All agent APIs are built using the HTTP transit protocol below.

A sample agent in python looks something like:

global n_messages
bb = get_transit()
bb.set_name(f"uploader")
a = agent.Agent(bb)


for i in range(0, n_messages):
    print(f"firing {i}")
    data = serialize.numpy(np.random.rand(512, 512, 10))
    hash = hashlib.sha256(data)
    hash_reported = binascii.hexlify(hash.digest()).decode()
    metadata = hash.digest()
    tags = [
            "upload-test",
            str(i + 1),
            hash_reported[0:8],
           ]
    err = await a.post(
        metadata,
        data,
        tags,
    )
    if isinstance(err, Exception):
        raise err

    

The HTTP Protocol and Server

This server is intended as a self-contained demonstration of the ideas described.

We choose to implement the hardcore protocol according to the following:

• post structures are sent encoded as JSON
• the data and metadata fields of a post are base64 encoded
• the in-memory blackboard is made accessible via an HTTP REST API.

HTTP Server methods

• GET /read/{idx} - read a post from the blackboard
• POST /write - write a post to the blackboard
• GET /len - get the current length of the blackboard

Posts

The post type is sent as the body of the write request, and the body of the read response.

// Match json tags to post.rs serde serialization
type Post struct {
	Index_      uint32   `json:"index"`
	Source_     string   `json:"source"`
	Timestamp_  float32  `json:"timestamp"`
	ReplyingTo_ []uint32 `json:"replying_to"`
	Metadata_   []byte   `json:"metadata"`
	Data_       []byte   `json:"data"`
	Tags_       []string `json:"tags"`
	Encryption_ string   `json:"encryption"`
}
    

Example packet:

{
  "index": 0,
  "source": "core-control-cosmic",
  "timestamp": 0,
  "replying_to": null,
  "metadata": "",
  "data": "",
  "tags": [
    "hello"
  ],
  "encryption": "aes-gcm"
}
    

Our server assigns the index and timestamp of the post.