Building a live chart with Deno, WebSockets, Chart.js and Materialize

Introduction

❗️ This demo includes examples for an unsupported version of Materialize (0.26.x) ❗️

This is a self-contained example of a real-time chart powered by Deno, Web Sockets, Chart.js, and Materialize.

Deno is a simple and secure runtime for JavaScript and TypeScript that uses V8. Deno, just like Materialize, is also written in Rust.

In this demo, we will build a simple live dashboard app that displays real-time data from a Deno Web Socket server. Deno will then connect to Materialize and TAIL our live materialized view to get the latest data and display it in a real-time chart using Chart.js.

Overview

Here is a quick overview of the project:

• A mock service to continually generate user score events.
• Redpanda instance to store the user score events in a topic.
• Materialize instance that is connected to the Redpanda instance and ingests the data from the topic in a live materialized view which we can query in real-time using just SQL.
• A Deno backend service that connects to Materialize and TAIL the live materialized view to get the latest data and display it in a real-time chart.
• Frontend service that connects to the Deno app via a web socket and displays the data in a real-time chart using Chart.js.

Here is a diagram of the project:

Prerequisites

To run this demo, you need to have the following installed.

• Install Docker.
• Install Docker Compose.

Running the demo

To get started, clone the repository:

git clone git clone https://github.com/bobbyiliev/materialize-tutorials.git

Then you can access the directory:

cd materialize-tutorials
git checkout lts
cd mz-deno-live-dashboard

With that you can then build the images:

docker-compose build

And finally, you can run all the containers:

docker-compose up -d

It might take a couple of minutes to start the containers and generate the demo data.

After that, you can visit http://localhost in your browser to see the demo:

Next, let's review the Materialize setup and the Deno backend setup.

Materialize setup

The Deno service will execute the following DDL statements on boot so that we don't have to run them manually:

• Create a Kafka source: Creating a source in Materialize does not actually start the data ingestion. You can think of a non-materialized source as just the metadata needed for Materialize to connect to your source but not process any data:

CREATE SOURCE score
FROM KAFKA BROKER 'redpanda:9092' TOPIC 'score_topic'
FORMAT BYTES;

• Create a create non-materialized view, that essentially only provides us with an alias for the SELECT statements they include:

CREATE VIEW score_view AS
    SELECT
        *
    FROM (
        SELECT
            (data->>'user_id')::int AS user_id,
            (data->>'score')::int AS score,
            (data->>'created_at')::double AS created_at
        FROM (
            SELECT CAST(data AS jsonb) AS data
            FROM (
                SELECT convert_from(data, 'utf8') AS data
                FROM score
            )
        )
    );

• Create a materialized view:

CREATE MATERIALIZED VIEW score_view_mz AS
    SELECT
        (SUM(score))::int AS user_score,
        user_id
    FROM score_view GROUP BY user_id;

To check if the views and the sources were created, launch the Materialize CLI:

```shell session docker-compose run mzcli

> This is just a shortcut to a docker container with postgres-client pre-installed, if you already have `psql` you could run `psql -U materialize -h localhost -p 6875 materialize`.

Then check the views and the sources:

sql SHOW VIEWS; -- Output: -- +-----------------+ -- | scoreview | -- | scoreview_mz | -- +-----------------+

SHOW sources; -- Output: -- +-----------------+ -- | score | -- +-----------------+

### Using `TAIL`

Next, to see the results in real-time we can use `TAIL`:

sql COPY ( TAIL scoreviewmz ) TO STDOUT;

You will see a flow of the new user score that was generated in real-time.

We can also start a `TAIL` without a snapshot, which means that you will only see the latest records after the query is run:

sql COPY ( TAIL scoreviewmz WITH (SNAPSHOT = false) ) TO STDOUT;

This is what we will use in our Deno application to get the top user scores and display them in a real-time chart.

For more information on how the `TAIL` function works, see the [Materialize documentation](https://materialize.com/docs/sql/tail/).

## Deno

Now that we have Materialize ready, let's review the Deno setup.

We would use two Deno modules:
- The Postgres module to connect to Materialize.
- The Web Sockets module to create a Web Socket connection to our Frontend service.

You can find the code in the [`backend` directory](https://github.com/bobbyiliev/materialize-tutorials/tree/main/mz-deno-live-dashboard/backend).

ts import { WebSocketClient, WebSocketServer } from "https://deno.land/x/[email protected]/mod.ts"; import { Client } from "https://deno.land/x/postgres/mod.ts";

// Specify your Materialize connection details const client = new Client({ user: "materialize", database: "materialize", hostname: "materialized", port: 6875, });

await client.connect(); console.log("Connected to Postgres");

// Start a transaction await client.queryObject('BEGIN'); // Declare a cursor without a snapshot await client.queryObject(DECLARE c CURSOR FOR TAIL score_view_mz WITH (SNAPSHOT = false));

const wss = new WebSocketServer(8080);

wss.on("connection", async function (ws: WebSocketClient) { console.log("Client connected"); setInterval(async () => { const result = await client.queryObject<{ mztimestamp: string; mzdiff: number, userid: number, userscore: number}>(FETCH ALL c); for (const row of result.rows) { let message = { userid: row.userid, userscore: row.userscore }; broadcastEvent(message); } } , 1000);

});

// Broadcast a message to all clients const broadcastEvent = (message: any) => { wss.clients.forEach((ws: WebSocketClient) => { ws.send(JSON.stringify(message)); }); }

Rundown of the code:

- As Materialize is Postgres wire compatible, first we import the `Client` class from the `https://deno.land/x/postgres/mod.ts` module. This is the class that we will use to connect to the Materialize instance.
- Next, we create a new `Client` instance and pass it the credentials for Materialize.
- Then we call the `connect()` method on the client instance to connect to Materialize.
- Next, we call the `queryObject()` method on the client instance to start a transaction and also call the `queryObject()` method on the client instance to declare a cursor without a snapshot.
- Finally, we create a new `WebSocketServer` instance and pass it the port to listen on.
- We then define a `connection` event handler on the `WebSocketServer` instance, which is called when a client connects.
- We then set an interval to fetch the latest data from Materialize and broadcast it to all clients.

## Frontend setup

For the frontend, we will not be using any JavaScript framework, but just the [Chart.js library](https://www.chartjs.org/).

Thanks to the web sockets connection, we can now receive the latest data from Materialize and display it in a real-time chart.

html