(This is the fourth in a four-part series; click to jump to posts 1, 2, and 3)
Genvid Viewer Web Stream Overview
We will go over the structure of the Genvid SDK web-streaming layer in the context of the GenvidTanks Sample. You will be able to edit and customize how viewers can view data being sent from your Unity game client via the Genvid Services, as well as sending data back from the web to the game.
Now, let’s view a live stream with the services and stream running locally. We can see a few buttons and the actual gameplay footage.
Genvid Overlay
The Genvid web stream is composed of 2 layers: the video stream and the Genvid Overlay. The video stream is live gameplay captured in real-time by the spectator client via the GenvidVideo component and streamed over the Genvid Services. On top of the streamed footage is an interactive layer for viewers to engage with the spectator client which is known as the Genvid Overlay.
Interactive Elements
The Genvid Overlay provides a few interactive pieces for viewers to engage with.
-
- In the top left corner of the video, we can see a ‘Toggle Map’ button. If you click that button, you can see a top-down map of the game arena and the tanks. As the tanks move around you, can see that the map updates to show the relative position of the tanks. Clicking the ‘Toggle Map’ button again hides the top down view of the game from the overlay.
-
- To the right of the ‘Toggle Map’ button, you can see a series of buttons allowing you to vote for the next loot drop. Repeatedly clicking a button for a particular vote will allow the loot with the most votes to spawn.
-
- And lastly towards the bottom of the video player, you can see banner buttons. Clicking on the colored banner displays a panel which shows the respective colored tank’s statistics, such as the health, shots fired, shots hit, accuracy, current active buff, and the number of cheers. You can click on the actual tank in the video overlay to see the same information too.
-
- Inside the banner button is another button with the thumbs up icon. This button allows you to cheer for your favorite tank and clicking it displays a text overlay of cheering for a tank.
Custom Graphics
If you draw your attention to the center of the screen towards the billboard, you will notice a graphic stating ‘Your Ad Here’. If you look over to the actual spectator client, you will notice that the same billboard simply says `Tanks!’. The Genvid Overlay is extensible to allow you to place custom graphics which are aligned to the game camera in order to create unique views for your viewers.
Data Streams
So how does this all work? The Genvid SDK passes data via several types of streams with different use cases. For example, to display the player-number label consistently on top of the tanks, the tanks’ position is sent over the game-data streams. For detailed information about game data-streams, please view the documentation.
Game Data
Open the unity.ts tile in C:\Genvid\GenvidServices\web\public.
The data we retrieve from the spectator client for the web stream is structured in the IGameData interface on line 30:
// Conversion from Json data into structure for a high refresh rate game state
export interface IGameData {
matProjView: IUnityMatrix4x4;
tanks: ITankData[];
adBillboardMatrix: IUnityMatrix4x4;
lootCountdown: number;
lootMatrices: IUnityMatrix4x4[];
shellMatrices: IUnityMatrix4x4[];
explosionMatrices: IUnityMatrix4x4[];
}
The web stream retrieves JSON data and parses it into the structured interface format. This operation is performed in the ‘on_streams_received’ function on line 596, which is registered to the constructed Genvid client when a viewer joins the stream.
// Upon receving the stream, get the timecode and the data
private on_streams_received(dataStreams: genvid.IDataStreams) {
for (let stream of dataStreams.streams) {
for (let frame of stream.frames) {
if (this.last_game_time_received < frame.timeCode) {
this.last_game_time_received = frame.timeCode;
}
}
}
// Parse the JSON from each elements
for (let stream of [...dataStreams.streams, ...dataStreams.annotations]) {
for (let frame of stream.frames) {
try {
if (stream.id == "GameData") frame.user = <IGameData>JSON.parse(frame.data);
if (stream.id == "MatchState") frame.user = <IMatchStateData>JSON.parse(frame.data);
}
catch (err) {
console.info("invalid Json format for:" + frame.data + " with error :" + err);
}
}
}
}
Data found in the game-data streams, such as the ITankData interface, are used to display information to the viewers such as the tanks position in the top-down Toggle Map and the panel with the tanks’ firing and health statistics. Since there is a delay between the video stream and the actual gameplay, Genvid SDK compensates by synchronizing game-data with the video stream. For example, if a viewer wanted to click the tank and view the statistics, without synchronization between game-data and the video stream, the viewer would have to try and guess where the tank actually is instead of clicking the tank’s position on the video stream. This is done in the ‘on_new_frame’ function which is passed as a callback to the client’s ‘onDraw’ function in the unity.ts file.
// ---------------------------------------------------------Enter frame section---------------------------------------------------------
private on_new_frame(frameSource: genvid.IDataFrame) {
let gameDataFrame = frameSource.streams["GameData"];
let gameData: IGameData = null;
if (gameDataFrame && gameDataFrame.user) {
gameData = gameDataFrame.user;
let tanks = gameData.tanks;
this.tankData = tanks;
this.explosionMatrices = gameData.explosionMatrices;
this.lootMatrices = gameData.lootMatrices;
this.shellMatrices = gameData.shellMatrices;
let matchDataFrameAnnotations = frameSource.annotations["MatchState"];
if (matchDataFrameAnnotations) {
for (let annotation of matchDataFrameAnnotations) {
let matchState = <IMatchStateData>annotation.user;
this.refreshMatchState(matchState, tanks);
}
}
// Setup the player table once the game data is ready
if (this.playerTableSetupCompletion === false) {
this.playerTableSetupCompletion = true;
this.initPlayerTable(tanks);
}
let mat = gameData.matProjView;
this.gfx_prog_data_viewproj = [mat.e00, mat.e10, mat.e20, mat.e30, mat.e01, mat.e11, mat.e21, mat.e31, mat.e02, mat.e12, mat.e22, mat.e32, mat.e03, mat.e13, mat.e23, mat.e33];
this.lootDropUpdate(gameData.lootCountdown);
this.tankOverlayUpdate(tanks, gameData.matProjView);
this.adOverlayUpdate(gameData.adBillboardMatrix);
this.gfx_draw3D();
}
if (this.videoReady) {
this.genvidInformationOverlayUpdate(); // Update the Genvid information overlay
this.visibilityUpdate(); // Update the visibility on the overlay when using key press
}
let isFullScreen = this.checkFullScreen();
if (isFullScreen !== this.isFullScreen) {
this.isFullScreen = isFullScreen;
}
}
Since the IGameData contains information about the spectator client’s projection view matrix and billboard, we can construct a billboard effect on top of the video stream to display a custom graphic. The custom graphic is currently located in the GenvidServices under the web/public/img directory.
// billboard ad
{
// Only prepare textures.
let cmd = new RenderCommand();
let options = {
"wrap": gl.CLAMP_TO_EDGE,
"aniso": true,
};
cmd.tex = gl.createTexture();
cmd.img = new Image();
cmd.img.onload = function () { handleTextureLoaded(cmd.img, cmd.tex, options); if (onload) onload(); };
cmd.img.src = "img/billboardImage.png";
cmd.visible = true;
this.gfx_cmd_ad = cmd;
}
The camera’s projection view matrix of the spectator client is reconstructed in the video overlay via WebGL and the position of the billboard is updated with the received game data. The texture is then loaded and at each rendered frame, drawn at the position of the billboard that was received in the game-data stream. To view the specified logic, please see the WebGL section in the unity.ts file.
// Upon receving the stream, get the timecode and the data
private on_streams_received(dataStreams: genvid.IDataStreams) {
for (let stream of dataStreams.streams) {
for (let frame of stream.frames) {
if (this.last_game_time_received < frame.timeCode) {
this.last_game_time_received = frame.timeCode;
}
}
}
// Parse the JSON from each elements
for (let stream of [...dataStreams.streams, ...dataStreams.annotations]) {
for (let frame of stream.frames) {
try {
if (stream.id == "GameData") frame.user = <IGameData>JSON.parse(frame.data);
if (stream.id == "MatchState") frame.user = <IMatchStateData>JSON.parse(frame.data);
}
catch (err) {
console.info("invalid Json format for:" + frame.data + " with error :" + err);
}
}
}
}
Annotations
In the same on_streams_received function, the function also processes annotation streams, which represent a one time non-persistent event. In the GenvidTanks Sample, the spectator client occasionally just needs to send match state data. MatchStateData consists of the state ID, round number, and winning tank ID. This can be found in the unity.ts file on line 23.
// Conversion from Json data into structure for a low refresh rate game state
export interface IMatchStateData {
stateID: number;
roundNumber: number;
winningTankId: number;
}
- The annotated match state is used to refresh the viewer web stream to display the round number after a tank is defeated. Annotations are updated in the on_new_frame function on line 361 in the ts file.
private on_new_frame(frameSource: genvid.IDataFrame) {
let gameDataFrame = frameSource.streams["GameData"];
let gameData: IGameData = null;
if (gameDataFrame && gameDataFrame.user) {
gameData = gameDataFrame.user;
let tanks = gameData.tanks;
this.tankData = tanks;
this.explosionMatrices = gameData.explosionMatrices;
this.lootMatrices = gameData.lootMatrices;
this.shellMatrices = gameData.shellMatrices;
let matchDataFrameAnnotations = frameSource.annotations["MatchState"];
if (matchDataFrameAnnotations) {
for (let annotation of matchDataFrameAnnotations) {
let matchState = <IMatchStateData>annotation.user;
this.refreshMatchState(matchState, tanks);
}
}
// Setup the player table once the game data is ready
if (this.playerTableSetupCompletion === false) {
this.playerTableSetupCompletion = true;
this.initPlayerTable(tanks);
}
let mat = gameData.matProjView;
this.gfx_prog_data_viewproj = [mat.e00, mat.e10, mat.e20, mat.e30, mat.e01, mat.e11, mat.e21, mat.e31, mat.e02, mat.e12, mat.e22, mat.e32, mat.e03, mat.e13, mat.e23, mat.e33];
this.lootDropUpdate(gameData.lootCountdown);
this.tankOverlayUpdate(tanks, gameData.matProjView);
this.adOverlayUpdate(gameData.adBillboardMatrix);
this.gfx_draw3D();
}
if (this.videoReady) {
this.genvidInformationOverlayUpdate(); // Update the Genvid information overlay
this.visibilityUpdate(); // Update the visibility on the overlay when using key press
}
let isFullScreen = this.checkFullScreen();
if (isFullScreen !== this.isFullScreen) {
this.isFullScreen = isFullScreen;
}
}
Scaling and Aggregation
Now, imagine a scenario with a large number of viewers. If viewers are repeatedly clicking loot votes and the data needs to be consumed by the game, the Genvid Services would be potentially under heavy load. To ensure that your game is not a victim of its own success with a large viewership, the Genvid SDK employs a technique known as Map Reduce. Map Reduce is an algorithm which aggregates and maps common data together and reduces the common events into a single structure. This allows the spectator client to consume a single event instead of consuming many events. To define the structure of data and what map reduce applies to, you can edit the events.json file under the GenvidServices/config directory.
Notifications
Once the loot votes data is aggregated and consumed by the game, the spectator client needs to notify the viewer web streams immediately to show the current number of votes per loot. This is retrieved using the Genvid SDK notifications API. You can find the notification logic to retrieve notification data on line 621 in the unity.ts file. For the Unity integration, please see the GenvidTanksGameDataStream component in the SubmitLootData function.
// Upon receiving a notification, get the notification content
private on_notifications_received(message: genvid.IDataNotifications) {
for (let notification of message.notifications) {
if (notification.id === "LootVotesData") {
let datastr = genvid.UTF8ToString(notification.rawdata);
try {
// Get the latest loot drop data
let userData = <ILootVotesData>JSON.parse(datastr);
this.latestLootVoteData = userData;
}
catch (err) {
console.info("invalid Json format for:" + datastr + " with error :" + err);
}
}
}
}
Commands
If you head over to the top right corner of the viewer web stream, you can click the Admin button to enter the Admin panel. Type in ‘admin’ for both the username and password. The Admin panel uses the Genvid SDK command channel to send commands directly to the game. This is because events are consolidated and reduced before going to the game, while commands go straight to the game, unaltered. This allows for direct input to the game, but does not scale to allow for thousands of incoming viewer-interactions.
We can add buffs to a particular player, such as directly providing player 2 with a shield.
- View the ts file and head over to line 143. The buffTank function builds an ICommandRequest which is an ID value pair. The ID allows commands to be mapped directly between the admin panel and the spectator client, while the value allows you to pass in custom arguments such as the tank you want to buff and the type of buff you want to apply. This is the concatenated string value of player number and buff number. This is subsequently parsed as 2 integers in the GenvidTanksCommand component over on the spectator client side.
// send buff commands to tanks
buffTank(tankName: string, tankId: number, buffId: number) {
this.message = this.error = "";
let command: ICommandRequest = {
id: "PowerUpTank",
value: `${tankId.toString()}:${buffId.toString()}`
};
var buffName: string = this.getBuffName(buffId);
let promise = $.post("/api/admin/commands/game", command).then(() => {
this.message = `PowerUpTank ${tankName}:${buffName}`;
this.displayMessage();
});
promise.fail((err) => {
this.message = `Failed with error ${err} to do changeSpeed ${tankName}:${buffName}`;
this.displayErrorMessage();
});
}
Now, we’ve seen how the Genvid SDK sends and receives audio, video and data via the Genvid Services, both from and to the Unity Game client and displays them in the Web Stream for viewers to interact with. This is just the beginning of the creative possibilities that you can build into your game.
