Make page tokens shorter

[david-crespo]

This is helpful but not essential for oxidecomputer/console#1102, putting page tokens in console URLs. The real blocker is #436.

---

Page tokens are base64ed JSON and they come out pretty long. Here's an example:

> atob('eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJuYW1lX2FzY2VuZGluZyIsInByb2plY3QiOiJhbGFuIiwibGFzdF9zZWVuIjoienp6LWluc3QtMTE0In19')
'{"v":"v1","page_start":{"sort_by":"name_ascending","project":"alan","last_seen":"zzz-inst-114"}}' 

oxiderack.com/projects/mock-project/instances?page=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJuYW1lX2FzY2VuZGluZyIsInByb2plY3QiOiJhbGFuIiwibGFzdF9zZWVuIjoienp6LWluc3QtMTE0In19 would work fine I guess, but to me it looks a bit silly. Here's where we do the serialization:

dropshot/dropshot/src/pagination.rs

Lines 417 to 430 in 6b41086

	let token_bytes = {
	let serialized_token =
	SerializedToken { v: PaginationVersion::V1, page_start };
	let json_bytes =
	serde_json::to_vec(&serialized_token).map_err(|e| {
	HttpError::for_internal_error(format!(
	"failed to serialize token: {}",
	e
	))
	})?;
	URL_SAFE.encode(json_bytes)
	};

It should be pretty easy to encode the token in a more efficient format like MessagePack and maybe reduce the size of the data itself by, e.g., making some keys shorter.

Method	Length of token
Current: base64 JSON string	128
MessagePack JSON as-is	104
MessagePack JSON with page_start -> p	92
MessagePack struct directly with serde_rmp (code below)	52

Rust program to compare base64 JSON and MessagePack

# Cargo.toml
[package]
name = "serialization-test"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
rmp-serde = "0.15.4"
base64 = "0.13.0"

// src/main.rs

extern crate base64;
extern crate rmp_serde as rmps;
extern crate serde;
extern crate serde_json;

use serde::{Deserialize, Serialize};
use std::error::Error;

#[derive(Debug, PartialEq, Deserialize, Serialize)]
struct MyStruct {
    v: String,
    page_start: PageStart,
}

#[derive(Debug, PartialEq, Deserialize, Serialize)]
struct PageStart {
    sort_by: String,
    project: String,
    last_seen: String,
}

fn main() -> Result<(), Box<dyn Error>> {
    let data = MyStruct {
        v: "v1".to_owned(),
        page_start: PageStart {
            sort_by: "name_ascending".to_owned(),
            project: "alan".to_owned(),
            last_seen: "zzz-inst-114".to_owned(),
        },
    };

    // Serialize with JSON
    let json_data = serde_json::to_vec(&data)?;
    let encoded_json = base64::encode(&json_data);

    // Serialize with MessagePack
    let msgpack_data = rmps::to_vec(&data)?;
    let encoded_msgpack = base64::encode(&msgpack_data);

    // Compare the lengths of the encoded strings
    println!("Base64 JSON Length: {}", encoded_json.len());
    println!("Base64 MessagePack Length: {}", encoded_msgpack.len());

    Ok(())
}

[ahl]

What about compressing the text first?