I wrote an API Client in Rust recently for EVE-NG, a network emulator. Some of its API endpoints (that represent lists of things) return different JSON representations for the same endpoint depending on the state of the instance. As Rust is strict about types, I had to figure out how to deserialize these quirks while ensuring consistency in my program.
An example of this quirk is the list of nodes in a lab. When the lab has no nodes, the data is an empty array:
{
"data": [],
}
When the lab has nodes, the representation changes from an empty array to a hash map:
{
"data": {
"1": {
"name": "R1",
"status": 0,
// ...
},
"2": {
"name": "R2",
"status": 0,
// ...
}
}
}
Another example is a list of interfaces for a node. A typical node returns an array for Ethernet interfaces, and an empty array for Serial interfaces as they’re unsupported:
{
"data": {
"ethernet": [
{
"name": "Gi0/0",
"network_id": 1
},
{
"name": "Gi0/1",
"network_id": 3
}
],
"serial": [],
}
}
The node type that supports Serial interfaces assigns both interface types in groups of 4, called a portgroup. In this case, the representation for both interface types changes from an array to a hash map:
{
"data": {
"ethernet": {
"0": {
"name": "e0/0",
"network_id": 0
},
"32": {
"name": "e0/1",
"network_id": 0
},
"48": {
"name": "e0/2",
"network_id": 0
},
"16": {
"name": "e0/3",
"network_id": 0
}
},
"serial": {
"1": {
"name": "s1/0",
"remote_id": 0,
// ...
},
"33": {
"name": "s1/2",
"remote_id": 0,
// ...
},
"49": {
"name": "s1/3",
"remote_id": 0,
// ...
},
"17": {
"name": "s1/1",
"remote_id": 0,
// ...
}
}
}
}
Looking at the examples, patterns start to emerge:
- Empty lists are represented as arrays
- Lists with sequential keys starting from 0 are represented as arrays
- Lists with sequential keys not starting from 0 or non-sequential keys are represented as a key-value map
Why does this happen? EVE-NG and its API are written in PHP, where the concept of an array is quite flexible. Lists and maps in JSON are both considered arrays in PHP, which causes different representations during serialization depending on the key/index.
The solution here is to convert one type to the other. Converting maps to arrays won’t suit most of the cases here as the keys are non-sequential, but sequential arrays can be converted to maps with the index as the key. This diverts from the structure in the API documentation but it also ensures that the endpoints return hash maps in all cases. So I chose to convert arrays to maps.
Custom deserialization first pass
serde provides ways to write a custom deserialization function. I already used one such approach in another part of the project: a field attribute called deserialize_with, that takes a custom function to deserialize a struct field. For my first pass, I wrote a deserialization function called vec_as_map:
struct Interfaces {
#[serde(deserialize_with = "vec_as_map")]
ethernet: HashMap<u32, Ethernet>,
}
use std::collections::HashMap;
use std::fmt;
use std::hash::Hash;
use serde::{Deserialize, Deserializer};
use serde::de;
fn vec_as_map<'de, D, K, V>(
deserializer: D,
) -> std::result::Result<HashMap<K, V>, D::Error>
where
D: Deserializer<'de>,
K: Eq + Hash + Deserialize<'de>,
V: Deserialize<'de>,
{
todo!()
}
In the JSON snippets from earlier, the keys were all strings, yet the key is generic here as I would like to deserialize some of the maps to integer keys1. Apart from the Deserialize trait bound on K and V, K takes two additional traits that are required by HashMap keys - Eq and Hash.
The implementation uses an untagged enum to detect the type of the incoming value2:
#[derive(Debug, Deserialize)]
#[serde(untagged)]
#[serde(bound(deserialize = "T: Eq + Hash + Deserialize<'de>, U: Deserialize<'de>"))]
enum MapOrArray<T, U> {
Map(HashMap<T, U>),
Array(Vec<U>),
}
let value = MapOrArray::deserialize(deserializer)?;
Next, I matched on the value, where maps are returned as is, and arrays are deserialized to maps with the index as the key. If the array is empty, the hash map stays empty:
match value {
MapOrArray::Map(map) => Ok(map),
MapOrArray::Array(arr) => {
let mut map = HashMap::new();
for (index, value) in arr.into_iter().enumerate() {
let key = K::deserialize(serde_json::Value::from(index))
.map_err(de::Error::custom)?;
map.insert(key, value);
}
Ok(map)
}
}
Since the key is generic, the numeric index needs to be parsed to K. This works if the K is a numeric type, but will create an error if K is a String. Since arrays are consistently converted to numeric key maps in my program, I am ignoring this case.
For the purposes of this post, I’ll be using these unit tests representing the different cases:
#[cfg(test)]
mod tests {
use super::*;
#[derive(Debug, Deserialize)]
struct User {
name: String,
}
#[derive(Debug, Deserialize)]
struct Interfaces {
#[serde(deserialize_with = "vec_as_map")]
ethernet: HashMap<u32, String>,
}
#[test]
fn empty_data_becomes_empty_map() {
let result: HashMap<String, User> = serde_json::from_str("[]").unwrap();
assert!(result.is_empty());
}
#[test]
fn map_with_string_keys_remains_a_map() {
let result: HashMap<String, User> =
serde_json::from_str(r#"{"admin": { "name": "Administrator" }}"#).unwrap();
assert_eq!(result["admin"].name, "Administrator");
}
#[test]
fn map_with_numeric_string_keys_becomes_numeric_key_map() {
let result: Interfaces =
serde_json::from_str(r#"{"ethernet": {"1": "a", "2": "b", "3": "c"}}"#).unwrap();
assert_eq!(result.ethernet[&1], "a");
}
#[test]
fn array_becomes_numeric_key_map() {
let result: Interfaces = serde_json::from_str(r#"{"ethernet": ["a", "b", "c"]}"#).unwrap();
assert_eq!(result.ethernet[&0], "a");
}
#[test]
fn empty_array_becomes_empty_map() {
let result: Interfaces = serde_json::from_str(r#"{"ethernet": []}"#).unwrap();
assert!(result.ethernet.is_empty());
}
}
The first pass worked...kind of.
Empty data returns an error as there is no custom deserialization applied there yet. vec_as_map can be used only for struct fields, not the return type itself.
---- tests::empty_data_becomes_empty_map stdout ----
thread 'tests::empty_data_becomes_empty_map' (75) panicked at src/lib.rs:62:75:
called `Result::unwrap()` on an `Err` value: Error("invalid type: sequence, expected a map", line: 1, column: 0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
The map with numeric keys fails to deserialize because the untagged enum failed to match the data to either variant:
---- tests::map_with_numeric_string_keys_becomes_numeric_key_map stdout ----
thread 'tests::map_with_numeric_string_keys_becomes_numeric_key_map' (76) panicked at src/lib.rs:76:85:
called `Result::unwrap()` on an `Err` value: Error("data did not match any variant of untagged enum MapOrArray", line: 1, column: 44)
An untagged enum converts the data to an intermediate representation, which doesn’t deserialize numeric keys to numbers. So the keys remains as strings while the enum expects an integer key, thus returning an error. I came across an issue describing a similar situation, so this is a known bug.
The next two sections solve these errors one by one.
Deserialization with a custom Visitor
Since the untagged enum approach didn’t work, I looked into serde’s suggested way to deserialize a type: implementing a custom Visitor. This implementation supports integer keys as the deserialization happens in the same way as serde_json.
The Visitor implementation provides methods for the possible types, and the custom visitor can override one or more of these methods. Since we’re looking to deserialize maps and arrays to maps, the custom visitor implements both visit_seq and visit_map.
Creating a visitor requires creating a struct and implementing Visitor on it, with the required functions:
use std::marker::PhantomData;
use serde::de::{IntoDeserializer, MapAccess, SeqAccess, Visitor};
struct VecMapVisitor<K, V>(PhantomData<(K, V)>);
impl<'de, K, V> Visitor<'de> for VecMapVisitor<K, V>
where
K: Eq + Hash + Deserialize<'de>,
V: Deserialize<'de>,
{
type Value = HashMap<K, V>;
fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
formatter.write_str("a map or an array")
}
PhantomData works as a placeholder type (called a marker type in Rust) to store the generic parameters. Value is a mandatory type for the trait that refers to the output type, which in this case is a HashMap. expecting is a mandatory function that takes a message describing the input type/s, i.e., what the visitor expects to receive.
Maps are deserialized as is:
fn visit_map<A>(self, mut access: A) -> std::result::Result<Self::Value, A::Error>
where
A: MapAccess<'de>,
{
let mut map = HashMap::with_capacity(access.size_hint().unwrap_or(0));
while let Some((k, v)) = access.next_entry()? {
map.insert(k, v);
}
Ok(map)
}
While arrays are deserialized to maps with a similar implementation to vec_as_map:
fn visit_seq<A>(self, mut access: A) -> std::result::Result<Self::Value, A::Error>
where
A: SeqAccess<'de>,
{
let mut map = HashMap::new();
let mut index: usize = 0;
while let Some(value) = access.next_element::<V>()? {
let key = K::deserialize(index.into_deserializer())?;
map.insert(key, value);
index += 1;
}
Ok(map)
}
}
The visitor only provides access to the next element of the array, so enumerate is replaced with an index variable that is incremented at the end of the loop. One improvement from the previous approach is simplifying the key deserialization by replacing the conversion to serde_json::Value with IntoDeserializer.
vec_as_map then calls the visitor:
pub(crate) fn vec_as_map<'de, D, K, V>(
deserializer: D,
) -> std::result::Result<HashMap<K, V>, D::Error>
// ...
{
deserializer.deserialize_any(VecAsMap(PhantomData))
}
I use deserialize_any here (instead of deserialize_map or deserialize_seq) because there are multiple possible input types.
With this change, map_with_numeric_string_keys_becomes_numeric_key_map passes!
Adding a custom map type
For return types, I came across the concept of a newtype in Rust, that wraps an existing type and lets one add functionality to it. I created one called CustomMap:
#[derive(Debug)]
pub(crate) struct CustomMap<K, V>(HashMap<K, V>);
Instead of deriving Deserialize directly, this type gets a custom Deserialize impl that calls vec_as_map and wraps the result with the newtype:
impl<'de, K, V> Deserialize<'de> for CustomMap<K, V>
where
K: Eq + Hash + Deserialize<'de>,
V: Deserialize<'de>,
{
fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
where
D: Deserializer<'de>,
{
vec_as_map(deserializer).map(CustomMap)
}
}
One way to use CustomMap is to replace all instances of HashMap in my code with it, and export CustomMap as a public type that users import in their code. This would also make vec_as_map redundant as I can specify the newtype in both struct fields and return types:
pub struct Interfaces {
ethernet: CustomMap<u32, String>,
}
pub async fn list_nodes(path: &str) -> Result<CustomMap<u32, String>> {}
The newtype doesn’t have any of the functions of a HashMap, so I’d either have to provide a getter to access the underlying hash map, which didn’t seem nice for library code. Using CustomMap as an internal type made more sense: a function first deserializes to CustomMap, and returns the underlying HashMap. An example of this is the updated test empty_data_becomes_empty_map, which passes:
#[test]
fn empty_data_becomes_empty_map() {
let result: HashMap<String, User> =
serde_json::from_str::<CustomMap<String, User>>("[]")
.unwrap()
.0;
assert!(result.is_empty());
}
This way the quirks and the newtype are hidden away from the user, and they can use the full range of HashMap functions in their program.
Wrapping it up
After a good amount of trial and error, the final implementation includes three parts (available on Rust Playground with the tests):
- a custom visitor that does the deserialization
- a function to use in struct fields
- an internal newtype for everywhere else
use std::collections::HashMap;
use std::fmt;
use std::hash::Hash;
use std::marker::PhantomData;
use serde::de::{IntoDeserializer, MapAccess, SeqAccess, Visitor};
use serde::{Deserialize, Deserializer};
struct VecAsMap<K, V>(PhantomData<(K, V)>);
impl<'de, K, V> Visitor<'de> for VecAsMap<K, V>
where
K: Eq + Hash + Deserialize<'de>,
V: Deserialize<'de>,
{
type Value = HashMap<K, V>;
fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
formatter.write_str("a map or an array")
}
fn visit_map<A>(self, mut access: A) -> std::result::Result<Self::Value, A::Error>
where
A: MapAccess<'de>,
{
let mut map = HashMap::with_capacity(access.size_hint().unwrap_or(0));
while let Some((k, v)) = access.next_entry()? {
map.insert(k, v);
}
Ok(map)
}
fn visit_seq<A>(self, mut access: A) -> std::result::Result<Self::Value, A::Error>
where
A: SeqAccess<'de>,
{
let mut map = HashMap::new();
let mut index: usize = 0;
while let Some(value) = access.next_element::<V>()? {
let key = K::deserialize(serde_json::Value::from(index))?;
map.insert(key, value);
index += 1;
}
Ok(map)
}
}
pub(crate) fn vec_as_map<'de, D, K, V>(
deserializer: D,
) -> std::result::Result<HashMap<K, V>, D::Error>
where
D: Deserializer<'de>,
K: Eq + Hash + Deserialize<'de>,
V: Deserialize<'de>,
{
deserializer.deserialize_any(VecAsMap(PhantomData))
}
#[derive(Debug)]
pub(crate) struct CustomMap<K, V>(HashMap<K, V>);
impl<'de, K, V> Deserialize<'de> for CustomMap<K, V>
where
K: Eq + Hash + Deserialize<'de>,
V: Deserialize<'de>,
{
fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
where
D: Deserializer<'de>,
{
vec_as_map(deserializer).map(CustomMap)
}
}
I had most of these parts figured out individually, but it was a PR from Veit Heller—a collaborator on the project—that help me make sense of the parts together. I realized that each part was solving a different purpose despite the code looking similar—a pattern I encountered in other aspects of the project too.
See also
- A PHP array conversion in the opposite direction, aka converting maps to arrays
- Serde’s example on deserializing a custom map type