Serializers

Serializers provide a mechanism for some data types to be copied from one process to another by converting Python objects into byte strings and vice-versa. Currently, two serializer classes are supported:

• msgpack provides efficient serialization for all supported types, including large binary data.
• json is somewhat less efficient in encoding large binary data, but is more universally supported across platforms where msgpack may be unavailable.

The basic types supported by both serializers are int, float, str, dict, and list. Further data types are serialized by first converting to a dict containing the key ___type_name___ in order to distinguish it from normal dicts (see Serializer.encode() and Serializer.decode()):

datetime = {
    '___type_name___': 'datetime',
    'data': obj.strftime('%Y-%m-%dT%H:%M:%S.%f')
}

date = {
    '___type_name___': 'date',
    'data': obj.strftime('%Y-%m-%d')
}

nonetype = {
    '___type_name___': 'none'
}

objectproxy = {
    '___type_name___': 'proxy',
    'rpc_addr': obj._rpc_addr,
    'obj_id': obj._obj_id,
    'ref_id': obj._ref_id,
    'type_str': obj._type_str,
    'attributes': obj._attributes,
}

Types containing byte strings are handled differently between msgpack and json. In msgpack, byte strings are natively supported:

np.ndarray = {
    '___type_name___': 'ndarray',
    'data': array.tostring(),
    'dtype': str(array.dtype),
    'shape': array.shape
}

# no need to convert; msgpack already handles this type
bytes = bytes_obj

However json does not support byte strings, so in this case the strings must be base-64 encoded before being serialized:

ndarray = {
    '___type_name___': 'ndarray',
    'data': base64.b64encode(array.data).decode(),
    'dtype': str(array.dtype),
    'shape': array.shape
}

bytes = {
    '__type_name__': 'bytes',
    'data': base64.b64encode(bytes_obj).decode()
}

Note that both serializers convert tuples into lists automatically. This is undesirable, but is currently not configurable in a consistent way across both serializers.

It is possible to add support for new serializers by creating a subclass of Serializer and modifying pyacq.core.rpc.serializer.all_serializers.

class pyacq.core.rpc.serializer.Serializer(server=None, client=None)

Base serializer class on which msgpack and json serializers (and potentially others) are built.

Subclasses must be registered by adding to the all_serializers global.

Supports ndarray, date, datetime, and bytes for transfer in addition to the standard types supported by json and msgpack. All other types are converted to an object proxy that can be used to access methods / attributes of the object remotely (this requires that the object be owned by an RPC server).

Note that tuples are converted to lists in transit. See: https://github.com/msgpack/msgpack-python/issues/98

decode(dct)

Convert from serializable objects back to original types.

dumps(obj)

Convert obj to serialized string.

encode(obj)

Convert various types to serializable objects.

Provides support for ndarray, datetime, date, and None. Other types are converted to proxies.

loads(msg)

Convert from serialized string to python object.

Proxies that reference objects owned by the server are converted back into the local object. All other proxies are left as-is.