Wednesday, November 1, 2023

Tips: Message Formats

There are just too many messaging format that we can choose from when building a project. The table below summarize a quick cheat sheet on which messaging format to choose for different use cases.
FormatBSONMessagePackApache AvroProtocol Buffers (protobuf)JSON
Features- Binary format with rich data types- Compact, lightweight binary format- Binary serialization format with schema- Strongly typed, schema-driven format- Human-readable text format
- Supports complex data structures- Supports common data types- Self-describing format with schema- Code generation based on schema- Simplicity and ease of debugging
- Supports specific MongoDB types- Schemaless (no predefined schema)- Support for evolution of data schema- Type safety and structured data- Requires minimal encoding/decoding overhead
- Field names included, leading to larger data size- Language-agnostic with wide language support- Language support in multiple programming languages- Widespread language support
Common Use Cases- MongoDB data storage and exchange- Cross-platform data interchange- Data storage and exchange (Hadoop, Kafka, etc.)- High-performance data serialization (e.g., gRPC)- Web APIs and data interchange
- Specifically suited for MongoDB ecosystem- Microservices communication- Big data processing and analytics (Apache ecosystem)- Language-agnostic communication in large-scale systems- Configuration files and settings
- IoT and embedded systems- When schema evolution and versioning are important- Google ecosystem and related projects (e.g., gRPC)- When human readability is important