Announce: the Piqi project

Anton Lavrik alavrik@REDACTED
Mon Sep 13 10:07:35 CEST 2010

It is my pleasure to announce the Piqi project to the Erlang community. I've
just released a new version which features Erlang support.

Piqi is a cross-language data serialization system compatible with Google
Protocol Buffers with native support for Erlang and OCaml.

OCaml and Erlang programmers can use Piqi to conveniently serialize and
deserialize data using a portable binary format which can be understood by
programs written in Python, Java, C++ and other languages supported by Google
Protocol Buffers. In this sense, Piqi provides Protocol Buffers support for
OCaml and Erlang.

Compared to Protocol Buffers, Piqi relies on a richer data definition language
which has a more natural mapping to functional languages. For instance, in
addition to records and enums, Piqi supports variants (tagged unions), variant
subtyping, lists and type aliases.

Some other Piqi features include:
- Support for data schema evolution (Protocol Buffers style).
- "import" and "include" for reusing type definitions from other modules.
- Powerful extensions mechanism for records and variants.
- Structured default values for optional fields.

Compatibility between Piqi and Protocol Buffers is maintained as follows:
- Piqi specifications can be automatically converted to equivalent Protobuf
  specifications and vice versa (with very few exceptions).
- Piqi and Protocol Buffers use the same binary encoding as a serialization

Piqi comes with several additional tools:
- Piq -- a high-level data representation language. It has full support for Piqi
  data types and it is easier to read, write and edit compared to XML or JSON.
- Piqi tools for validating, pretty-printing and converting data between Piq,
  JSON and portable binary encoding.

Erlang support highlights

- Generated Erlang type and record definitions as well as
  serialization/deserialization code are EEP8-compliant and provide all necessary
  type information for Dialyzer.

- Support for recursive record types.

- Support for Unicode strings which can be passed as string() (i.e. list of
  Unicode codepoints) or utf-8 binary().

- Support for custom Erlang identifiers. For example, if some identifier from
  existing Proto or Piqi specification conflicts with an Erlang keyword, it is
  possible to define a custom Erlang-specific identifier without changing the
  original one and breaking backward compatibility.

- Capitalized and "CamelCase" Piqi/Protobuf identifiers are automatically
  converted to uncapitalized and "camel_case" Erlang ids.

- Configurable prefixes for type and record identifiers (since Erlang doesn't
  have any kind of namespace support for types and records, we need to prefix
  them to avoid conflicts with names defined in other modules).

- Tested with Erlang/OTP R13B04 and R14A.

Further references


GitHub repository and several Erlang examples:

Details of Piqi to Erlang mapping:

Your comments, suggestions and contributions are welcome! Don't hesitate to
contact me if you have any questions.


More information about the erlang-questions mailing list