Protocol Buffers_01: Getting Started with Protobuf Basics (English Version)

"이 블로그 포스트는 영어로 작성되었지만, 영어에 익숙하지 않은 분들을 위해 다음 글에서 한국어 버전도 제공하고 있습니다."

 

---

1- Introduction to Protocol Buffers

Protocol Buffers (Protobuf) are a method for serializing data, which means they convert data into a format that can be easily transmitted over a network or stored in files. While other formats like JSON and XML are also used for this purpose, they are not as efficient for communication between multiple microservices in a way that works across different platforms. This is where Protobuf stands out.

Protobuf is faster and more efficient than JSON or XML because it is designed to quickly serialize (convert to a format for transmission) and deserialize (convert back to its original format) data. The structure of the data is defined in special files called .proto files. These files describe the data format using structures known as messages.

Here are some key features that make Protobuf a popular choice:

1- Binary Transfer Format: Protobuf sends data in a compact binary format, which saves space and bandwidth.
2- Separation of Context and Data: Protobuf keeps the data and its context separate, unlike JSON and XML.

lets explain about this part as it might sound complicated

Imagine you want to send information about a person, like their name and age.

Using JSON
In JSON, you would write the data like this:

{
 "name": "Alice",
 "age": 30
}

 

What’s happening here?
The keys (“name” and “age”) tell you what the data means.
The data is mixed with its meaning.
This means every time you send this data, you have to include the keys, which makes the overall size larger.

 

Using Protobuf
In Protobuf, you first define the structure in a separate file (let’s say person.proto):

message Person {
  string name = 1;
  int32 age = 2;
}

What’s happening here?
You define what a “Person” is and what information it includes (name and age).
When you actually send the data, you only send the values, not the keys. The context (what “name” and “age” mean) is already known from the .proto file.

3- Message Format: Data is serialized and deserialized based on the configurations defined in the .proto files.

2- Step-by-Step Guide: Getting Started with Protobuf

1. Install Protocol Buffers Compiler

To use Protocol Buffers, you’ll first need to install the compiler (protoc).

pip install protobuf

2. Create Your First .proto File

Let’s define a simple .proto file for a user profile. Create a file named user.proto:

syntax = "proto3";

message User {
  string name = 1;
  string email = 2;
  int32 age = 3;
}

• syntax = "proto3";: Specifies the Protobuf version.
• message User: Defines a structured data type.
• Each field has a type (e.g., string, int32), a name, and a unique number.

In the .proto file, we only define the structure of the data. For example, we specify that the User message has a name, email, and age field, but we don’t provide any actual data values. The values (like "John Doe" for name, or 30 for age) are provided later in the program when we create a User object.

When we serialize the data (convert it into binary format), it’s the values of the fields — such as name, email, and age—that get encoded. The serialization process transforms these field values (e.g., "John Doe", "johndoe@example.com", 30) into a compact binary format, which is more efficient for transmission and storage.

The field numbers (1, 2, 3) specified in the .proto file are crucial. They act as identifiers for the fields when the data is serialized. These numbers help Protobuf know how to map the fields when it reads the binary data back into an object.

3. Compile the .proto File

Run the following command to generate code for your language (e.g., Python):

protoc --python_out=. user.proto

This will generate a Python file (user_pb2.py) containing code to work with the User message.

4. Use the Generated Code

Here’s a Python example of how to use the generated code:

import user_pb2

# Create a new User object
user = user_pb2.User()
user.name = "John Doe"
user.email = "johndoe@example.com"
user.age = 30

# Serialize the user object to binary
serialized_user = user.SerializeToString()
print("Serialized User:", serialized_user)

# Deserialize the binary data back to a User object
new_user = user_pb2.User()
new_user.ParseFromString(serialized_user)
print("Deserialized User:")
print(f"Name: {new_user.name}, Email: {new_user.email}, Age: {new_user.age}")

Output is as follows:

Name: John Doe, Email: johndoe@example.com, Age: 30

---

 

 

Explanation of the above Python Code

1. Import the Generated Protobuf Code:

import user_pb2

• imports the user_pb2 module, which was automatically generated when you compiled the user.proto file with the Protobuf compiler (protoc). This module contains the Python classes that correspond to the messages you defined in the .proto file. In this case, it includes the User class that we can use to work with user data.

2. Creating a New User Object:

• The following code creates a new instance of the User message:

user = user_pb2.User()
user.name = "John Doe"
user.email = "johndoe@example.com"
user.age = 30

• user = user_pb2.User() initializes a new User object. This object is essentially an instance of the User message, ready to hold the data that we want to serialize.
• After creating the object, we assign values to its fields (name, email, and age) using the = operator. These values correspond to the fields we defined in the user.proto file.

3. Serialize the Data to Binary:

serialized_user = user.SerializeToString()
print("Serialized User:", serialized_user)

• The SerializeToString() method converts the User object into a binary format.
• This binary data is much more compact and efficient for transmission or storage compared to string-based formats like JSON or XML.

4. Deserialize the Binary Data Back to a User Object:

new_user = user_pb2.User()
new_user.ParseFromString(serialized_user)
print("Deserialized User:")
print(f"Name: {new_user.name}, Email: {new_user.email}, Age: {new_user.age}")