Protobuf over RabbitMQ

Discussion in 'Computer Science & Culture' started by Crunchy Cat, May 3, 2012.

Thread Status:
Not open for further replies.
  1. Crunchy Cat F-in' *meow* baby!!! Valued Senior Member

    Messages:
    8,423
    Anyone used this combination before? If so, what has the experience been like? I just set up RabbitMQ today and found it very cool, but I do want to encapsulate my data with Protobuf.
     
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. Crunchy Cat F-in' *meow* baby!!! Valued Senior Member

    Messages:
    8,423
    I *suspect* nobody has any idea what I am talking about :3
     
  4. Google AdSense Guest Advertisement



    to hide all adverts.
  5. Chipz Banned Banned

    Messages:
    838
    I use protobuf with the C++ and Python bindings for RPC and as a message-passing system with introspection (sometimes).

    I think you may have a confusion about what Protobuf is. You first take a meta-language like..

    Code:
    message some_message
    {
      required string variable = 1;
      required int64  intvar    = 2;
    }
    
    You then use the application protoc to generate NATIVE code for other languages.
    Code:
    protoc message.proto --cpp_out=. 
    And it will generate message.h, message.cc.

    Currently, Protobuf ONLY supports Java, Python and C++. There are 3rd party developers who have implemented other languages, you'd have to see if Erlang implementations exist.
     
  6. Google AdSense Guest Advertisement



    to hide all adverts.
  7. Crunchy Cat F-in' *meow* baby!!! Valued Senior Member

    Messages:
    8,423
    Sounds good.

    I am not sure what I said that might have made you think I wasn't clear on what protobuf is?

    Sounds accurate.

    RabbitMQ is a criteria based, queued, network, and pub/sub message router. I am pretty new to it but I have done a ton of work with it in the past 1.5 days. I am going to try to make a simple RPC client in the morning and after that was going to try to make it send Protobuf messages across an SSL channel.

    My interest is in what people's experience is when combining Protobuf and RabbitMQ. The answer may simply be that there is no experience in that area right now.
     
  8. Chipz Banned Banned

    Messages:
    838
    Well, doesn't RabbitMQ itself have an MPIC and RPC mechanisms?
     
    Last edited: May 4, 2012
  9. Crunchy Cat F-in' *meow* baby!!! Valued Senior Member

    Messages:
    8,423
    RabbitMQ lets you establish channels and endpoint routing of bytes. Anything else you have to program yourself. Protobuf of course gives you some message standardization.

    I was able to successfully emulate RPC with RabbitMQ this morning. It did it quite well I might add. I am working on getting Protobuf integrated.
     
  10. Chipz Banned Banned

    Messages:
    838
    Keep me informed.
     
  11. Crunchy Cat F-in' *meow* baby!!! Valued Senior Member

    Messages:
    8,423
    The result is success. Protobuf works very easily of RabbitMQ.
     
  12. Chipz Banned Banned

    Messages:
    838
    How are you doing it? What does your implementation look like? Please elaborate! By the way, my issue was a misunderstanding of RabbitMQ. I thought it was a RPC implementation based on message passing rather than a message passing platform alone.
     
  13. Crunchy Cat F-in' *meow* baby!!! Valued Senior Member

    Messages:
    8,423
    The server application sets up a named request queue:

    Connection connection = factory.newConnection();
    Channel channel = connection.createChannel();
    channel.queueDeclare("RequestQueue", false, false, false, null);

    The client declares an anonymouse queue:

    Connection connection = factory.newConnection();
    Channel channel = connection.createChannel();
    String responseQueueName = channel.queueDeclare().getQueue();

    The client sets up some queue properties that can be attached to a message. The important part is to note is that I am encoding the name of the response queue.

    String corrId = java.util.UUID.randomUUID().toString();
    BasicProperties props = new BasicProperties
    .Builder()
    .correlationId(corrId)
    .replyTo(responseQueueName)
    .build();

    The client builds my Protobuf message and converts it to a binary stream:

    TestEntity.Builder builder = TestEntity.newBuilder();
    builder.setName("Crunchy Cat");
    builder.setId(666);
    TestEntity data = builder.build();
    ByteArrayOutputStream oStream = new ByteArrayOutputStream();
    data.writeTo(oStream);

    The client sends the binary message to the queue with the properties I set earlier:

    channel.basicPublish("", requestQueueName, props, oStream.toByteArray());

    The server is listening for a message:

    QueueingConsumer.Delivery delivery = consumer.nextDelivery();

    Once the server receives the binary message, it hydrates a protobuf message:

    ByteArrayInputStream iStream = new ByteArrayInputStream(delivery.getBody());
    TestEntity.Builder builder = TestEntity.newBuilder();
    builder.mergeFrom(iStream);
    TestEntity data = builder.build();

    Now the server is free to interpret the message and respond via the anonymous queue in the message properties. In this case it will just do a plain text response for simplicity:

    BasicProperties props = delivery.getProperties();
    BasicProperties replyProps = new BasicProperties
    .Builder()
    .correlationId(props.getCorrelationId())
    .build();
    String response = "Crunchy devil cat accepted";
    channel.basicPublish("", props.getReplyTo(), replyProps, response.getBytes());

    Meanwhile, the client is listening for a response:

    QueueingConsumer.Delivery delivery = consumer.nextDelivery();

    Once the response is received, the client can process it:

    if (delivery.getProperties().getCorrelationId().equals(corrId))
    {
    String response = new String(delivery.getBody());
    // Do something with the response.
    }

    And that's it :3.
     
Thread Status:
Not open for further replies.

Share This Page