RabbitMQ.Dataflows

RabbitMQ & Dataflows

A RabbitMQ library of .NET tools to help rapidly develop well performant services or to just help manage durable connectivity with the RabbitMQ.Client!

Status

Updated: 05/03/2024
Test Server: Windows 11 Pro (v10.0.22631 [Build 22631])
RabbitMQ Server: v3.13.2
Erlang: v26.2.5

Why Make A RabbitMQ Powered Dataflow?

Dataflows have configurable concurrency, serialization, monitoring, compression, and encryption all as first class citizens. This paradigm allows developers to just focus on the important stuff - getting work done!

Here are some features ready with RabbitMQ.Dataflows.

Workflows

• Pipelines (v1) are light weight workflow-esque function-as-step executor.
  • All steps process in the order provided allowing you to still control order of execution.
  • Configurable parallelism and concurrency built-in.
  • No automatic OpenTelemetry.
  • No automatic exception catching/async error handling (thrown exceptions kill Pipelines).
• Dataflows (v2) are the more complex, yet complete, workflow class.
  • All steps process in the order provided allowing you to still control order of execution.
  • Configurable parallelism and concurrency built-in.
  • Async processing, batch processing, consumer cloning and more!
  • Async error handling (simplify functional error handling by allowing functions to throw).
  • AutoPublish (to the next Queue) functionality built-in.
  • OpenTelemetry with native distributed tracing for Publish/Consumer.

Built-Ins

• A RabbitMQ IConnectionPool and IChannelPool (connection durability).
• Supports ILogger<T> via LogHelpers static singleton.
• Contracted IWorkState simplifies functional generic returns and integration.
• Has ISerializationProvider support for System.Text.Json and MessagePack.
  • Easy to write your own providers.
• Has ICompressionProvider built-in support for Gzip, Deflate, Brotli, and LZ4.
• Has IEncryptionProvider built-in support for AesGcm and BouncyCastle AesGcm.
• RecyclableMemoryStream options for ICompressionProvider and IEncryptionProvider.
• Publishers AutoPublish seamless encryption/decryption steps (adds headers too).
• Consumers allow seamless compression/decompression steps (uses headers).
• Publisher/Consumers have built-in OpenTelemetry support.

Testing

• All built-in steps will have integration tests that should remove concerns from end-user developer.
• Future case will include much more complex abstract UnitTesting as time allows.
• The developer should only need to unit test their functional business code.

Guides, Code, and Examples

• Getting started with RabbitMQ.Dataflows ConnectionPool.
• Getting started with RabbitMQ.Dataflows ChannelPool.
• Getting started with RabbitMQ.Dataflows ChannelPool and BasicPublish.
• Getting started with RabbitMQ.Dataflows ChannelPool and BasicGet.
• Getting started with RabbitMQ.Dataflows ChannelPool and BasicConsume.
• Getting started with RabbitMQ.Dataflows Serialization.
• Getting started with RabbitMQ.Dataflows Publisher.
• Getting started with RabbitMQ.Dataflows AutoPublisher.
• Getting started with RabbitMQ.Dataflows Consumer.
• Getting started with RabbitMQ.Dataflows ConsumerDataflow.

More to come and visual aids can be found towards the bottom!

You can also find various library examples inside the tests/UnitTests or the tests/RabbitMQ.Console.Test project.

Main RabbitMQ Library

HouseofCat.RabbitMQ

A library that focuses on RabbitMQ connection and channel management to create fault tolerant Publishers and Consumers.
Formerly called CookedRabbit.Core/Tesseract.

DataFlow Library

HouseofCat.Dataflows

A library that provides the base magic Dataflows for RabbitMQ.Dataflows.

• Custom TPL Block - ChannelBock used as a Channel-based BufferBlock<TIn>
• Has DataFlowEngine and ChannelBlockEngine.
• Has Pipelines (Dataflow alternative).

Core Productivity Libraries

These libraries are here to help you build powerful Dataflows for your messages.

HouseofCat.Serialization

A library that has a collection of .NET ISerializationProvider or the interface to make your own.

• Supports MessagePack and System.Text.Json and Newtonsoft.Json.

HouseofCat.Compression

A library that has a collection of .NET ICompressionProvider or the interface to make your own.

• Supports LZ4, Gzip, Brotli, and Deflate.
• Supports RecyclableMemoryStream variants.

HouseofCat.Hashing

A library that focuses on implementing hashing.

• Supports Argon2.

HouseofCat.Encryption

A library that provides encryption contracts and the base AesGCM/AesCBC .NET IEncryptionProvider as well as the interface to make your own.

• Supports AesCbc via CryptoStream (good for encrypted file/memorystreams).
• Supports .NET AesGcm 128, 192, 256 (non-streams).
• Supports BouncyCastle AesGcm 128/192/256.
• Supports RecyclableMemoryStream variants.

HouseofCat.Utilities

A library that focuses on general purpose re-usable functions and extensions that simplify the coding experience.

Non-Critical Library Integrations

HouseofCat.Data

A library that provides the provides helper classes for data manipulation and transformation.

Also provides Database abstractions, a simple Dapper integration, and SqlKata integration SQL query generation.

Database Connection Factory Support

• System.Data.SqlClient
• Microsoft.Data.SqlClient
• MySql.Data.MySqlClient
• Npgsq
• MySql.Data
• Oracle
• SQLite

Visual Aids

I have included a high level Excalidraw file with screenshots below. Be sure to open the file though for better quality and more up-to-date information as I will continue enhancing it over time. The goal isn't to have everything in here, but I myself understand the flow of a library with the addition of a visual aid. There is nothing truly special about this library, it's just the basics, but there are a lot of moving parts when you step back.

RabbitService - Start Life Cycle

Consumer - Start Life Cycle

ConsumerDataflow - Data Flow

HouseofCat.io