Meet our team- Nikola Štuban

Meet our team- Nikola Štuban

If you are curious to know what Nikola Štuban, one of our Software engineers in Zagreb answered during our latest interview #MeetOurTeam then check out the full blog post. 

 

TACTA: Hi Nikola, please introduce yourself in a sentence.

Nikola: Hi, my name is Nikola, nice to meet you!

TACTA: Now you’re part of our team in Zagreb but could you tell us more about what’s your background in IT industry?

Nikola: I started my professional career 5 years ago during my college as a junior java developer in Corvus Info – a company that is part of a bigger MSAN group located in Zagreb, but also has its offices throughout the SEE region. Primarily, Corvus Info is making our everyday lives better by making it easier for customers to pay for goods and services online. Following that goal, they developed CorvusPay, the most popular internet payment gateway solution in Croatia. I really enjoyed having an amazing opportunity to work on such an unbelievably sensitive piece of software, that also needs to be very scalable and highly available, and at the same time does financial transactions, communicates with the banks and card processing software, has complex architectural solutions, etc.

After more than 2 years in Corvus, where I met a lot of fantastic experts and friends, I joined the TACTA team at the beginning of 2017. Although working on payment software in Corvus was a great experience, I was always seeker for a closer contact with the people we are working for – clients of all sorts, stakeholders, but also the end-users. In my opinion, there is no higher award at the end of the project than seeing a happy, satisfied client whose life is easier and better because of the software we delivered. I think you will agree with me that software can be (hypothetically speaking) perfectly written, but if the client at the end feels like that’s not what they were seeking – we have not fulfilled our mission.

Being a bridge in overlapping that gap speaking with the clients and understanding their needs, while at the same time using best practices and technologies in development, plus understanding the whole picture of how the specific system works – that was always my ideal environment. That is exactly what was made possible for me in TACTA. I had that opportunity the first day I started working here, and I’m still enjoying it every day.

TACTA: How did you cross paths with Tacta?

Nikola: I found TACTA software engineer advertisement randomly, and it seemed interesting – so I applied for the job. After figuring out that one of my colleges from the previous company was already working here, I decided to meet the team from Zagreb and Sarajevo. The rest is history.

TACTA: IT industry is growing rapidly, do you have any advice for Software engineers to be?

Nikola: I could give a million different advice here, but the most important things (even they might sound like a cliché) are to act naturally, to be kind, professional, and not to be afraid of asking questions. Just ask anything you don’t understand, without fear. I will also mention a quote from Vlad Mihalcea (Java Champion): “The best teams I’ve ever worked on were not made of 10x developers who knew everything, but of developers that were kind, funny, humble and professional.

TACTA: Corona changed our daily routine for sure but could you tell us what your typical workday looks like before and during the Corona?

Nikola: I have to say that my typical workday is not that different at this moment than it was before the COVID-19 pandemic. True, we were in sort of a lockdown during Spring this year when we all worked from home, but since June we are back in the office and more or less we continued to keep up with that practice. Sometimes we all decide to work from home for a couple of days, but that’s it. We will see what Autumn and Winter brings in terms of the pandemic, but if work from home 5 days a week will (again) became our reality, I am not concerned at all, since we already proved the ability to work extraordinary in those conditions earlier this year (and also developed some amazing apps completely in the lockdown period).

TACTA: Back to the work. Tell us what’s the most interesting challenge you’ve come across since you joined our team?

Nikola: The most interesting challenge to me is, as already mentioned, to overcome the lap between the clients with their requirements and software developers on the other side while using best practices and methodologies in the profession. I strongly believe that those two sides, if approached correctly, are not opposed, but complementary. In TACTA I had an opportunity to learn that from the best people in this area, and I still have a lot to learn.

TACTA: What’s your favorite desk snack?

Nikola: Normally I avoid having snacks on my desk since it usually ends up in peanuts all over my keyboard or monitor covered with ice cream.

TACTA:Share the secret, what’s your hidden talent?

Nikola: I would like to keep it hidden.

TACTA: When you’re not at Tacta, what’s your favorite thing to do?

Nikola: I am the father of two beautiful kids (2 years old son and 2 months old daughter), so when not occupied with work, I am trying to spend as much quality time as possible with them and my wife.

What did math teach me about business analysis

What did math teach me about business analysis

“The essence of mathematics is not to make simple things complicated, but to make complicated things simple.” –  S. Gudder

When we say the term business analysis, usually the main associations are exhaustive documents about business models and data analytics, and sometimes, if you allow yourself to be a bit exotic, you will think about a few diagrams in form of graphs or flowcharts (how wild, right?).

In my few years of experience as a developer, I was invested in solving problems and, as almost every developer, was addicted to the daily process of logical thinking and the general feeling of accomplishment once you solve a problem, however trivial or not it may be. When you think of business analysis, somehow you don’t get the feeling that you’ll be invested in a lot of creative solution design.

When I decided to change my career, I was worried – will I know how to live a day without these things? I spent years developing logical skills, mathematical skills, growing into an engineer and now I found myself thinking – will all those skills go to waste?

However, I’ve found that approaching business analysis can be quite an interesting mental exercise. Contrary to being a developer, where problem-solving is your main occupation (a very general phrasing), by being a business analyst your main occupation is defining the problem.

Once I realized that I am not that far away from my usual thinking process, my fetish kicked in.

SIDE NOTE: I know the popular phrasing of problem in IT nowadays is CHALLENGE, however, being a mathematician at my core, I am not scared of the word, since I solved like a billion of them in my freshmen year only, therefore I will not adhere to the modern standards in this part.

A problem is mainly and only an equation – meaning that, in most of the cases, it is solvable. Not only is it solvable, but it is also solvable in multiple ways and can have even more than one solution. Isn’t this comforting to hear? So, no need to use the tacky word challenge.

In the following text, I will try to compare certain aspects of business analysis to some typical mathematical aspects. So let’s start with some basic math.

 

The n + 1 proof

At the very beginning of your abstract mathematical journey, they teach you that, to prove a certain theorem (or in plain English – a statement) you need to prove that it is true for all known cases in the world (so, for every n that belongs to the set of N, which is the set of all-natural numbers).

Ok. To actually (mathematically) prove that, you need to find a (creative) way to show that it is true in the (n + 1) case. What is the (n + 1) case anyway?
Well, since you said that it should be valid for all n from the set of N, and the number n can reach infinity, your claim should also be valid in case your buddy n decides to level up to infinity + 1.

If you’re already a bit lost, no pressure. I have a shortcut.

Since all of this mentioned above seems like a lot of hassle, I’ll tell you what I do. You don’t need to go through all those cases in your head – you only need to find one that doesn’t work. Just one n, to put you out of your misery. One n that says your statement, well, is false.

In business analysis, we call this the edge case.

Whenever contemplating a certain business case, either with the customers or together with the team, once someone proposes a possible solution, I immediately rewire my brain from trying to find a solution to trying to break a solution mode (I do this to my proposed solutions too).

An edge case is not only an exception – it is a proof that your proposed way just does not work for all n from the set of N. In the math world, but that is also a deal-breaker, and to be honest, I wish it was the same in designing IT solutions. However, in IT, sometimes we accept solutions that only work in most cases, but afterward we scratch our heads when the unthinkable happened – the edge case has emerged in Production.

Mathematics dwells in perfectionism and we certainly cannot transfer the same rulings to the real world. However, as I mentioned above, a problem can have multiple solutions, and let’s try and find that buddy n that does not break once an edge case happens.

 

Suffering through solutions

One time during my studies, a few days after attempting an exam, the professor approached me while I was standing by the elevator.

“I cannot believe what you did!”- he said.

The exam I took was mainly about solving integrals. I was satisfied after I gave in my papers, even thinking about a good grade. Of course, that lasted approximately 15 mins – everything fell apart once I talked to my colleagues about the solutions and figured mine seemed nothing like theirs.
However, sometimes in math, if you apply a different approach to a problem it might seem that you’ve got a different solution. You see, since the result is a function and functions sometimes do not look alike but BEHAVE THE SAME (so, they’re the same), I still had faith that my solution could be correct.
I mean, I checked it multiple times. Line after line. Page after page…

“What? Page after page? How long did your approach take? I solved it in like two rows with a simple substitution.” – said my friend after the exam.

Well, I did not. I solved the task but it took me 3 pages. 3 pages of raw mathematical calculus. Even the darkest math kinksters would shudder at that sight.

“Once I saw your task, I immediately thought it must be wrong. However, as your professor, I am obliged to inspect it. I was hoping that you would make an error at some point so I wouldn’t have to go through all of it, but no, you did it. You solved it correctly. But man, did I suffer through it!” – the professor said while storming off into the elevator, leaving me behind, probably because he could not bear the sight of me anymore. I understood since I basically put that man through torture.

To draw a parallel line here – imagine the professor being a client and me being a business analyst, where the task from the exam was his business problem – what would his user experience be? From 0 to 10, probably much below 5 (and it’s a stretch), if we map that talk by the elevator to a feedback meeting for instance. The product works, but the client is suffering while using it (2/10).

The point is, it is not only important that you find a correct solution to your client’s problems, but it is also important that you try and optimize their way of working. Every working solution does not mean it’s acceptable. If the client needs to click and scroll multiple times for information you know they’ll be using constantly, you need to rethink your design. As a business analyst, you are required not only to identify the problem and figure a way out, you are required to understand the product, but also all the possible pains of it.

Bonus tip: Use your development team as allies on this one, they are far more advanced in matters of removing nonsense from solutions.

Understanding by magic

One long summer during my studies, I attempted to learn mathematical theory for the very first time. For those of you who do not know how mathematical theory looks like – first question: how does it feel to be God’s favorite? Secondly, mathematical theory is just like a description of a theorem (statement) and its proof. Or counterproof. Only not in sentences, but in mathematical symbols. When you first see it, it kind of looks a bit schizophrenic.

Day after day sitting by the book, it just wouldn’t go in my head. Luckily, one of those days my brother came along who already graduated from a similar faculty, recognized my agony, and gave me some advice.

Little did I know, that to learn mathematical theory you first need to learn how to learn it. You need to learn it by heart, but with understanding.

Let’s focus a bit on the understanding part.

When you see a bunch of symbols, your mind naturally tries to disregard as much as possible of it, in order to preserve memory. But this won’t help you much in the understanding part. Symbols in the equations don’t appear by magic, so you certainly cannot even try remembering them by heart without asking yourself where does every single symbol come from? And why? It’s not magic, it’s math –  and math is precise and can always tell you where something comes from and what is the purpose of it. Once I realized that it made much more sense to me.

Looking at this from a project perspective – let’s say you’re dealing with a service placed somewhere in a complex IT ecosystem. Imagine that all the symbols mentioned above in math. theorem are little services that somehow communicate with each other inside this architecture. Applying my first method would mean you are greatly skilled in your service domain knowledge, but generally, you’re not sure where your service is actually placed in the ecosystem and what is happening around it.

As a good business analyst and business developer, you need to understand the purpose of all those services, when and how they communicate, and to what purpose. You cannot isolate yourself only to your own domain, since your service actually communicates to the outside world and intersects with the others. If you do, you are at great risk of designing solutions which are not true, not complete, and not satisfactory.

A problem can occur somewhere outside your service model, and sooner or later, it will become your problem too. It’s the same in math – one equation after the other, at one point you’re going to figure out that the symbol you so easily disregarded, now is the key factor to the solution.

———————–

To conclude, just like with business analysis described by exhaustive documents, you could look at math the same way. But why should you? Math was never meant to be that way, nor should the business analysis be. That’s just boring. Abstract thoughts of certain processes – be it the way your application responds to a certain API call triggered by an outside service, or be it the longitude of a tangent – in a similar way they coexist, trying and reaching out to you to find the most elegant solution.

Emina Džuzdanović

Implementing Event Store in C#

Implementing Event Store in C#

What is this all about?

When looking for examples of EventStore in Event Sourcing implementation, you will probably find a lot of them, very detailed in theory, but lacking practical implementation. In this article my goal is to explain how to implement simple, yet robust, event store, test it and what are the pitfalls you should expect based on my own experience.

Event store presented here will be implemented in .NET Core, C# and MS SQL LocalDB server as a database.

Full project which contains working solutions is available on my github

Short introduction to Event Sourcing

I will not go into much detail about what Event Sourcing is since this topic is covered in many books and articles around the web.

Instead, I will focus on what is important when considering implementing an actual Event Store. But, nevertheless I will quickly summarize the main idea behind Event Sourcing and main challenges.

Aggregates are considered as a whole represented by the Aggregate Root. Conceptually an Aggregate is loaded and saved in its entirety. (Evans, 2001).

Event Sourcing is a software architecture pattern which states that state of the application should be persisted as sequence of Events. Main benefits of Events Sourcing are:

  • Auditability — since state is constructed from sequence of events it is possible to extract detailed log since the begging and up to current date
  • Projections/queries — it is possible to recreate state in different time frame since we have all the events from the beginning. For example it would be possible to check what was the bank account state one year ago. This also allows to generate some queries/reports we never thought of when starting the system, since all the data is in the Events.
  • Performance when inserting data — since EventStore is append only store with optimistic locking, we would expect to have much less deadlocks(or concurrency exceptions) happening. Also, no long running transactions which insert graph of data in multiple tables.
  • Flat database structure — we would usually use 1(or max 2 tables) as event store. In both cases it would be de-normalized form with some form of weak serialization field to store payload of the event like JSON for example. This means that when adding new fields to the database it is not needed to add them to any table — simply adjusting Event and adding required field will save it into JSON. This allows much more rapid write side development time

As with every pattern we must be aware of limitations/challenges. If used incorrectly, Event Sourcing will probably cause more harm than good. So, the main challenges we should keep in mind are:

  • Storage Growth — since data store is append only, table will grow indefinitely. This can be mitigated using snapshots or retention policy strategies.
  • Replaying Events — if the amount of events for constructing Aggregate is large it might lead to some performance issue when reconstructing the current state of the aggregate. This can be mitigated by using snapshots.
  • Events versioning and events handling — when changing existing event, or adding/deleting features, a code which projected old events MUST be in place, since it is used to reconstruct the state to the actual state. This means that if some feature is deprecated, its code cannot be removed since it is used to reconstruct the state at that time. This challenge is a bit harder to overcome but it can be mitigated.

Event Store considerations

Requirements for the event store are the following:

  • It is append only which means there is no update only insert
  • It should store aggregate state and allow fetching events for given aggregate in order they were saved.
  • It should use optimistic concurrency check: Optimistic concurrency check does not use locking on database level, therefore reducing risk of deadlocks. Instead, concurrency check is done when saving.

Optimistic concurrency check

When inserting into a database, where multiple clients exist, it can happen that 2 or more clients are trying to modify the same aggregate. Since we don’t use pessimistic concurrency check, there will be no lock and no waiting, but the check itself would be applied when trying to persists the actual data.

To make things clear let us consider an example:

Assume that there are two requests that want to modify the same aggregate named Aggregate. Implementing concurrency check should be done on database level.

  1. Both of them will fetch current version from the event store which is 1
  2. First aggregate is saved, second aggregate version is set to 2
  3. Second aggregate in this case will fail concurrency check since its version is 2 and expected version is 3. This indicates that data has been changed since it was read. In this case saving second aggregate should fail with Concurrency exception.

Example of optimistic check using Aggregate version

Database schema

Event Store will be one table, append only, which allows version tracking per aggregate and implement concurrency check on database level.

SQL for Event Store table with JSON field for data(this is where event payload is serialized):

CREATE TABLE [dbo].[EventStore](
    [Id] [uniqueidentifier] NOT NULL,
    [CreatedAt] [datetime2] NOT NULL,
    [Sequence] [int] IDENTITY(1,1) NOT NULL,
    [Version] [int] NOT NULL,
    [Name] [nvarchar](250) NOT NULL,
    [AggregateId] [nvarchar](250) NOT NULL,
    [Data] [nvarchar](max) NOT NULL,
    [Aggregate] [nvarchar](250) NOT NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO

SQL for EventStore example table

AggregateId and Version are two fields used for concurrency check. We create Unique index with these two fields. AggregateId is the id of our aggregate and can be whatever we want(therefore it is defined as string). Depending on the domain it can be GUID, int, combination of two, it doesn’t really matter.

Note that AggregateId is defined as nvarchar(250)

CREATE UNIQUE NONCLUSTERED INDEX [ConcurrencyCheckIndex] ON [dbo].[EventStore]
([Version] ASC, [AggregateId] ASC) WITH (
    PAD_INDEX = OFF, 
    STATISTICS_NORECOMPUTE = OFF, 
    SORT_IN_TEMPDB = OFF, 
    IGNORE_DUP_KEY = OFF, 
    DROP_EXISTING = OFF, ONLINE = OFF, 
    ALLOW_ROW_LOCKS = ON, 
    ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO

Unique index is enforced on Version and Aggregateld fields on the table on database level

Using this we ensure that the same AggregateId/Version combination is never saved. Instead a unique index check failed exception is thrown by the database. This is a transient error, which means that retry mechanism(see Retry Pattern) can(and should) be implemented on the client side.

Example project short introduction

Project is built using .NET Core 3.1.

Project architecture is layered with inversion of control.

Layers:

  • RestAPI — Web API which contains DTO and REST Controller definitions
  • Infrastructure — Factories, database model and repository implementations are defined here
  • Core — contains business logic as well as repository interface for aggregate. This project has no references to any other project or any other third party libraries( except Tactical DDD nuget which is pure C# code )

Other projects:

  • DbMigration — migrations projects used to initialize database
  • EventStoreTests — testing project, which demonstrates integration tests for event store

For Core business logic, there is only one aggregate named Person and two domain events:

  1. PersonCreated — this event is published when person is created
  2. AddressChanged — this event is published when address for given person has changed

How to set up and run the project can be found in readme file for github repository.

EventStore implementation

Let us take a look at the actual code that implements the event store. I will put only code snippets here, while fully functional project can be found on my github.

Interface for the EventStore can be defined as:

public interface IEventStore
    {
        Task SaveAsync(IEntityId aggregateId, 
            int originatingVersion, 
            IReadOnlyCollection<IDomainEvent> events,
            string aggregateName = "Aggregate Name");

        Task<IReadOnlyCollection<IDomainEvent>> LoadAsync(IEntityId aggregateRootId);
    }

The interface defines two methods.

  • SaveAsync method is used to persist Aggregate as stream of events. The aggregate itself is described as a collection of domain events, with a unique name.
  • LoadAsync method fetches aggregate, using AggregateId as param, from the event store and emits it as an array of events. This array can be used to load the aggregate.

IEntityId and IDomainEvent are both imported from a Tactical DDD nuget, which I strongly recommend for DDD in C#. Both of these are simple interfaces which mark EntityId class and DomainEvent class.

Let us analyze the actual implementation of these two methods:

Persisting Events

For persisting Aggregate into the EventStore we need 3 parameters:

  1. AggregateId — this is the id of the aggregate. In our case it will be class which implements IEntityId interface
  2. OriginatingVersion — this is the version of the aggregate that is being saved. Once saved version is incremented by one. As explained this is used in optimistic concurrency check
  3. IReadOnlyCollection<IDomainEvent> — this is the actual list of events that needs to be persisted into database. Each event is persisted as new row.

Implementing SaveAsync

Full implementation for this method can be found in EventStoryRepository.cs file.

First, insert query is created using provided parameters. For this, we use micro ORM Dapper which allows mapping parameters using @ notation.

var query = $@"INSERT INTO {EventStoreTableName} ({EventStoreListOfColumnsInsert}) VALUES (@Id,@CreatedAt,@Version,@Name,@AggregateId,@Data,@Aggregate);";
var listOfEvents = events.Select(ev => new
{
  Aggregate = aggregateName,
  ev.CreatedAt,
  Data = JsonConvert.SerializeObject(ev, Formatting.Indented, _jsonSerializerSettings),
  Id = Guid.NewGuid(),
  ev.GetType().Name,
  AggregateId = aggregateId.ToString(),
  Version = ++originatingVersion
});

Parameter name ( @Name for example) is matched with the object property and mapped. That is why in the next line, a list of anonymous objects is created with the same properties as defined in query.

Properties are:

  • Aggregate – this is a string name for an aggregate.
  • CreatedAt — is a date/time when the event has been created
  • Data — this is the event payload, serialized as JSON string. Complete event is serialized into JSON using provided jsonSettings.
  • Id — this can be any type of id. For this example I used Guid.
  • Name — this is the actual event name
  • AggregateId — this is the id of the aggregate. Using this field, events for given aggregate can be filtered.
  • Version — increments each time for given aggregate. Used in optimistic concurrency check.

List of events is then mapped to the actual query using:

await connection.ExecuteAsync(query, listOfEvents);

This line of code is using ExecuteAsync method from Dapper ORM which will map listOfEvents properties with the parameters defined in query string and create actual queries.

When persisted in this way, each event is persisted as new row in EventStore table, with the Data payload of the actual event. Here is how it looks like:

Each Event is saved as new row in database. Version changes per Aggregate and sequence is always incremental

When inspecting Data column this is the payload:

{
“$type”: “Core.Person.DomainEvents.PersonCreated, Core”,
“PersonId”: “d91f903f-3fb1–4b68–9a59-c1818c94f104”,
“FirstName”: “damir6”,
“LastName”: “bolic7”,
“CreatedAt”: “2020–02–20T07:24:54.0490305Z”
}

The payload of this event is actually mapped from PersonCreated event which is emitted when new person is created:

public class PersonCreated : DomainEvent
    {
        public string PersonId { get; }
        public string FirstName { get; }
        public string LastName { get; }

        public PersonCreated(
            string personId, 
            string firstName, 
            string lastName)
        {
            PersonId = personId;
            FirstName = firstName;
            LastName = lastName;
        }
    }

This domain event is published when new Person aggregate is created

DomainEvent class can be defined as follows:

public class DomainEvent : IDomainEvent
    {

        public DomainEvent()
        {
            CreatedAt = DateTime.UtcNow;
        }

        public DateTime CreatedAt { get; set; }
    }

Basically, CreatedAt is added based on IDomainEvent from DDD Tactical nuget.

Loading aggregate

Loading aggregate is done using AggregateId. For given aggregate all events are loaded and then Aggregate is constructed using those events which in turn results with the new Aggregate object in memory.

Aggregate is loaded using SQL query as list of events using EventStoreRepository.LoadAsync() method. The real magic happens when events are deserialized from JSON and converted to DomainEvent object:

var events = (await connection.QueryAsync<EventStoreDao>(query.ToString(), 
                                                         aggregateRootId != null ? 
                                                         new { AggregateId = aggregateRootId.ToString() } : null))
    .ToList();

 var domainEvents = events.Select(TransformEvent).Where(x => x != null).ToList().AsReadOnly();
 return domainEvents;

Selecting all events for the aggregate and transforming them to domain events

As we can see, events are fetched as EventStoreDao list of object, which are then converted to DomainEvent using TransformEvent method:

private IDomainEvent TransformEvent(EventStoreDao eventSelected)
 {
 var o = JsonConvert.DeserializeObject(eventSelected.Data, _jsonSerializerSettings);
 var evt = o as IDomainEvent;
 return evt;
 }

Here, actual payload of the event eventSelected.Data is deserialized into object which is then converted to IDomainEvent interface. Note that, should this conversion fail it will return null.

Once list of domain events is fetched Person aggregate can be constructed.

Testing

Testing of the event store is not hard.

For unit testing it has defined interface IEventStore which can be mocked.

For integration tests – in memory database can be used. In the example project, LocalDB is used for both testing and in actual project. This is located in the EventStoreIntegrationTests.cs file.

Summary

The goal of this blog is to show using concrete example how to implement simple Event Store in C#. For this we used some of the DDD concepts like Aggregate, Repository, Entity and ValueObject.

Example project, included with this blog, aims to be simple demonstration of the principles defined here.

Author: Damir Bolić

Life after Event Sourcing

Life after Event Sourcing

I am not going to talk about implementing Event Sourcing, pros and cons, or when and where to use it. I want to share my personal developer’s perspective and experience gathered from the two projects I worked on. Both were aiming to develop microservices using domain-driven design principles. The first project (let us call it A) had Event Sourcing and the second one (project B) did not. In both cases, a relational database was required to be used for data persistence.

Project A produced a microservice running in production without major issues. After resolving a few challenges with projection mechanisms, we end up implementing a faithful domain model representing business processes, domain events holding important domain facts and well determined ubiquitous language spoken by the team and the business experts. When the team was assigned to develop microservice B, the same practices were carried over. But then, I realized that it would not go as smoothly as before.

Useless Domain Events

When I first heard Once you get the hang of using Domain Events, you will be addicted and wonder how you survived without them until now, it sounded a bit pretentious and exaggerated, but that is exactly how I have felt since I was introduced to Domain-Driven Design. In my opinion, the greatest power of DDD comes from domain events.

When using event sourcing, everything is spinning around domain events. Working on project A could have been described as talking, thinking, storming, modeling, projecting, storing, trusting and investigating domain events. On the project B, however, we were saving only the latest state of the aggregate and were conveniently deriving the read models from that as well. So, no storing and no domain events needed for projections. To make things worse, there was nothing needed to be subscribed to the domain events the aggregate is publishing. Of course, we decided not to use the concept of domain events, but as a consequence, we had to change the mindset we were used to and continue with a weaker DDD.

Painful refactoring

By gradually expanding domain knowledge, continuous refactoring and adjusting of the domain model were the common practices during development of the microservice A. With event sourcing one can completely remodel the aggregates without having to change anything in the data persisting mechanism. Aggregate properties are just placeholders being filled with data loaded from stored facts — domain events. But when one is storing only the latest state of aggregates, every change drags adjusting database, altering or creating new tables or migrations of existing data. Even though we had created suitable data access objects mapped from the aggregate, changes on the database or in the mappers were inevitable and often time-consuming.

Less focus on the core domain

It did not take long for the team to feel that our focus has moved from the core domain to infrastructure while working on our project B. Even for the slightest change, we were spending too much time, the precious time we used to spend discussing and remodeling the business domain. Not to mention pull requests with more files to review and more code to cover with tests.

Wrapping up

After going through these pain points, I concluded that event sourcing is a natural fit and a powerful supporting tool that strengthens the domain-driven design. Event sourcing brings easier and faster development, testing and debugging. It helps to focus on what is really important — the facts happening in the system shaped as domain events. It facilitates adjusting and continuous improvement of the core domain model. The life of a developer is much easier with event sourcing. But I need to say, it does not mean this should be the primary factor when deciding should a system be event sourced or not. But that is another story for another time.

Author: Nusreta Sinanović

Let’s build a house

Let’s build a house

While staring out of the window of an airbus A319–100, traveling to Amsterdam for “DDD / Event Sourcing Europe”, I had a sudden rush of inspiration and decided to try and scribble a quick blog post.

Here goes nothing…

When talking about software, we often hear the term “architecture” mentioned a lot. That inevitably reminds us of buildings and construction. While this may be true up to some extent, there are a lot of discrepancies between the approaches we take towards architecting a building (a house for example) and a typical web application software system. Much more so if we employ agile practices while doing so (which I hope we all do).

Software architecture is more about defining and formalizing constraints and gathering quality attributes that we need to satisfy while implementing business requirements.

A good architect tries to defer most of the architectural decisions that don’t affect the most important quality attributes directly. These decisions can then be made downstream by the developers in charge of building the actual solution (the architect himself should be one of them).
This indeed means that software architecture “is not” about making “all” of the decisions upfront, but rather, only the important ones.

Let’s look at a simple example. Let us contrast building a house in the real world versus building a virtual house (a hypothetical software system, might be a simple web application).

Some things of concern, but not limited to might be:

  1. Land (Infrastructure)
  2. Floor plans (Architecture / technical diagrams)
  3. Foundations (Initial project setup)
  4. Floors and the roof, walls, bulkheads, etc… (Software Layers)
  5. Functional rooms + furniture (Actual features that bring value)

So how do these compare?

Land (Infrastructure)

When building a house we need to purchase (allocate) the entire piece of land that our house will sit on top of. Plus, we need to account for some more if we maybe decide to make some extensions to our hose (a garage maybe, a garden …).

This implies that most of the time we need to know the size of the land needed upfront which in turn implies that we are required to make a large upfront cost in order to purchase the land even before we even start with the actual construction.

In contrast, when building a software system the agile way, we try to avoid these upfront commitments. Why? Because we can!

Why is this the case?

Ask yourself, what is software? If you think hard enough, you will come to the conclusion that software, in essence, is just a pile of text, compiled to machine language (instructions) in order to be executed to (hopefully) perform some useful work for us people.

And what is the nature of “text”.
Well, most importantly, text can be changed, expanded, erased, reorganized, burned and thrown away. It’s subject to change.

If you think otherwise, you are living a lie and becoming aware of this asap will do you much, much good.

But even if you choose not to be aware of this, I can tell you one thing for sure: The clients are!

So, going back to our example, how much land (infrastructure) do we allocate for our software system? The answer is, as little as possible, or to put it differently, only as much as we need to.

Staying agile is all about iterations, tight feedback loops and building as little as needed when we need it.

We try to avoid making big upfront commitments and incurring costs that are uncalled for. As a side note, if you’ve ever wondered what cloud computing is all about, now you know (pay only for what you use)…

For the software system in our example (our virtual house), we would allocate/purchase only enough resources we need for our first iteration, feature or sprint. Nothing more, nothing less, because unlike with land, we can always but more.

Floor plans

When building a house in the real world, in 99% of the cases we have the whole plan ready upfront. We hire an architect, decide on almost all features (rooms, layout), materials, etc and again, we make another upfront payment and only now we can start building foundations for our house.

After this point, pretty much every decision is immutable, and any change to the initial plan is hard to make or would surely incur a lot of extra unplanned cost because once concrete hardens it’s really tough to break.
That’s just the nature of it.

We take this for granted because we understand the reasoning behind it and have learned to live it.

But surprisingly this isn’t true for software. Many people still compare building software with building houses, and this cannot be further than the truth.

As we said, software is different. A good architect does not make all of the decisions up front. Why? Because software is easy to change (and will change, it’s just text) so we don’t want to limit our legroom that we will most surely need later.

Most importantly, we don’t want to spend any more money than necessary. Will we like that bedroom on the second floor? Will we even need it?!

Everything else

After we have the foundations are laid down, it’s time to start building our floors.

Floor by floor, layer by layer, weeks and months pass while we wait with our fingers crossed and hope for the best.

Finally, when the house is done, final work can begin and only then can we start throwing in the furniture.

A few months later, we finally get to use our house.

I think you can see the pattern by now.

Architecture deals with horizontal slices. Floor by floor, layer by layer, because it’s easier to do it this way (and it’s cost-effective).

But when building software systems, we deal with vertical slices (or should). What does this mean?

When creating software solutions, we start from the most important thing (the core), build all the vertical slices for it, and then work our way out (horizontally) by adding new features to the core or improving the core itself.

In our virtual house example, this could be anything really, but let’s take a bedroom as an example.

Let’s pretend that we sat with the customer and decided that having a place to sleep in asap would be of the highest priority.

Thus, our main goal should be to deliver a single useful feature for our customer which can be tested out and for which we can get constructive feedback at the earliest moment possible.

I hope you can understand why this can be useful (in contrast to waiting for months for first feedback)

So, some of our basic goals are:

How would we go about doing this? We do the simplest thing. We simply follow the path of the least resistance. In analogy to the real house, building our virtual house might follow steps similar to these:

We are now ready to call in our client and receive some constructive feedback about it. Hell, he could even spend the night.

Once we have our feedback, the client decides what is the next most important thing for him…

This way we ensure that the customer always gets what he wants, and not what he asked for! — Remember what we said about the nature of software! Customers have a nose for this and they will always want something other than what they initially specified.

So in a nutshell:

Final words

If you take away anything, let it be this:

Building software is nothing like building a single house/building.

Building software is much more involved than this. To build, evolve and maintain a software system is more similar to city planning than to building a single house. (Let’s face it, it’s never a single house anyway).

Cities like software, have lives of their own. Cities evolve, grow (or even shrink) in size. New buildings are added (features) all the time, old ones are refurbished or demolished(refactoring).

New water and heating lines are added, removed and replaced on a daily basis. Metro lines are bored under the city.

While doing all of this, you can never shut down a city. You can close down some parts of it but as a whole, it must always remain functional.

This is why building software is neither simple nor easy, but only by embracing change, and failure (this might be a topic for another blog post) we can be prepared, and triumph!

Author: Anes Hasičić

Generic repository pattern using Dapper

Generic repository pattern using Dapper

Implementing Repository pattern in Dapper can be done in many ways. In this blog I will explain what repository pattern is and how to implement it using reflection in C#.

When searching around the web I found various implementations none of which was satisfactory — mostly due to having to manually enter table names/field names, too complex implementations with Unit of work etc.

Similar repository, as presented here, is in use in production CQRS/ES system and works quite well.

Even tho use case depicted here is quite “unique”, I think implementation of this repository can be applied for most of the relational base database systems, with minimum refactoring.

What is Repository pattern?

When talking about Repository pattern it is important to distinguish between DDD implementation of repository and generic repositorypattern.

Generic repository is simple contract defined as an interface on per object basis. That means that one repository object is related to one table in database.

DDD repository pattern works with aggregate root object and persists it one or more tables or if event sourcing is used as series of events in event store. So in this instance, repository is actually related not to one database but to one aggregate root which can map to one or more tables. This is a complex process due to impedance mismatch effect which better handled with ORM’s, but this is not our use case.

Generic repository UML diagram:

  • GenericRepository abstract class implements IGenericRepository interface. All shared functionality is implemented in this class.
  • ISpecificRepository interface should have methods required for specific use case( if any)
  • SpecificRepository class inherits from GenericRepository abstract class and implements any specific methods from ISpecifiRepository.

Unit Of Work and transaction handling

Unit of Work pattern implements single transaction for multiple repository objects, making sure that all INSERT/UPDATE/DELETE statements are executed in order and atomically.

I will not be using Unit Of Work but rather save each repository directly using Update/Insert method. The reason for this is that these repositories are designed toward a specific use case detailed below.

All transaction handling is done manually by wrapping multiple repository command into .NET Transaction object. This gives a lot more flexibility without adding additional complexity.

Repository implementation

Let us define interface first:

public interface IGenericRepository<T>
    {
        Task<IEnumerable<T>> GetAllAsync();
        Task DeleteRowAsync(Guid id);
        Task<T> GetAsync(Guid id);
        Task<int> SaveRangeAsync(IEnumerable<T> list);
        Task UpdateAsync(T t);
        Task InsertAsync(T t);
    }

Bootstrap code for repository class has the responsibility to create SqlConnection object and open the connection to database. After that, Dapper will utilize this connection to execute SQL queries against database.

public abstract class GenericRepository<T> : IGenericRepository<T> where T: class
    {
        private readonly string _tableName;

        protected GenericRepository(string tableName)
        {
            _tableName = tableName;
        }
        /// <summary>
        /// Generate new connection based on connection string
        /// </summary>
        /// <returns></returns>
        private SqlConnection SqlConnection()
        {
            return new SqlConnection(ConfigurationManager.ConnectionStrings["MainDb"].ConnectionString);
        }

        /// <summary>
        /// Open new connection and return it for use
        /// </summary>
        /// <returns></returns>
        private IDbConnection CreateConnection()
        {
            var conn = SqlConnection();
            conn.Open();
            return conn;
        }

        private IEnumerable<PropertyInfo> GetProperties => typeof(T).GetProperties();

Make sure you have connection string named MainDb. I am using MSSQL LocalDb, lite MSSQL version database provided with Visual Studio.

<add name="MainDb"
connectionString="Data Source=(localdb)\mssqllocaldb;Integrated Security=true;Initial Catalog=dapper-examples;"
providerName="System.Data.SqlClient"/>

Implementing most of these methods, except for Insert and Update, are quite straightforward using Dapper.

public async Task<IEnumerable<T>> GetAllAsync()
        {
            using (var connection = CreateConnection())
            {
                return await connection.QueryAsync<T>($"SELECT * FROM {_tableName}");
            }
        }

        public async Task DeleteRowAsync(Guid id)
        {
            using (var connection = CreateConnection())
            {
                await connection.ExecuteAsync($"DELETE FROM {_tableName} WHERE Id=@Id", new { Id = id });
            }
        }

        public async Task<T> GetAsync(Guid id)
        {
            using (var connection = CreateConnection())
            {
                var result = await connection.QuerySingleOrDefaultAsync<T>($"SELECT * FROM {_tableName} WHERE Id=@Id", new { Id = id });
                if (result == null)
                    throw new KeyNotFoundException($"{_tableName} with id [{id}] could not be found.");

                return result;
            }
        }

        public async Task<int> SaveRangeAsync(IEnumerable<T> list)
        {
            var inserted = 0;
            var query = GenerateInsertQuery();
            using (var connection = CreateConnection())
            {
                inserted += await connection.ExecuteAsync(query, list);
            }

            return inserted;
        }

 

For SaveRangeAsync list of items is provided which are saved to database and returns number of items saved. This can be made atomic by wrapping foreach in Transaction object.

Implementing Insert and Update queries

Implementing insert and update requires a bit more work. In general idea is to use reflection and extract field names from model class and then generate insert/update query based on field names. Field names will be used as parameter names for Dapper therefore it is important to make sure that DAO class field names are the same as column names in actual table.

In both cases the idea is the same: take object, provided as input parameter, and generate SQL query string with parameters. The only change is that different query is generated, INSERT or UPDATE.

Both methods use reflection to extract field names from model object. This class can be made static, since its not using any of instance variables and for performance purposes.

private static List<string> GenerateListOfProperties(IEnumerable<PropertyInfo> listOfProperties)
        {
            return (from prop in listOfProperties let attributes = prop.GetCustomAttributes(typeof(DescriptionAttribute), false)
                where attributes.Length <= 0 || (attributes[0] as DescriptionAttribute)?.Description != "ignore" select prop.Name).ToList();
        }

 

What this does is extracts a list of attribute names into List<string> using reflection. It will not extract fields marked with ignore description attribute.

Once we have this list we can iterate it and generate actual query:

public async Task InsertAsync(T t)
        {
            var insertQuery = GenerateInsertQuery();

            using (var connection = CreateConnection())
            {
                await connection.ExecuteAsync(insertQuery, t);
            }
        }

private string GenerateInsertQuery()
        {
            var insertQuery = new StringBuilder($"INSERT INTO {_tableName} ");
            
            insertQuery.Append("(");

            var properties = GenerateListOfProperties(GetProperties);
            properties.ForEach(prop => { insertQuery.Append($"[{prop}],"); });

            insertQuery
                .Remove(insertQuery.Length - 1, 1)
                .Append(") VALUES (");

            properties.ForEach(prop => { insertQuery.Append($"@{prop},"); });

            insertQuery
                .Remove(insertQuery.Length - 1, 1)
                .Append(")");

            return insertQuery.ToString();
        }

 

Update method has some small differences:

public async Task UpdateAsync(T t)
        {
            var updateQuery = GenerateUpdateQuery();

            using (var connection = CreateConnection())
            {
                await connection.ExecuteAsync(updateQuery, t);
            }
        }

private string GenerateUpdateQuery()
        {
            var updateQuery = new StringBuilder($"UPDATE {_tableName} SET ");
            var properties = GenerateListOfProperties(GetProperties);

            properties.ForEach(property =>
            {
                if (!property.Equals("Id"))
                {
                    updateQuery.Append($"{property}=@{property},");
                }
            });

            updateQuery.Remove(updateQuery.Length - 1, 1); //remove last comma
            updateQuery.Append(" WHERE Id=@Id");

            return updateQuery.ToString();
        }

 

Additional thing we need to take care of here is what happens if record for updating is not found. There are couple of solutions for this, some include throwing an exception others returning empty object or somehow notifying calling code that update was not done.

In this case we are relying on Dappers executeAsync method which return int which is a number of affected rows.

Example of generic repository usage:

public static async Task Main(string[] args)
        {
            var userRepository = new UserRepository("Users");
            Console.WriteLine(" Save into table users ");
            var guid = Guid.NewGuid();
            await userRepository.InsertAsync(new User()
            {
                FirstName = "Test2",
                Id = guid,
                LastName = "LastName2"
            });


            await userRepository.UpdateAsync(new User()
            {
                FirstName = "Test3",
                Id = guid,
                LastName = "LastName3"
            });


            List<User> users = new List<User>();

            for (var i = 0; i < 100000; i++)
            {
                var id = Guid.NewGuid();
                users.Add(new User
                {
                    Id = id,
                    LastName = "aaa",
                    FirstName = "bbb"
                });
            }

            var stopwatch = new Stopwatch();
            stopwatch.Start();
           
           
            Console.WriteLine($"Inserted {await userRepository.SaveRangeAsync(users)}");

            stopwatch.Stop();
            var elapsed_time = stopwatch.ElapsedMilliseconds;
            Console.WriteLine($"Elapsed time {elapsed_time} ms");
            Console.ReadLine();
        }

 

Use case in CQRS/ES Architecture

CQRS stands for Command Query Responsibility Segregation, and is an architectural pattern, which separates read model from write model. The idea is to have two models which can scale independently and are optimized for either read or write.

Event Sourcing(ES) is a pattern which states that state of the object is persisted into database as list of events and can be later reconstructed to the latest state by applying these events in order.

I will not go into explaining what these two patterns are and how to implement them, but rather focus on specific use case I’ve dealt with in one of my projects: How to use relational database (MSSQL) for read model and event store, utilizing data mapper Dapper and Generic repository pattern. I will also touch, albeit briefly, event sourcing using the same generic repository.

Example architecture:

Before going any further let us consider why using data mapper would be more beneficial than using ORM for this particular case:

Impedance Mismatch in Event Sourcing

An object-relational impedance mismatch refers to a range of problems representing data from relational databases in object-oriented programming languages.

Impedance mismatch has a large cost associated. Reason for this is that developer has to know both relational model as well as object oriented model. Object Relational Mappers(ORMs) are used to mitigate this issue but not eliminate it. They also tend to introduce new problems: like virtual properties requirement by EF, private properties mapping issue, polluting domain model etc.

When using using Events as storage mechanism in Event Store, there is no impedance mismatchThe reason for this is that events are domain concept and are persisted directly in Event store without any need for object relational mapping. Therefore, need for using ORM is minimal, and using Dapper/Generic repository becomes more practical.

Database model considerations

In this use case MSSQL will be used for both write and read sides, which adds to re-usability for dapper repository since it can be used on both read and write sides.

Primary key consideration

In this example I used Guid (.NET Guid and uniqueidentifier MSSQL datatype) as primary key identifier. It could have been something else like long or int, or string.

In any case this would require some additional work on Dapper repository. First, interface would need to change to accept additional primary key type. Then, depending on the type, there might be some additional work to modify queries generated.

Having more than one column as primary key would also imply some additional work and in this case using dapper/generic repository pattern would probably counter productive. We should opt for using full blown ORM in this case!

Bulk inserts with Dapper

Dapper is NOT suitable for bulk inserts, ie. performing large number of INSERT statements. The reason is that ExecuteAsync method, internally will use foreach loop to generate insert statements and execute them. For large number of records this is not optimal, and I would recommend using either SQL Bulk copy functionality or Dapper extension which allows bulk copy(its commercial extension) or simply bypassing dapper and working with database directly.

Transactions handling

Use case for applying transaction is when saving into more than one table atomically. Saving into event store can be this example: save into AggregateRoot table and Events table as one transaction.

Transactions should be controlled manually either on Command level (in CQRS Command implementation) or inside repository.

This example with two tables is inspired by Greg Young’s design which can be found here: https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf

using (var transaction = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
            {
                // Fetch Aggregate root if exists, create new otherwise
                var aggRoot = await _aggregateRootRepository.FindById(aggregateId.Identity);
                if (aggRoot == null)
                {
                    aggRoot = new AggregateRootDao
                    {
                        Id = Guid.NewGuid(),
                        CreatedAt = DateTime.Now,
                        AggregateId = aggregateId.Identity,
                        Version = 0,
                        Name = "AggregateName"
                    };

                    await _aggregateRootRepository.InsertAsync(aggRoot);
                }
                else
                {
                    if (originatingVersion != aggRoot.Version)
                        throw new EventStoreConcurrencyException($"Failed concurrency check for aggregate. Incoming version: {originatingVersion} Current version: {aggRoot.Version}");
                }

                // Optimistic concurrency check
                var domainEvents = events as IDomainEvent[] ?? events.ToArray();

                foreach (var e in domainEvents)
                {
                    // Increment Aggregate version for each event
                    aggRoot.Version++;
                    e.Version = aggRoot.Version;

                    // Store each event with incremented Aggregate version 
                    var eventForRoot = new EventDao()
                    {
                        CreatedAt = e.CreatedAt,
                        AggregateRootFk = aggRoot.Id,
                        Data = JsonConvert.SerializeObject(e, Formatting.Indented, _jsonSerializerSettings),
                        Version = aggRoot.Version,
                        Name = e.GetType().Name,
                        Id = e.Id,
                    };

                    await _eventsRepository.InsertAsync(eventForRoot);
                }

                // Update the Aggregate
                await _aggregateRootRepository.UpdateAggregateVersion(aggRoot.Id, aggRoot.Version);


                transaction.Complete();
            }

 

If aggregate root is not found, it is created and inserted in AggregateRoot table. After that each event is converted to domain event and saved into Events table. All this is wrapped in transaction and will either fail or succeed as an atomic operation. Note that transaction has TransactionScopeAsyncFlowOption.Enabled option, which allows transaction to invoke async code inside its body.

Conclusion

Implementation here can be further optimized for use in CQRS/ES systems however that is outside of the scope of this post. This implementation gives enough flexibility to extend required specific repository with new functionality easily, just by adding custom SQL queries to the repository.

Author: Damir Bolić