Saturday, January 30, 2016

Domain Driven Design: what is it really about?

According to the "Implementing Domain-Driven Design" book (by Vaughn Vernon and foreword by Eric Evans - one of the precursors of DDD), the core of Domain Driven Design philosophy is not about technical stuff.

What do you mean??? That was the first thought that came to my mind when I read it. Let's dig a little bit more in this subject in order to figure out what the above information really means.

So... What DDD is about?

DDD is about understanding how your domain really works. Identifying the core, support and generic Subdomains will make it possible to build well designed models. A Domain, in a broad sense, is the business where you or your company are in. The emphasis of DDD is to identify Subdomains which together compose the whole Domain. Once you did it, you can develop your models inside of Bounded Contexts

The benefits

Let's think about a business Domain of an e-commerce website (a pretty regular one). Without DDD approach, we might start thinking in a single Domain Model to support all the business requirements, which might include Product Catalog, Orders, Invoicing, Shipping, Inventory, etc.

If you have some hands on experience on modeling and programming a Domain Model for a system as described above, you know how can be difficult and extremely complex to define and keep a single class "Product" that support all those business requirements! Here is where the DDD philosophy shines: you don't need to go this way! Getting in the DDD philosophy, we should identify Subdomains inside the core Domain.

This way, Subdomains could be arranged so that each one of the business requirements mentioned above represents a Subdomain. For each identified Subdomain, you should set a Bounded Context. A Bounded Context, basically is a context where your classes, entities, etc. have a unique meaning and are in conform with a Ubiquitous Language.

Again, being very simplistic, a Ubiquitous Language means that there must not be divergence about any concept inside a Bounded Context. This way, it is possible and completely reasonable to have, for instance, different classes (with different meanings) using the same name but in different Bounded Contexts.

A class representing the abstraction of "Product" can exist inside the "Product Catalog" Bounded Context, meaning an item available for searching and selling and having its own pertinent attributes and methods. At the same time, another class "Product" can exist inside the "Shipping" Bounded Context, meaning what must be delivered and having its own specific attributes and methods too.

When you model thinking in a small and very specialized part of the problem, it is much easier to get a simpler, reliable and robust model!

The trade off

Of course, "there's no free lunch" and the trade off law is always present. Having "N" Subdomains, Bounded Contexts and consequently different models, adds complexity to the system as whole. In order to have a functional system able to support a real e-commerce operation, it would be necessary integrate your models and make all these parts working together.

For this, DDD offers a set of patterns and ways to treat and classify your Bounded Context relationships,  like "Partnership", "Shared Kernel", "Customer-Supplier", "Conformist", "Anti-corruption Layer" and others. All of these are concepts regarding how you see your problem and how you organize it. Therefore, the fact is: it is going to demand more attention, time, complexity and work regarding integration between Bounded Contexts.

DDD Concepts Misinterpretation

Unfortunately, there is still a lot of misinterpretation of these concepts. For example, I've seen people publishing tutorials on the internet referring to a layer used to decouple the application front-end layer from the application business layer as the "Shared Kernel". Well... according to the DDD concepts, a Bounded Context might even be an independent system. "Shared Kernel" is a technique to share Model Concepts (see Ubiquitous Language above) between Bounded Context, not between layers.

In the "Product" class example above, if it had the exact same meaning (same meaning, attributes, methods, etc.) for more than one Bounded Context, it could be treated as a "Shared Kernel". Domain Drive Design concepts and philosophy might be a little bit hard to understand at first. That has lead people to misinterpret some concepts and consequently causes mess.

Conclusion

I hope I have been able to convey and explain the very basic and real meaning of "DDD Core philosophy". As explained, it is much more about how to approach your problem instead of technical stuff. I strongly recommend the reading of the book "Implementing Domain Driven Design", by Vaughn Vernon, for those who want to master the Domain Driven Design concepts.



Wednesday, January 13, 2016

Parsing IIS logs with Logstash, sending data to ElasticSearch and analyzing in Kibana

Nowadays, for most modern web applications, it is very important to have information about requests coming to backend servers in order to know if your application is performing as expected. To be able to rely on historical data of your application's API (like the number of requests grouped by resource type) can be very useful.  When you run on a cloud environment like AWS, you do not need to worry about that because they can provide all such information. However, if you are running on your own infrastructure, well.. you should find out a solution by yourself. This is what has led me to the subject of this post.

Bellow, we can see a general deployment view in my case:
Image 1: general deployment view

The main idea is to get Logstash installed in each middle tier server parsing IIS logs and sending this parsed data to Elastic Search. Further, this data can be queried using Kibana in order to produce graphs and useful information about what happens with your application.

Now, we are going to see how I have achieved this. The purpose here is to share what I needed to do without going deep in details, therefore giving you the necessary background to accomplish this by yourself in your particular scenario.

IIS Logs: Installing  IIS Advanced Logging and configuring it

I have chosen to use Advanced Logging due to its extra fields and configuration capabilities. It can be installed using the IIS "Web Platform Installer" option (found in the IIS Services Manager panel interface, server selection) or it can be downloaded at:


Once you install it, don't forget to disable the default IIS logging. You don't want IIS generating two different log files. It is possible to do it clicking on "Logging" => "Disable" (in the right of IIS panel). This same way, you should enable Advanced Logging by clicking on "Advanced Logging" => "Enable Advanced Logging". After that, it is possible to tweak some configuration in order to keep the log file format as you want:

Image 2: IIS Advanced Logging configuration

Most settings are ok as default. The main concern should be about the order of fields in the log file (which you can set using the options presented in the image above) and which fields should be part of the log (clicking on "Select Fields..." button will bring up the following window):

Image 3: IIS Advanced Loggin configuration - Select Logging Fields

It can be "a pain in the ass" to configure it for more than one server. Not surprisingly, it is possible to obtain what you have done in one server and replicate to others. IIS 7.5 on Windows server 2008 stores these configurations in the applicationHost.config, which can be found in the path "c:\windows\system32\inetsrv\config". Inside the "<system.webServer>" configuration section of this file, you can find the "<advancedLogging>" element. Copy and replace it in another server's file.  I have provided my configuration section in this .txt file. This way, if you want, you can use my Logstash configuration file too (further in this post) and save some time.

Logstash: Installing and configuring

Logstash will be in charge of getting data from IIS log files, filtering and parsing and sending it to Elastic Search. Logstash can be downloaded from the Elastic company website. Details about how to install Logstash 1.5 version (this is the version used in my solution) can be found here. After getting Logstash installed, you should create and configure your Logstash .conf file. It can be named as you want and should be passed to Logstash starting comand with option "-f" (see more below). Basically, a Logstash configuration file has three sections:

input section

You can use this section to set the origin of Logstash incoming data. A typical configuration for getting data from IIS log files is like this:

input {
  file {
    type => "IIS Advanced Log"
    path => "C:/inetpub/logs/AdvancedLogs/*.log"
  }
}

filter section

Basically, it is used for processing and filtering data.

output section

You can use this section to tell Logstash what it should do to the processed data, like sending it to Elastic Search Server, for instance.

There is a lot of information on Elastic website about how to configure and take advantage of all the features that each of these sections can provide. As I said before, I have shared my .conf file. If you use it in conjunction with the given configuration for IIS Advanced Logging (as exposed above), you should be able to get this working just like me.

In order to keep Logstash working as windows service, I used Nssm - the Non-Sucking Service Manager. It has worked properly.  Bellow, a configuration example of how a script can be built to install Logstash as a windows service:

@echo off
set nssm_path=c:\nssm\nssm-2.24\win64
set logstash_path=c:\Elastic\logstash-1.5.2
echo.
echo Instaling logstach as service...
echo.
echo Expected logstach path: %logstash_path%
echo Expected nssm path: %nssm_path%
echo.
echo.
cd %nssm_path%
nssm install logstash %logstash_path%\bin\logstash.bat
nssm set logstash AppParameters -f %logstash_path%\bin\[your logstach config file] -l %logstash_path%\log\logs.log
nssm set logstash AppDirectory %logstash_path%
nssm set logstash AppEnvironmentExtra "JAVA_HOME=%JAVA_HOME%"
nssm set logstash AppStdout %logstash_path%\nssm\stdout.log
nssm set logstash AppStderr %logstash_path%\nssm\stderr.log
REM Replace stdout and stderr files
nssm set logstash AppStdoutCreationDisposition 2
nssm set logstash AppStderrCreationDisposition 2
REM Disable WM_CLOSE, WM_QUIT in the Shutdown options. Without it, NSSM can't stop Logstash properly
nssm set logstash AppStopMethodSkip 6
REM Let's start Logstash. I assume a correct configuration is already in place
REM net start logstash

After that, a new "logstash" windows service name will be available in your server. Type "net start logstash" in command prompt window running under admin privilege and the service should be started:

Image 4: starting logstash service after setting it using NSSM.

Kibana: the final step to get done!

Installation of Kibana is easy: just follow the available instructions on Elastic website. Once it is installed, it is reasonably easy to get your reports. As I said before, the target here is not to teach you how to use Kibana or even Logstash. You can take a look on tutorials and documentation on Elastic website (Kibana User Guide). Bellow, just an example of what Kibana can do for you:


Image 5: Data visualization on Kibana


Conclusion

Using Logstash, Kibana and NSSM, it is possible to have a service analyzing your IIS log data, processing it and building some cool graphs :)