Grafana - Data visualization
I don't know if you ever heard of grafana, few people even in the computer industry has. Grafana is a great tool to visualise data. I have been using grafana for over a year now to have statistics on the usage of eXpansion. A tool I made for the Maniaplanet game.
More know to many people is Kibana, and indeed I also started there. I first used Kibana to make visualise statistics on the usage of eXpansion. But I realized very quickly that Kibana was not designed for that. Kibana is designed for logs monitoring; I could get nice graps showing me stats about the different versions of the tool or the different php versions used....
But I couldn't get the number of server using my tool at a certain time. Basically each server pushes one entry every 5 minutes. In Kibana if I zoomed in to much the aggregation was not sufficient and I saw empty spaces, if I zoomed out to much it aggragated to much and suddenly had hundred of users on my tool.
The second issue I had with Kibana is user permissions, there is (or at least wasn't) any simple way of giving access to the kibana to certain persons only.
At that time I discovered Grafana, I was lucky, the elasticsearch support had just been added which meant my current dataset was usable as it was. Grafana was very easy to install fallowing their guide, and I could create user groups & users to give persons that worked with me on the project proper accesses.
There isn't much to say, you can't use Grafana to search throught your logs like you would use Kibana, but when it comes to visualisation it's a complete different matter. Both Kibana & Elasticsearch are 2 completeley different tools for different usages. I still use Kibana(quite an old version) for my logs but for the visualisation I use Grafana.
I alread have a munin running on my server, allowing me to see different stats of my server. Lately I also installed metricbeat on my server form the elastic stack. This pushes system information into the the elasticsearch; after spending a bit of time then in Grafana you can visualise your system metrics in Grafana as well.
You can't of course really compare munin to grafana & elastic. The first is incredibly stable and uses early no resources. The second on the other hand is using alot of ram to store all the elasticdata in. But the second also offers more flexibility when it comes to visualisation.
To end this topic I would like to talk about google analytics & to be more precise google tags. It has been a tendancy in commerce website to use google tags more and more in order to collect metrics on websites; such as amount of the carts, type of products ...
Google Tags
The selling point made is that the tags can be configured very easily by a non technic person and therefore you can make whatever you wish without calling an expensive developper.
The issue is that this is only partially true, google tags can't do anything if the data is not there, and it's a technical person that will push the data by modfying the source code. Google tags is soo complicated that for some changes you will need a google analytics expert and sometimes a developper ontop of that. So there is no really much time gain in my point of view
At this point there is multiple things that bothers me :
- The front contains code for making statistics. I see no reasons why the front should be in charge of this.
- Changes to the design can affect the data pushed, which is simply not logical.
- The user can actually see the data sent, so you can't have everything.
- Google tag manager isn't transparent and when a lot of data is pushed will actually slow down your website.
The issue is not with google tag manager, that actually works very well; and can be used efficiently for statistics; no it's about what doesen't go into google analytics and what goes elsewhere.
Making the back, call an elasticsearch(or logstash) using UDP call to push analytics is more efficient. You can then use Grafana to visualise the data. You can actually even attach grafana to your mysql server replica, This add's more flexibilty and allows a more logical coding. We could even push profiling data this way to analyse issues on certain pages.
Just to end up on this, website behind varnish isn't an issue, either you continue to use google analytics for you page view stats, or you can also read in varnishlogs to have pages that were opened. Even if for page view stats it's inded easier to use google analytics. But for order amounts and such, those pages aren't cached in varnish any way so the back will be able to send the data to the elastic.
Conclusion
Grafana/Kibana is in my point of view a must have for any medium size/big project. Depending on the project one of the other might be sufficient. The amount of logs you keep depends also on the resources you can spare.
Having profiling that can be activated quickly with nearly no performance cost can make the difference on a website with growing trafic and that needs to be stable. Of course you don't need such tools on your blog. But even for a company website it can have a benefits. There are plenty of exemples of companies having their website crash when they appear in the news or so. Proper tools, proper profiling means you can learn from this so that next time you can be ready.