monitoring and/or logging specialist (InfluxDB, Prometheus, ELK, ClickHouse)

We have to renew our monitoring- and logging-infrastructure. We played a little with Zabbix (and graylog) and were not really satisfied (but maybe we don't try hard enough), so we now choose a different way to speed up our decision process: Do you have a demo / mockup / real live installation of a nice (!) looking and integrated monitoring- and logging-system you are able to show us? Yes, we just want to see and talk about it! Or maybe you can answer one or two of the questions below! In case we like what we see, we will hire you to help us to implement it (or in case you don't offer such a service, we will search for someone else who does). Yes this job is only for consulting and only takes one upto a few hours ... Later we talk about much more... What we are looking for: We have ~30 machines (Web CMS with author system, load-balancer, database, live-webserver) offering ~1000 websites (but 'only' ~100 hits per second). We are looking for a system to - monitor our machines (VM available? CPU ok? Disk full? Webserver reachable and answers fast enough? Databases OK? ...) - analyse logs (Webserver access + error; system logs; ~20G per day) - tracing - check for problems e.g. by querying the logs by hand - see trending usage of resources We need a solution with a intuitive and nice webinterface / dashboard, nice graphics for the management, flexible alerting and everything should be easily maintainable. We started to play with Zabbix and graylog, but we hope to see a better solution. Or maybe you have configured it better!? Let me see! What about InfluxDB (TICK) using Grafana? Is it usable in the real world? Is InfluxDB able to interpret our webserver logs, too? (See [login to view URL]) That would be fun! What about Prometheus and TimescaleDB? Hm, why don't normalize our Logs to JSON push them into Postgres instead of ELK / graylog? Stupid idea?! Is ClickHouse a usable replacement for ELK / graylog? Or even for monitoring? That would be very promising! Will loki ([login to view URL]) be able replace ELK / graylog? There might be more ideas! Tell me! As you see: We have many ideas and questions. Maybe you can answer one or more or be able to show us something.... OK, and what special needs do we have? The data is not allowed to be normalized (e.g. like in round-robin databases). The absence of a value does not mean that the value is '0'! Integration of Netdata must be possible to see almost live data. We have a few Java-programs: Is your propageted solution able to show something like jconsole? Sometimes there is no communication between monitoring and client. Somehow the values must be cached somewhere. And when the communication is up again the values are imported at the correct times and not at the time of arriving! The perfect system is able to push and pull the data... There must be an API to configure the system/Adding new hosts/rules/alterting the system(s) for automating. There must be an API to switch off sending e-mail/sms-alerts for a time period for special checks. (If I know that a system is going down (e.g. for maintentence) I do not need e-mails for this! But in the webinterface I see it, because it is down!) It must be possible to aggregate (e.g. summarize access per day) old data. It must be possible to delete old data. So show me your solution or answer me some of the questions....