A proof of concept is a pilot that runs a product on your own infrastructure so you can decide, on evidence, whether to buy it. It works only if you turn up. We recently finished a three-month monitoring PoC for an East African financial institution. Every success criterion in the signed scope of work was met, however the pilot proved almost nothing. Because the one input that mattered never arrived.
If your organisation is about to accept a free pilot from anyone, these are the failure modes to check yourself against before you start.
The setup
An on-premises monitoring pilot, up to 500 devices, three months of support included — something a lot of companies would be happy to have for free.
The institution provided a server and a VPN path. We provided the software, the engineering and the support hours. Cost to the customer: nothing. The scope of work carried the pilot cost on our side, recoverable only if a production agreement followed.
Their only real obligation was attention.
Senior management framed the project as urgent and important. It was, at the executive level, genuinely wanted. That turned out to be a poor predictor of how the project went further on.
This is the single most predictive signal in any pilot, and it’s visible before day one. Executive sponsorship gets a PoC approved. But it’s the operator’s time what makes it mean something. When the sponsor is enthusiastic and the operator’s calendar is untouched, the pilot is already running on empty.
So, the initial test is simple: can you name the engineer who will open a new system daily for the next twelve weeks, and have you taken something off their plate?
”We’re too busy for the questionnaire”
Before deployment we send a requirements questionnaire. It asks what you monitor today, what breaks, who needs to know when it breaks, and what “normal” looks like on your critical services.
It was declined, though, as the team was “too busy” to answer the questions — in written or during an online meeting.
That questionnaire isn’t vendor paperwork, for a PoC team it’s the specification. Declining it doesn’t save potential client a morning — it transfers the design decisions to people who have never seen your environment and don’t know your goals.
Here’s what that looked like in practice, from our team’s perspective.
”I had only four ports. I did everything with the information I had.”
For application monitoring, our engineer received four port numbers and no description of what ran behind them. So she monitored what can be inferred from outside: whether the ports answer, response time, certificate expiry, the Windows processes serving the pages, thread and handle counts. She set a response-time threshold at two seconds because two seconds is a reasonable guess.
What she couldn’t monitor was whether the business function behind those ports actually worked. Not because the platform can’t (NetXMS can watch a login succeed, parse a returned page, read application logs for exceptions, look inside a JVM), but because nobody answered what the services even did.
Nobody owned frictions and the outcome
The e-mail notification channel was configured with the credentials we were given, and was never tested. No recipient address was ever supplied, and no decision was made about who should receive which alert.
One of the signed success criteria was that the institution always receives alarms according to configured thresholds. Alarms were raised correctly in the system. Whether anyone would have been woken up by one remains unknown, three months later.
That’s not a product gap, it’s an organisational readiness gap, and the pilot is what made it visible. Which is arguably useful, but it was an expensive way to discover it.
Remote access dropped repeatedly during configuration and took time to restore. The customer raised this themselves during the review, fairly and without prompting.
Taken alone it’s a logistics problem. Taken alongside everything else it’s the same shortage of ownership. When nobody is empowered to unblock the pilot within a day, the pilot quietly stops being anybody’s priority.
What could be delivered with no time investment
- Automatic discovery covering 26 subnets and 75 nodes.
- Per-device dashboards for twelve Cisco switches and routers — CPU, memory pool, fan and temperature sensors, and whether the running configuration differs from the saved one.
- Oracle and PostgreSQL monitoring with instance state, connections, tablespace usage, deadlocks and cache ratios.
- Four application services with thresholds.
- A business service tracking availability at roughly 99.9% across the month.
- A live topology map with per-link traffic.
Hardware, picked by the automatic discovery, and where standard templates could be applied, was added too. And this is a good point — once one device of a class is configured, the rest of that class inherits everything automatically. So, a hundred SNMP switches cost roughly what one costs in time and effort.
Unfortunately, there is no template for what your business considers an outage.
During the review session, the gap became visible. We ran a walkthrough of the deployed system with the team. And all the questions were the capability questions: what can be monitored on an application, can the traffic graph show a longer time range, can the system detect IP address conflicts.
Not one question was “here’s what breaks at two in the morning, can your solution catch it earlier?” We asked for pain points and begged for more input. But after the “Sure, thank you, and we will do that,” nothing followed. The pilot ran out its term in silence.
Why monitoring punishes low involvement specifically
A monitoring platform has no opinion about what matters to you, and that’s by design. Discovery is automatic, device classification is automatic, standard templates apply themselves — all of that happens whether you participate or not, which is exactly why an unengaged pilot still produces impressive-looking dashboards.
Alerting is where it stops. During the review we confirmed that duplicate IP and MAC address detection are built in and raise alarms. But no notification goes out by default, deliberately, because who gets woken up is a customer decision that no vendor can guess.
The same applies to thresholds: ours stayed at sensible defaults for the whole pilot because nobody can guess what normal looks like on the client’s estate.
So the pilot proved the engine works, but it never proved the engine was tuned to their reality, and it couldn’t.
What we learned the hard way
Five things to have in place before you accept a pilot from any vendor.
- Name the operator who’ll use the system daily, and clear time in their calendar for it
- Complete the requirements questionnaire before kick-off
- Bring three real incidents you want detected earlier, not a list of features you want demonstrated
- Supply working notification recipients and an escalation path
- Book a mid-pilot review with a written list of changes
In the end of the day…
A well-built demo can show you what a product does. A pilot goes much further and shows you something a demo can’t: whether it fits your environment, your thresholds, your definition of a bad incident. That second thing requires you.
Skip your side of it and you haven’t run a proof of concept, but a very slow demo on your own hardware.
If an organisation can’t find time to say what it needs during a free three-month pilot, it won’t find that time during a paid rollout. The engagement didn’t fail to prove the product, it proved, though, something about the buyer.