How Vault is tested
A historian has one job: when someone asks what happened, the record has to be right, and where it is missing it has to say so. This is how Vault is built and tested to do that.
Where a value can be lost, and where it cannot
Every historian has a window between receiving a value and making it safe. Vault states its window and lets you set it, instead of claiming it does not exist.
The test record
Each test was designed to break something. Failures found along the way were fixed and the test re-run; the figures below are from the final runs.
| Ref | Test | Result | Verdict |
|---|---|---|---|
| T-01 | Hard kill under load400 crash-test runs against the real program, including 339 hard kills while collecting, starting and shutting down and 56 clean stops, each followed by a full check of every stored value | 0 acknowledged values lostWorst loss at a kill: 39 ms at a 20 ms log sync on an idle rig; 71 ms with another process loading the CPU. Linux test rig; process kills, not power cuts. | PASS |
| T-02 | Worst-nightmare fault injectionA test server sending known values through a proxy that drops, stalls and corrupts traffic; full disks, bit rot, 140 kills, hostile queries | 0 wrong, 0 duplicated, 0 gaps unmarked2,942,223 values compared with what was sent. 26 defects found and fixed along the way. | PASS |
| T-03 | Windows soak2,500 signals from a simulated battery site over OPC UA (security None), one Windows service | 6.9 days without a restart632 M samples collected in that run (1,058 a second on average). All 675 M samples in the eight day files decoded and checked against their hashes: 0 errors. Seven 30-second stalls caused by an OPC UA library fault were stored as marked gaps (99.96 % captured). The fix is verified in a proxy test; a soak re-run on the fixed build is still to do. | PASS |
| T-04 | Load3,000 tags at one second, no deadband, release build | 0.5–0.65 % of one CPU core2.8 GHz Xeon core. About 0.6 GB a day of raw one-second data. | PASS |
| T-05 | Year-long queryOne tag, a year of one-second data (31.5 million samples), read at one-day intervals | 0.09 sChunk summaries answer most of the query without decoding. | PASS |
| T-06 | Real PLCsA real Siemens S7-1500 and a Phoenix Contact PLCnext, over OPC UA with Sign & Encrypt, on an evaluation tester's bench | In progressResults will be published here, dated, as they come in. | OPEN |
| T-07 | Power cut on WindowsThe plug pulled on a Windows panel PC under load, five times, each followed by a full check | Not yet runProves the disk honours the log sync, which a process kill cannot. | OPEN |
| T-08 | 30-day soak3,000 signals at one second on the target panel PC, tracking memory, handles and disk | Not yet run | OPEN |
What each test does
T-01 Hard kill under load
A harness starts the real executable, feeds it known values, kills it at random moments (while collecting, while starting and replaying, while shutting down), restarts it and checks every stored value against what was generated: nothing acknowledged missing, nothing duplicated, every crash detected.
T-02 Worst-nightmare fault injection
A test OPC UA server sends values whose correct stored form is known in advance, through a proxy that drops, delays and corrupts the traffic. On top of that: full and vanishing disks, damaged files, configuration storms, hostile web requests and a query flood. A checker compares every stored value with the truth.
T-03 Windows soak
A week of continuous collection from a simulated battery site on a Windows mini PC, then every day file decoded sample by sample and checked against its hash and chunk summaries. The soak found a real fault in the OPC UA library's reconnect handling; in the proxy test Vault now resumes 0.4–0.7 s after a cut and marks anything the server could not resend.
What the design does about failure
- Torn writes. The log tells a half-written tail from corruption, keeps everything before it and marks the rest.
- Clocks. Values stamped in the future raise a fault. If a clock was set ahead and then put right, one button in Setup sets the future-dated records aside, keeping them, and collection carries on.
- Full disks. Old sealed files are removed first, never the last seven days. At 95 % collection stops with the gap marked and the busiest signals named.
- Silent sources. A source that stays connected but sends nothing raises a fault, so a quiet PLC is not mistaken for a quiet plant.
- Lost notifications. After a reconnect Vault asks the server to resend what it missed, and marks what it could not get.
- Damaged files. A file that fails its hash is quarantined and reported; the rest of the record stays readable.
Put Vault through your own tests
Early access sites can run Vault alongside what they have today, pull the plug, and compare. We would like to hear what you find.