A supplier sends you a report with a large green mark. At last, a complicated subject appears to have a simple answer. You want to approve the purchase and get on with work.
The good result may be valuable. The question is what it is a good result about. A score can summarise selected checks, but it cannot create answers to questions those checks never asked.
How separate results become one colour
A checking tool is a program: instructions telling a computer what to do. It may request public pages, examine selected settings and record the results. The program used by a person to visit pages is a browser; the computer system replying to page requests is commonly called a server. A public-page tool can learn something from these exchanges without entering the working service customers use after sign-in. 11
A score is an additional step. Somebody defines rules for turning the separate results into a number, grade or colour. One rule might give a check more importance than another; another might decide how to handle an unfinished check. The summary is understandable only in relation to those choices. These are examples of scoring possibilities, not claims about the calculation in any particular product.
Consider a made-up report with ten completed public-page checks, all favourable. Another report also checks the signed-in service and openly says an important permissions test was not completed. The first report might look tidier. It has not answered the extra question simply by displaying no gap. We must compare what was examined before comparing its colour with the second report.
A school mark works in a similar way. A good mark for spelling remains good, but it cannot tell you how the pupil did in mathematics. The useful conclusion is not that all marks are worthless. It is that a mark belongs to the work assessed.
Good security reporting recommends explaining the systems examined, the practical limitations and the meaning of the results. It also warns that an assessment is not a guarantee of finding every possible problem. These are reasons to read the explanation beneath a score, not reasons to reject favourable evidence. 8
A private journey does not decide who receives the document
Here is one protection that is useful but easy to overinterpret. When your browser asks an invoice service for a document, the request and reply travel through a connection. Those messages can be protected against being read or changed by somebody along the journey.
To provide that protection, the programs use agreed methods for communication. Encryption transforms the message into a form that someone observing the journey cannot simply read, while the programs at both ends can recover the information. The connection also uses checks intended to detect tampering. A widely used mechanism providing these communication protections is Transport Layer Security, shortened to TLS. It concerns the protected communication between those programs. 12
You do not need to understand the mathematics to see where the job ends. The service must be able to work with the request and your browser must be able to display the reply. Protecting the travelling message does not decide whether the service chose the correct document to send. Nor does it explain the separate protection of copies stored later.
Suppose an employee signs into a shared invoice service. Signing in identifies the account being used. The employee asks for an invoice. Before returning it, the service should check whether the employee may read that particular record. If the service incorrectly returns another customer’s invoice, it can send that wrong document through a perfectly protected connection. The journey may be private while the decision about the recipient is wrong. 3
A favourable connection check and a document-permissions test therefore answer different questions. One examines the communication; the other examines what the service allows this user to do with this record. A green result for the former cannot supply the missing result for the latter.
Recovery is different again. Imagine the right person receives the right document through a protected connection, but later a storage failure makes the retained copy unavailable. The earlier connection result never demonstrated that saved copies could be restored. If the provider supplies storage, recovery belongs to that promise. If your own accounting program keeps the copies, ask about recovery there instead of expecting an archive from a delivery-only package.
A missing answer is not a successful answer
There are several ways the permissions question in our example can remain unanswered. It might not be part of the public-page tool’s work. It might be planned for a wider assessment, but the assessor never receives the agreed test account. Or the check may start and stop before reaching a result.
In the first situation the question was not tested. In the second, missing access prevented testing. In the third, testing did not finish. None is the same as a favourable result. None automatically proves the permission check would fail either. Reporting guidance recommends making practical limits, including missing access, visible. 8
There is also “does not apply”. If the provider genuinely supplies delivery but no storage, a check of its customer archive may not apply to that package. Your business still needs to know where its own usable copies are kept. Saying a provider does not supply a particular feature is not the same as saying the business no longer needs it.
Products can use different names for these states. My recommendation is to keep their meaning distinguishable from passed and failed checks. A buyer should not have to discover later that a reassuring summary quietly left a crucial question unanswered.
For an agreed customer-separation assessment, the supplier and assessor can use invented businesses, documents and approved accounts. That is separate from a normal demonstration of sending one invoice successfully. Buyers should not try to fill a report’s gaps by experimenting with real customers’ records.
Why a changed score needs an explanation
A dashboard can combine different concerns. It might also assess speed, how usable pages are for people with disabilities, or how easily search engines find content. Those are worthwhile subjects, but making a page load faster does not show that customers are better separated. If an overall score improves because a different category improved, the security conclusion must not quietly improve with it.
The same applies over time. A later report might examine more pages, introduce a new question, change the scoring rules or reflect a change in the service. In an imagined example, the old tool checked only the homepage and the new one includes sign-in. A lower score would not by itself prove the service had deteriorated; the new report may have revealed something the old one never considered.
Conversely, fixing a particular defect can improve its check while leaving other questions untouched. A confirmed repair is useful evidence about that repair. The rest of the service does not become proven safe by association. Even a careful assessment describes particular conditions and times, and later changes can alter the result. 8
When a supplier shows green, ask: “Does this report include the service our staff will use with invoices, and which important questions are outside it?” A plain answer can distinguish public pages, the protected connection and document permissions without requiring you to become a tester.
Keep the favourable result for the work it actually assessed. Then resolve the remaining question most relevant to your own use, whether that is access to documents, recovery of supplied storage or the delivery state your accountant will see. Green is a helpful starting summary, not a replacement for understanding the service.
Sources
3 — OWASP API1:2023 Broken Object Level Authorization
12 — RFC 9846: TLS 1.3
8 — OWASP WSTG v4.2 Reporting
11 — MDN: How the web works