Computer Systems Validation Part 6 – Vendors

In this article, I will discuss the Suppliers, the Vendors, the Software Developers and the Hardware Providers.

I will look at what we in the regulated industry need from them, what we should expect from them, and what we get (or in what is very often the case, don’t get) from them.

Strap in, this one is going to be bumpy…

CSV Vendors

In my experience most project delays, hold-ups, issues, and failures are caused by two things. A lack of resources made available for the implementation, and most of all by some distance, vendors.

The Third of the Four Horseman of the Validation Apocalypse, “Conquest”, is potentially the most troublesome of all, even unwittingly. I described it (twice) in the second blog in the series on “Why Validate?” thus:

“Vendors can be your best friend or be your worst enemies. A well-run supplier with good quality systems and an understanding of healthcare industry requirements absolutely should be the first requirement for every new system.”

There, I have stated it again. And I genuinely believe it cannot be said often enough.

The GAMP Guide has an ESSENTIAL section on Supplier Responsibilities – Chapter 7 – and I would highly recommend that this is read and understood. Once you have done that, read it again.

The key words throughout this whole chapter are “leverage” and “assurance” – i.e. getting the vendor to do as much of the work as possible for us, having a high level of confidence they have done it correctly, and using this as assurance to support our qualification activities.

The key phrases throughout the chapter are “The Supplier should provide…”, and “The Supplier should manage…”. And absolutely the Supplier should do this.

Which sounds fairly straightforward when described like that – as all a customer would require is the vendor to have developed, tested and recorded what was done based on a recognised and effective quality system, and is able to provide appropriate evidence to support this.

You would like to think they would all do this, but the word “should” above does a lot of heavy lifting. This is where it all rather starts to unravel…

In my experience over the years of having dealings with many vendors, there have been a small number that could truly tick all of the boxes listed above. And from that point onwards there is an alarming sliding scale downwards.

I will discuss the sliding scale later on in this piece – what this means to you in terms of expectations from the vendor, and what additional work this will potentially mean for your organisation.

Note – I won’t name any of the companies here in this blog – but we would of course be happy to provide advice on selecting suitable vendors if this is a service that might be of benefit to your organisation.

How the Process Should Work

So, you have created your initial URS describing everything you might need in plain language.

I discussed the URS creation in detail in an earlier blog here, but it is important to once again stress the importance of wording, and how things can be misinterpreted.

For example, I heard a joke the other day from Sweden, and it goes like this (apologies for paraphrasing):

A software engineer says to his wife he is off to the shops and does she want anything. She replies that they need a bottle of milk and if they have eggs, to get 6. The man goes to the shops and returns with 6 bottles of milk.

“Why have you got 6 bottles?”, she asked

“They had eggs”

There is a very famous cartoon of a tree-swing, and how each individual in the development process understands everything differently and left to their own devices will produce wildly dissimilar solutions to the same requirement. Although it is tongue-in-cheek, the reality of this is actually very real indeed:

CSV Vendors

Joking aside, these 2 examples in many ways highlight the challenges with the initial discussions and understanding with vendors.

Without putting too fine a point on it, a vendor is unlikely to have much knowledge of your business processes, so it is critical that the language used throughout any interaction with the vendor is such that everyone understands each other explicitly with no space for ambiguity.

Yet again this is a job for the Translator – that one individual who can understand the different languages and terminology that each of parties in the process use, and can seamlessly explain what one is saying to any of the others

The Translator is critical here to ensure that the above cartoon never happens

(Note – I’m beginning to think I should devote a whole blog piece on the role of the Translator, such is the frequency I bring it up!).

CSV Vendors

GAMP Categories

I have discussed GAMP Levels in previous blogs, but in particular in the preceding blog on Risk Assessments.

Depending on the expected GAMP Level of the system, you will need the vendor to provide different quantities of supporting information and documentation. This becomes more critical as the GAMP level increases.

GAMP Levels 3 and 4

For a typical GAMP Level 3 and 4 systems we would expect to see some form of documented evidence that the system has been developed and tested as per the vendor’s quality management systems. For Level 4 systems, the additional assurance provided by a successful quality system audit should be enough to provide everything that is needed.

Both of course may also require installation and configuration records and evidence if this has been done by the vendor.

GAMP Level 5

It is when a bespoke Level 5 application is being implemented / updated that things get more…interesting.

Below is the Level 5 ‘V’ Model:

The pale-blue boxes at the base of the V are the important ones for this discussion, as they relate directly to what we expect from the vendor – and this can be challenging.

What we require here is:

  • A module design specification – i.e. a document describing how the vendor will code the system. This should include:
    • Programming standards
    • Development methods (see the next blog in the series for this one)
    • Module Titles and Function Descriptions
    • Any Pre-requisites, co-requisites and restrictions
    • Supplementary regulations affecting the module
    • Expected Outputs
  • A Module Development document that is essentially the report from the application project:
    • Development Plans
    • Regular feedback sessions with the customer (again, the next blog entry)
    • Logs of development milestones
  • Vendor Testing Evidence
    • Evidence that the system has been tested against the developer specifications

These do not have to be separate documents, and indeed I have seen some that have been combined very effectively into a single volume.

Again, it is all about assurance. No one wants to have to verify each keystroke in a system!

Note – I did have to do one of these in the distant past. Luckily it was not a very complex system, but writing the test scripts and then subsequently testing every user step was a Herculean effort. I do not recommend it.

As you can imagine there are many vendors who would be unable (or more likely unwilling) to indulge you on this endeavour, which means a round of discussions and understanding exactly what they can and will provide you with.

I recall one such vendor who promised everything we might need at the start. But as time went on the documents did not appear despite repeated chasing.

When the documentation did finally appear (just as the system was installed ready for testing) it amounted to about 4 pages in total – which essentially said “We developed it and then tested it, and this is the version number [which for reference, was incorrect]. It was all good.”

That was a stressful project…

Much of the discussion below in fact relates to a vendor for a GAMP Level 5 system – but the experiences and considerations should also be used for providers of the lower-level systems.

Vendor Representative

So this leads on to another thing to note, and that is the Vendor Salesperson.

With many small vendors, the sales pitch is often provided by a senior (or only) developer, and this can often be very constructive. But as soon you are dealing with a medium or large operation, you are having to deal with the “SALESperson”.

And that is the thing. Always remember that the Salesperson’s job is to get your money so that they ultimately get paid.

They will be smiley.

They will be very lovely people.

They will promise you that their system / any system they create will do everything you require, and they will provide you with all the documents you could ever want.
With bells and whistles.

But.

  • They might not understand what you are asking for or the general requirements for GxP applications
  • Their system may just not, in fact, do what they say it will, or everything you require
  • You may not actually get the supporting documentation you need

So, there are clearly some questions you need to ensure that you ask when you are approaching a vendor.

Assuming you are not directly concerned with finance and delivery details, these really should include:

  • What experience do you have with Regulated Industries?
    • And yes, this does echo my much-repeated statement
  • What Quality Systems accreditations do you have?
    • If none, describe your quality systems
  • Have you reviewed the URS, and if so, how will you meet our requirements?
    • A created package describing the approach in detail is a good win here!
  • What documentation will you provide?
    • And can you provide some examples up front?
  • What development approach do you use? (see the section below and the next blog)

If they can answer all of these satisfactorily, then you are off to a good start.

If they cannot, then start considering what this will mean to your organisation, and the extra resource and effort this may require from yourself.

Of course, the reality is that often the choice of vendor is out of your hands, with your input at best one voice of several, but you must ensure that GxP requirements will be met.

And as is usual with these things the loudest voice is the person tasked with finding the cheapest option out there to ‘tick the box’. The additional pain that you will suffer as a result of their decision is not their concern.

My only advice is to stick to your guns and speak up. And maybe convince someone who has some clout to speak up for you as well.

Development Methods

There are many development methods used by the software industry, and these can include Agile, Sprint/Scrum, Rapid, Waterfall, DevOps etc (note – the last of these is used on systems for regular and routine updates and deployment). There are others.

I won’t go into detail on what these mean – as I will discuss this more in the next blog post.

Approaches to Different Vendors

Depending on the experience of the vendor and their ability/willingness to cooperate with the customer, the level of challenge and workload that is required to validate will shift accordingly.

Obviously, the scope of any exercise will be defined in the initial Risk Assessment, but this can only be based on what the vendor has stated they will do, and not what the vendor actually does.

The Perfect Vendor

The Perfect vendor is one where they have plenty of experience with the regulated industries and understand the requirements and expectations that go with that.

They will work with the customer and provide all the supporting documentation that could ever be required on time.

This is the ideal, and you should be able to leverage them for everything you may need to support your validation effort.

The Typical OK Vendor

A typical vendor will have limited or no experience with regulated industries but will have some kind of quality systems in place – often ISO 9001.

You can have a level of confidence in their internal systems, and although they may not know exactly what documentation you require – the conversation often will be “what information do you need?” – there is at least an expectation that you will receive something to support your effort.

Whether the documents meet your expectations or are actually delivered anywhere near on time is another matter. Work with them as best you can.

The Typical Not Quite OK Vendor

This vendor is the one that doesn’t really have a QMS of any note and has no experience with regulated industries.

If you are lucky then they will be generally helpful and will endeavour to create the documentation you request – but it is likely that this will not be anything like the ideal, and certainly not enough to allow significant reductions in workload on your part.

If you are not lucky then they will offer very little – meaning you almost have to do the work for them.

A perfect example of this is where I remember with one vendor where I had to write their Design Specification (DS), test document and summary for them. Slightly less time needed than if I had just done this as part of my own validation, but not much. And even then, I got the lasting impression that the vendor was a trifle bewildered by it all

The Not at All OK Vendor

Oh dear.

There are a few variations here. These can include:

  • The vendor where the salesperson vastly over-promises and the developer vastly under-delivers
  • Where the vendor repeatedly doesn’t deliver on their targets, then lies about it with a gush of excuses.

They know they are lying. You know they are lying. They know you know they are lying. But you just can’t prove it.

  • Where the vendor ignores you and develops or provides you with whatever they want and not what you want; seemingly forgetting that you are the customer.

They then get awfully offended when you dare to bring it up.

You absolutely should avoid these vendors at all costs. Of course, however you might not find out that they are a Not at All OK until it is too late…

I have had experience with all of these, and they generally have fallen into the Typical OK / Typical Not Quite OK categories.

I will however recant the story of about my experience in a future blog with regards to possibly the worst experience I have ever had with an utterly awful vendor.

Sub-Developers

One thing to maybe be aware of is that many software vendors use third-party developers for the actual coding. And they are not always terribly forthcoming about this fact.

The general rule of thumb here is that the expectations you would have with your vendors would then also apply to their third parties – except the responsibility for that sits with the vendor and not yourself.

Ask those difficult questions, you will need assurance.

Hardware

There are 2 different scenarios with regards to hardware provision.

The first is where the vendor has built custom hardware. This is usually where a piece of equipment has been purchased (essentially the reason for the purchase), and this comes with associated applications and supporting software.

In this scenario you may or may not be involved with the validation of the equipment as well as the software, but be aware that the vendor may well not be the software experts, and:

  • May not have developed the software in-house
  • May have had one engineer develop the software who may well have not followed any good practices in doing so

The second is where the vendor says they will provide a supporting PC to run their software. From a purely business perspective, keep an eye on the cost of this – it is likely to be vastly overpriced!

Better to find out the optimum specification of the PC hardware from the vendor then involve your IT department and get them to purchase it themselves for a fraction of the cost.

In both scenarios the same CSV rules apply for the software elements as part of a wider equipment system.

Design Specifications

So, one of things you should always insist on is that the vendor provides you with a Design Specification – i.e. a document describing how they will meet your user requirements. This is an important document as if it is well done it allows your Design Qualification (DQ) to run very smoothly.

In my experience I have found that the likelihood of getting this document with everyone except the Perfect Vendors can be quite hit or miss. Some vendors will work hard to provide an entry that will address each and every line of your URS, some may provide general assurances, and some will essentially just provide a statement saying you will get a system that will do exactly what you want. Be very wary of this last one as any such vague statement typically follows with a system that does not do exactly what you want.

And of course, trying to complete a DQ with vague and incomplete response from a vendor can be difficult indeed. Do you trust the vendor to meet the requirements? In which case is a cut down DQ that essentially states “the vendor has provided assurance” really good enough? Arguably not really (and especially if you follow the GAMP Guide), but this is the reality you may have to deal with. It will mean more testing later on.

Hosting

Arguably most systems these days, unless they are integrated with equipment as part of a control system, are hosted. Be this as a wholly remote application and database, or partially where the application is locally installed as a client, but the data hosted.

And usually, it is the vendor who will decide how and where the system is hosted.

This is fine as long as the vendor will provide full information, or at least tell you where they have hosted it so that you can get the required supporting Installation Qualification (IQ) information from the host directly.

The problem yet again here is that the vendor may not be that forthcoming when it comes to offering-up the details you need.

I remember the technical contact at a vendor I dealt with not so long ago didn’t even know who the host was for their system, nor indeed where it was located.

Eventually after much pushing, they suggested it might be at a host in California – but I knew this was not correct as the system had to be based in the UK/EU for it to be valid for use.

We got there in the end (it was based in Germany), but not before me contacting the US Host they insisted they were using with a request for basic information (general physical and electronic security statements, ISO certifications etc – all the usual stuff), which they refused saying that this information was confidential.

I will talk specifically in a later blog about SaaS and Hosting, but I will finish this section with a discussion I had with a (very) small-scale developer in Scandinavia.

I was asked to perform some validation on a simple document filing system, and when I asked the question regarding where the system was hosted and how was it secured, the answer came back:

it’s on a PC in the corner of my bedroom, and only me and my flatmate have keys to the apartment”.

Try wording your way out of that one on the validation docs…

Charges

Although not specifically the validator’s problem, It is critical to understand and have full visibility of the charging models from the different potential vendors including the costs of the initial system/hardware (including installation),  hosting, software licencing, validation, ongoing support and upgrades. This is important for both budgeting purposes and to help compare the costs of different systems.

Vendor QMS

As you can imagine, different vendors will have a range of supporting Quality Management Systems, from those with fully blown mature electronic systems, through as ISO9001 or equivalent accreditation / approach, all the way through to little or no QMS whatsoever.

Obviously any audit will identify the effectiveness and scope of a vendor QMS, but as a general rule of thumb, a fully mature system, or an ISO9001 accreditation <ISO 9001:2015 – Quality management systems> would generally provide good assurance of good quality process governance. ISO 27001 (information security) <ISO/IEC 27001:2022 – Information security> is also a very good, if not essential, one to have.

One word of caution here – I do remember very well a company who claimed to have a legitimate ISO9001 accreditation, yet when quizzed not only did not have any Change Control, CAPA, Document Control processes etc in place, they didn’t even know what a Change Control process was. Of the 7 principles of ISO9001, they failed to demonstrate any.

I am not trying to belittle the ISO accreditation process here by the way; I am just suggesting that you maybe might like to check…

In general though, some sort of supporting QMS should be a good sign.

I have stated previously that you should avoid vendors with no systems in place, but if you are in a position where you have no choice (i.e. someone else has made a purchasing decision and this is what you have to work with), ensure you ask the right questions, and frame them in a way that the vendor may understand. There is little to be gained from going in using full Pharmaceutical Quality Management System terminology, as it is highly likely that they won’t understand what you actually mean.

Better then to ask the crucial questions in simple language – “Describe your development process”, “How do you test and how do you record those tests?”, “What would you do if I found a bug?”, “How do you control update releases?” etc.

You really need to work with the vendor and lead them through to get as much as you possibly can.

There are of course a whole host of additional things to consider such as the customer audits and online customer reviews.

The second of these… it is safe to assume that no customer reviews provided by the vendor are going to be bad (or indeed even necessarily genuine), and finding independent reviews is not always that easy, even online.

Better then to try to find out who their existing customers are and ask the customers directly what they think. There is a definite bonus here in having a good network across the industry and / or a wide footprint on LinkedIn so that you will be able to speak to someone with first-hand experience.

One word of warning with this – it does seem to be human nature that we will often equate “not quite right” with “bad”. If a user says that their software is “bad” (or similar), it is worth asking exactly why. I have found that often their issues are more to do with system setup or procedural inefficiencies around the system than the system itself.

I will talk about audits and auditors in a future blog, but for now, use your Translator to ensure that the questions are asked in a way that the vendor will understand – remember, it is very unlikely that they will be familiar pharmaceutical regulatory expectations.

One last thing to consider is Customer Service – i.e. what will happen when you find an issue post-Go-Live. I can give a good few examples where the vendor’s team I worked with up to the point of Go Live was excellent, but as soon as the system was in use I would now have to contact and deal with a help-desk operative, whose language, level of expertise, understanding, and general manner was not that of the implementation team.

Remember, you are the customer. If a response is not good enough, straight bat the ball back to them and escalate if you have to.

Conclusion

I could probably write a whole book on vendors, but I’ll leave it here – hopefully I have highlighted plenty of things for you to consider.

There are Perfect Vendors out there – those who are a pleasure to work with and can provide you with everything you need.

There are Typical OK vendors – those who will work tirelessly for you even if they are not necessarily perfect.

There are the Typical Not Quite OK vendors that although not ideal, allow you (with a little bit of extra work on your part) to get the job done.

And then there are the Not at All OK. Hopefully above I have given you some pointers on how to spot these and enough knowledge to allow you to steer well clear.

To summarise, there are a lot of things that vendors should do.

But the upshot is that many vendors will not do that for a whole variety of reasons. As validation experts we can only do what we can. Where the GAMP Guide describes the paradigm, hopefully I have given you some ideas about the sliding-scale reality and how to best adapt and deal with that.

In the next entry in this series, I will discuss System Development, and yes, this too involves vendors.

In that I will look at things to expect from a developer, how you really need to be part of the development process at all stages, and by extension (and not surprisingly), how “Communication is King”.

To find out how we can help your organisation with your Computer Systems Validation and Data Integrity needs please get in touch at consultancy@jensongroup.com or to read more about our CSV services, please click here.

Read all posts in this series:

Neil Rudd
2024-05-07T10:27:30+01:00May 8th, 2024|
Go to Top