Give me a G!
Give me an A!
Give me an R!
Give me a Y!

Gaaaaaaaaaaaaaaaary!!!!!

Gary Collins.

That’s the name of the guy who knows the most right now about clinical AI validation.

You can look him up in Lancet and stuff like that.

And he’s at Oxford, I think.


With his mates, he’s set up a sort of observatory of papers with AI algorithms for Covid.

They check to see if they have what they need to have.

And the results are devastating.

Most of what is done is wrong.

And only some of it is right.


Of course.

This is not surprising because this was already the case with medical literature before AI.

Most of it is garbage and every now and then something comes out that changes clinical practice.

It is neither better nor worse.


Now is when I tell you what is really interesting….


The other day, the Savana data team was at a course with Gary.

And he told us something that fit in with what we know.


I’m going to tell you about it so you can show it off when the AI topic comes up.

You’ll see.


A lot of people, even a lot of very smart people, think that the way to do the internal validation of an algorithm is to reserve 1/3 of the patients and do the validation with that 1/3.

That is not the way to do it.

Even if the NEJM editor tells you so.


The way is to create the algorithm with ALL the patients

And then do a process where you remove 1/10 of the patients each time and test the algorithm.

And you check that its performance remains stable.


Not only is it more reliable.

It also allows you to build your algorithm with more patients.


There is no reason why you shouldn’t do it that way.

And now you know.

Thank you, not for nothing, that you’re on this list.


And finally, for the sole purpose of getting you to buy me many beers when you see me, I´ll give you the name of the technique:

K-fold random sampling with replacement.


So there you go.

If you want to make a project that is validated, here.