This article reads more like PR than research. Research is based on 76 urine samples which is way too little. In the paper, there's little info on model hyper-parameters as well as how they trained and tested them. I suspect they overfitted their model, meaning it just learned all the samples. If that's not the case, the only other explanation for such high accuracy is that there's a very obvious correlation and a human would be able to diagnose cancer with high accuracy also, meaning you don't need any AI.
> Research is based on 76 urine samples which is way too little
I'm sure I'm out of my depth, but I feel compelled to point out that reputable clinical research can be done on even smaller samples[0].
I want to make sure we aren't conflating the training set and the test/validation/holdout set [1] -- I interpreted the article as stating "76 samples in the test set" and assumed the training set is far larger.
The paper is really about developing a multi-marker sensor approach for screening.
The analysis, however was done on a small set "Obtained data from 76 urine specimens were partitioned randomly into a training data set (70% of total) and a test data set (30% of total). " Later on it states the 76 samples are from 51 individuals.
So the GP complaint is correct in this case. They are comparing a RF approach and a NN approach - there isn't much detail in the paper but it's most likely the analysis is problematic. I suspect this was done because the group is a collection of hardware and clinical folks, not statisticians, and the paper would have benefited by review by a biostats person.
You are quite right generally that for some models and some effects, a relatively small verification set can be effective.
It really depends on the model. If you're estimating a mean from a distribution with low variance, 76 can be quite enough. But for any even half-involved ML model, that's nothing.
“significant results” can be yielded from 76 samples, provided they all be independent, and one can practically bet that they are not.
But this is a claim of accurate diagnosis which requires more than simple significance. The a.i. being demonstrated to simply be better than chance would be a significant result, which is obviously not a spectacular standard for diagnoses.
The issue is that the meaning of “significant” is very often overrated. It means nothing more than “We are highly confident that there is a stronger effect than zero.”; — this does not mean this effect is particularly large, and most importantly, it does not mean the effect is universal.
So, they only had 51 individuals in the study. Then they actually divided the patients in to 3 groups: normal, PCa with pre-DRE, and PCa with post-DRE. The cancer patients thus had 2 samples, pre and post-exam: 26 + (2*25) = 76.
Given the false positive rate of PSA screening, I would imagine that including more normal samples would have been more beneficial here. The method of analysis is pretty interesting though, which is where I think the novelty is for this paper. I assume that a more thorough clinical trial is planned with better numbers. This paper looks like it is more of a validation of the device than anything else.
That's right, sample size can vary a lot depending on several factors such as:
* Experiment design
* Sampling techniques
* Methodology being used on the model
But normally with a small sample size a bigger emphasis should be put on performing the appropriate statistical tests on the sample and validating it carefully, before applying any model. There are a lot of assumptions that should be validated if you are working with small data.
Unfortunately, I don't have access to the complete paper to verify the methodology used, so take it all with a grain of salt.
There's a great paper about small sample sizes in the context of experimental surgeries (where usually the work is done at a single regional hospital on 3 patients): "If nobody died, is everything OK?" It's cited in Glantz's Primer on Biostatistics.
Agreed. This article raises more questions than answers. How were the samples obtained? What does "almost 100 percent accuracy" mean? What was the source of non positive samples? How many total samples were there?
Granted...through some weird HHS/medicare loophole, this is commercializable. Our center has a AI "stroke/clot" detector algorithm software, that's based on a training set of 50 CTs. It's ok, say at a 2nd year medical student level at reading images, i.e. not to the point anyone would trust them. But if the clinical writes in their documentation that the software was utilized to analyze the images, the hospital gets an extra couple grand of reimbursement from the insurance company, apparently. Exactly what rules apply here, I don't know.
>The team developed an ultrasensitive semiconductor sensor system capable of simultaneously measuring trace amounts of four selected cancer factors in urine for diagnosing prostate cancer.
it would be interesting if they run as a control a human looking at those measurements.
One can also wonder how many other cancers can be detected by dipping a chip in urine and whether those chips can be just made into a smartphone accessory for home/field use.
> there's a very obvious correlation and a human would be able to diagnose cancer with high accuracy also, meaning you don't need any AI
IF (big if) that's the case, arguably you still prefer something that runs on some electricity in few servers instead of having thousands of human diagnosticians who need to be housed, fed, educated, entertained and later retired.
So this is super awesome if true and I hope they can use the technique for other cancers.
I'm curious though, my bf says his doctor asks if he wants a PSA at his checkup. His doctor then goes on to explain that pretty much all men eventually get prostate cancer and there's not much you can do about it and you're far more likely to die of something else first so, not really much reason to get the PSA
Anyone have any other info? Is there actually treatment and is it worth while?
A PSA test is easy and non-invasive and I personally wouldn't skip it. But keep in mind that it doesn't have a lot of diagnostic accuracy by itself, and the false positive rate can lead you to do things that are more invasive than necessary. Family history should be considered.
If the numbers are low, you're probably good. If the numbers are high, it isn't necessarily bad, but it could be good excuse to keep a closer eye on things. You might do additional PSA tests and look at whether it's trending up or down. A biopsy might be suggested, which is more invasive, and can be more accurate, but it also depends on where the doctor is able to collect the tissue -- if they miss the spot, a biopsy could be inconclusive or contradictory.
Finally, if you have prostate cancer and the numbers are bad and your doctor tells you that it needs treatment, the common treatment seems to be high intensity focused ultrasound (HIFU), which is unpleasant and invasive, but it isn't surgery, and it isn't anywhere near as damaging as radiation, and apparently it works rather well.
It is true that in old men there is a large percentage of slow growing mostly harmless prostate cancers, but you can also get it if you are younger (or it can be aggressive) and then treatment is possible (and required). The advantage of doing a PSA test is however not clear cut, because having a (slightly) elevated PSA can be caused by a benign enlarged prostate or prostatitis. This is why this new test is worthwhile (if it really works).
Source: I'm 47 and have prostate cancer (metastasized, so incurable but there are still a number of treatment options to improve quality and quantity of life).
What exactly do you mean by "a large percentage"? Cancer.org[1] says that it's 1 in 8 men during their lifetimes, the parent post says "almost all men get it at some point" -- where is this info coming from?
The way PSA is a useful marker is not the absolute value but the rate at which it increases. If the doubling time of your PSA is too high it means you're at risk of prostate cancer and need further investigation. Get it done twice in your 30s and 40s and more regularly when older than that.
Take 1000 men over the age of 50 and give them PSA and DRA for about 15 years. Take 1000 other men over 50 don't give them PSA nor DRA over the same time.
> About 10 out of every 1,000 men with screening, and 12 out of every 1,000 men without screening died from prostate cancer within 16 years. This means that 2 out of every 1,000 people could be saved from death from prostate cancer by early detection using PSA testing. This was not reflected in overall mortality.
Over all about 322 men died in both groups, so it doesn't seem to make much difference to all cause mortality.
However, in the group who was screened we see 155 people had a false alarm (and that includes unnecessary tissue removal), and 51 men with non-progressive cancer had unnecessary treatment (that sometimes includes impotence or incontinence).
The most important question you can ask a doctor is what happens if we don't do this?
There are different types of prostate cancer. Some are very slow growing and people tend to die with it, not of it. There are aggressive types of prostate cancer that do kill people, but these tend not to be found in time to make much difference to populations at the moment.
I think it's important to have some conception of a personal "risk tolerance" when it comes to screenings.
Personally, I've had false alarms and dealt with that stress, and I still find that screenings are worthwhile. Having seen family members/friends die of cancer, I don't find the additional and potentially unnecessary discomfort of screenings to be too much of a cost.
At the same time, I'd never judge someone for feeling the opposite, particularly in cases like prostate cancer, wherein most people die "with it, not of it" as they say.
Your BF's doctor is uninformed and offering bad advice.
Fact is, most men over age 70 do have elevated PSA due to small amounts of low grade prostate cancer cells (Gleason score of 3+3). Usually it advances slowly, never growing beyond the bounds of the prostate gland before the patient dies from some other cause.
But an elevated PSA (over 1 and below 10) in someone younger than 70 should not be ignored. Often a second PSA test is done perhaps a month later to confirm the first score and to see if the number is rising quickly. If repeated tests do not show PSA elevation AND the score is low, often the urologist will recommend a "monitor it, and wait and see" strategy.
If the PSA number is high or is rising, the next step is to get a biopsy where 12 to 18 samples of tissue are taken from the prostate to see how much of the gland has cancerous cells (Gleason 3, 4, or 5).
I know something about this topic because a routine PSA test at age 55 showed that I had asymptomatic cancer in 80% of my prostate. Luckily it was removed surgically. But if my doctor was as casually unconcerned as your BF's doctor, my cancer would have metastasized and by now it would be incurable.
>>> Your BF's doctor is uninformed and offering bad advice.
I wouldn't go that far. In your n=1 case it seems clear that not monitoring PSA would have been an error, but there exists a debate on the risk/benefit of this test. Namely, the unnecessary suffering that it can inflict to a healthy person with elevated PSA.
This is a good review [1]
"To screen or not to screen for prostate cancer? This remains an important question. Screening relies on a highly imperfect measure, the prostate-specific antigen (PSA) blood test, which is prone to false-positive results. And with mounting evidence that survival benefits from screening pale in comparison with the harms from overtreatment — particularly incontinence and impotence — the pendulum has steadily swung away from it. Still, screening research continues, in the hopes that some lifesaving benefits may be found."
I'm keenly aware of the PSA debate. Policy about its proper use been see-sawing back and forth for a decade.
Starting about 10 years ago, just before it mattered to me, official policy decided to NOT screen using the PSA test. The belief was that underinformed primary care docs were overreacting to a high PSA number and ordering too many biopsies which often led to "unnecessary" infections. Of course, the right response was to do a better job of interpreting the test results, ideally to refer the results to a more expert urologist before doing a biopsy, not to cut back on the test.
Given its low cost ($50) and very high sensitivity, the PSA test provided a very valuable service that could be equalled by no other test. Physical exam is often wrong, missing a large fraction of positive cases of cancer. And biopsies introduce infection most often in older patients. Younger ones can tolerate biopsy better, but were disallowed from PSA screening entirely due to this overprotective policy.
The right solution was clearly to interpret the PSA test results more judiciously by introducing more expertise prior to biopsy, knowing that PSA is overly sensitive for diagnosing cancer. Fortunately the official policy has since been reversed, and PSA has returned to routine use -- now with inclusion of a second test or a urologist prior to biopsy.
The same blunder took place in mammographies at about the same time. Too many false positives led to too many biopsies and thus routine screening with mammography was deemed unacceptable and it was deprecated as well. Fortunately that overreaction has also ended.
Sensitive medical tests are essential. Overreaction to possible misinterpretation of positive results by inexpert GPs is the problem, not the test itself. I routinely thank the stars above that my GP was expert enough to know that.
There are things that can be done, especially if the tumor is small and hasn't spread to adjacent organs. I'm not an expert, but in general, solid tumors are much more easily treated if caught early. Still, "watchful waiting" is a totally valid option for early-staged prostate tumors.
That said, the doctor might be trying to benevolently prevent unnecessary tests, stress, and treatment. Some people hear "cancer" and rush into risky treatments. This is part of the reason the United States Preventative Services Task Force "demoted" asymptomatic PSA screening from "recommended" to "have a talk with your doctor." Their review of the studies didn't show that screening asymptomatic men resulted in a large reduction in mortality. Mostly because prostate cancer is so common and often develops slowly. Somewhat because treatment can bring its own dangers.
At first, in 2012, they recommended against it. Then doctors and patients' groups protested, so now it's "have a talk." We are now starting to see the result of that first decision in the increasing incidence of late-staged prostate cancer.
All this is to say: PSA screening's not a bad idea as long as you can keep perspective if it comes back positive. Out of the major cancer types, prostate cancer has one of the highest 5-year net survival rates [0]. It's over 95% in my state[1].
[0] Net survival tries to exclude the risk of death from other causes. So 100% net survival for a cancer diagnosis means there's no difference from a similar person who didn't have cancer.
[1] The official statistic from the CDC may be lower, but I disagree with some adjustments they make. I believe they force the data to fit their mental model.
It's basically true that most men have it and live with it for a long time (even decades) -- one can see the evolutionary reasons not to select against it.
However some cases are more aggressive and will have a negative impact or even kill them.
The problem, as with most cancers, is to figure out which ones are worth treating rather than risk iatrogenic consequences.
I had prostate cancer 10 years ago when I was 42, prostate surgically removed and doing fine now.
I also heard from urologists that everyone eventually gets it but normally you can outrun it. If you have an aggresive variety or get it when you're young you should definitely do something about it.
The tricky part is that the current PSA test is not specific enough to avoid the stress and extra human cost of overtreatment so a better test would be great.
Everything else is pretty straightforward in terms of diagnosis and there are decent treatment techniques, I had a simple needle biopsy that confirmed the cancer but for many men this test (although easy-peasy in my case) is scary, expensive and often confirms there's no cancer at all.
My father was treated for prostate cancer with internal radation pellets.
I don't know the time between the 'treatment' and his death, could be anything from 12 - 24 months. I know at least one of the pellets had started 'wandering'.
He died of acute leukemia and I suspect that had to do with the radation. However, I am not pursuing research into this, it won't bring him back. He was 63 years old.
There is probably some truth in all men eventually getting it, but there are for sure contributing factors that increase and decrease risk. I've run across many of these by mistake in my nutritional research on nih.gov. That is a great starting place to find some of the research.
Not every cancer is created equal. I helped an urologist typeset his PhD thesis about prostate cancer treatment in TeX (he used some math equations within) and he told me the same.
Some cancers are aggressive and some are slooooow. In younger patients, cancers tend to move fast and kill fast. But a slow prostate cancer in a 70 y.o. may be better left as it is, because the risks of the operation may actually be higher.
This, of course, is a difficult judgment call and belongs to the experts only.
I always wonder about bias in these statistics around age.
It makes intuitive sense to me that a cancer diagnosed in a young patient, who is below the common screening age, is probably being diagnosed because it is presenting serious symptoms (i.e. is growing fast).
Cancers diagnosed in a 70 year old, on the other hand, would seem much more likely to be diagnosed while they are asymptomatic and relatively contained, or to be cancers that have been growing slowly for a long time (more likely as the 70 year old has been alive longer).
Obviously, I don't know if this is the case or not. If anyone has experience in this field, I'd love to hear about it.
How do clinical trials work for ML? Since the system is largely a black-box, I imagine that the slightest alteration (training on more data, tweaking any parameters) would require totally re-doing a full gamut of trials, wouldn't it?
I'm a regulatory consultant for AI software and submit several AI/ML FDA submissions per week. Overall, I've submitted 100+ AI/ML submissions to date. Generally, FDA has been really focused on locked algorithms even though there has been some guidance lately stating the the new Good Machine Learning Practice (GMLP) will provide some allowance for iterations without submitting new 510ks. I will say that a number of companies are slightly tuning their algorithms without FDA resubmission within the confines of the FDA Guidance "https://www.fda.gov/regulatory-information/search-fda-guidan...". That said, most companies most companies are just leaving their specifications at sensitivity/specificity of 80% which gives them some leeway to improve above those limits. In truth most companies are above 97%-99%. While most of the submissions I work on are MRI/CT image related, I'm starting to get a lot more in the predictive space centered around ovarian (CA125), breast or PSA cancer scores. Lately pathology AI is escalating rapidly.
A lot of thing are blackbox. There also things that we think we understand, but we do not, thats why we have methodologies like double blind testing to assert that the blackbox is indeed working, and it's behavior is stable enough to be used as a tool.
I guess it makes sense that unlike for actual treatments, for diagnostic systems you could just "re-use" the test data from past trials and not have to gather a whole new dataset
90% of male cadavers have cancerous tumors in their prostate. The cost of treating men would be enormous. The medical opinion will be that it is harmful diagnostic because the patient (and the medical system) are better off not knowing. If you think this is a joke, it is because your doctor has never said this to you.
I'm surprised it's that high so I looked for some papers. Only two references I found so far are below which show much lower than 90% in autopsies. I'm not in the medical field so please help me understand.
Last I saw that stat was 2008, and I think the study was much older. I shouldn’t be so sloppy. Looks like it’s 60% for men 79+, but this is the no prior screening group, which specifically excludes known cases, so maybe a bit higher. Breast cancer is 12% and all cancer is 40% lifetime diagnosis risk in the USA.
This news begs a big question: How much does this new test cost? That will decide how it might be adopted.
This test requires a chip rather than a simple lab assay. If the cost of the test is comparably as low as PSA ($50) it could entirely replace PSA for screening. However, if the chip is more expensive than PSA, it still may be very useful -- as a second test after a PSA screening test reports a high number but before a biopsy.
If this new test could diminish the number of infections from unnecessary biopsies that worry policymakers, routine PSA screening combined with this test could become a low-risk no-brainer new standard.
My dad had prostate cancer a few years ago, which was treated successfully with radiation. He now gets his PSA score twice a year -- once with his annual physical, and 6 months later with his cancer doc.
I wonder whether the biosensor could be something patients like him could have at home. (Also, now I'm imagining Apple telling customers they can urinate on the next generation of the watch.)
You can read it on a boat;
You can read it with a goat;
You can read it in the rain;
You can read it on a train;
You can read it in a box;
You can read it with a fox;
You can read it in a house;
You can read it with a mouse;
You can read it here or there;
You can read it anywhere!
This isn't AI. It's a random forest with FOUR inputs and ONE output. The fact they put a huge big blue AI brain in their paper to represent this algorithm tells me this is mostly hype.
