Why does everyone act like you need a huge dataset to train a decent model?

In my experience, using a focused set of just 500 high quality images with a tool like PyTorch and a lot of data augmentation can get you 85% of the way there for a basic classification task, which is plenty for a lot of real world uses.

3 comments

3 Comments

mary_schmidt3mo ago

Totally agree, it's like this weird myth in the air. I built a pretty good tool to sort my own photos (cats vs. dogs, you know) with maybe 400 pictures and a ton of flips and color shifts. It wasn't perfect, but it did the job for my messy personal library. People get hung up on these massive numbers when a small, clean set you really understand can work wonders.

mary_schmidt1mo ago

Is it just me, or does my cat/dog sorter have a weird obsession with labeling my lawnmower as a dog? It picked up the grass clippings as fur or something, I swear. But hey, it was right about 9 out of 10 fluffballs, which is more than I can say for my ability to find my keys most days. You're spot on about the small, clean set - mine only knew 400 photos, but it knew them like the back of my hand. Sometimes I think we overcomplicate things because it sounds smarter than just saying 'I got good results with what I had'.

blair3963mo ago

Yeah but @mary_schmidt, is a cat/dog sorter really that big a deal... it's just organizing photos.

-1