You only need to track the stuff with a GP cost; everything else your can use a focus for or assume it’s in a component pouch of you don’t use a focus. The GP cost items, yeah, I definitely track. Also the costs in time and ink (ie gold) for copying spells into your spellbook. My wizard of constantly broke, and the rest of the party have thousands of gold each.

I do use the materials when describing my spell casting, which I only describe if it feels pivotal.

Like insect plague needs fat, sugar and grains, so my cleric hocks a big loogie of powdered whale blubber sashimi he’s been chewing at the center of the radius of the spell.

You’re telling me that your component pouch has a 5,000 gp diamond? 🤣

My group just assumed all things that are not consumed by the spell and don’t have an explicit cost were just in a standard-issue component pouch.