How long should i leave fish food in for?

Every morning, i feed my one goldfish a couple of tetra flakes. When i am about to feed my fish at night, some of the flakes are still there but in smaller pieces. Should i take the flakes out or leave them in there for my fish to eat?