Will my files get corrupt if I will not use safety remove?

Hi,
Whenever I try to safety remove my flash disk I get an error message like “The device ‘Generic Volume’ cannot be stopped because a program is still accessing it”. But there are no programs accessing the drive. Sometimes it makes me troublesome that I just remove it on my own. What should I do to solve this problem?
Will my files get corrupt if I will not use safety remove?
