Yeah you are right. I know of someone who had a video ranking page one of Google for the term make money online and in 2 months he hit 200k views. The next thing he new his video was removed for violations and he contacted youtube and they said the video has been flagged to many times. This is a HUGE problem. But then again, a website owner faces the same risk.
Someone can easily buy 10,000+ backlinks to a website ranking on page 1 and KILL their ranking position because of those backlinks so its very risky when making and uploading YT videos!