Well, it's not really "legal" but it's also something people don't complain about unless they are dicks.Just make sure that you're actually allowed to drive them around your neighborhood. There was a giant blow up on my neighborhood message board a few years ago, when someone posted that they saw some 9-year old girl driving a golf cart around the neighborhood with about 10 kids piled into it, while she talked on her cell phone. She was like a little soccer mom in training. Anyway, it came to light that it was illegal to drive golf carts on public roads, which our neighborhood technically was, and that her parents were going to be liable for any accident she was in, regardless of the circumstances. The girl's parents were kind of pissed... at the guy who reported it.
On the other hand, maybe you live in a golf course community and everybody drives them around. In which case, it looked like fun.
I am going to check with our HOA about it though as this is a concern of mine.
Edit: I just called the MVa and they said there is no law against golf carts except that you can't take them on a main road where the speed limit is 25mph or more.
Otherwise, there is nothing people can do about it unless you are causing some issue.