Learning to let go...just a little...

So we have neighbors, and they have kids. They are older than mine and they like to play with mine. That should be ok, right? I'm having a problem though. I'm so not used to having somebody coming around all the time asking if my kids can come play! I feel a little violated! Maybe just annoyed that someone is coming into our world. Don't get me wrong, I want my kids to have friends, but I am used to organized play dates. I grew up in a farming community where most of my friends live a ways away. You couldn't just walk there. Every time we wanted to play, we had to beg and plead with our parents to make it so. So this whole neighborhood kids thing is so foreign to me. Dustin doesn't think its that big of a deal, though he is probably used to it growing up with friends just around the block to play with all the time. And I'm sure its good for them in some aspects, but I just can't help but want to shelter my kids from that right now. I guess its because the kids are older, I don't know the parents except by first name "We're you neighbor" kind of thing, and they pretty much run all over the neighborhood and I want the kids to stay close.(I mean they are 5 and 3, right?) They act different too when they've been around them. A little more mouthy and lippy and Gracie, especially, is alot more obnoxious than usual. So how do I balance this playing with random friends, which I know helps build friendships in the future, with wanting my kids to be at home where I can supervise them, their mouths and their attitudes? Suggestions?
I have been limiting the time they get to play with these friends. For instance, after school (we are done at 1:45p.m.) they get to play till 4 p.m. This is with friends or at home. Then they have to come inside and they work on homework, chores and generally stay inside the house and get ready for our night time schedule.(The kids are in bed between 7 and 7:30) Then on Sundays, its family day so no friends. And they have some activities and planned play dates throughout the week to break it up. It seems to work and they don't beg to go out, so hopefully this helps. I'm trying not to let it bother me but its hard. I had a tough time letting her go to Kindergarten, let alone run amok with neighborhood kids! And they are so little, I'm not ready to let them grow up and want to be with their friends all the time! I want my babies at home! I'm a little selfish and I know I'm a bit of a hermit, but I really like our little family and we usually have a pretty fun time together! The kids do get on my nerves once in a while, so this going outside to play thing does help some afternoons. But sometimes, I just wish the neighborhood kids would just go away for awhile!