Billy the House Goat?!

We got a cute little bottle fed Billy Goat for the boys about a week ago, who decided to name him "Billy". "Billy" the Billy Goat loves the kids, he follows them every where he can, and recently that means into the house. Billy is a bottle fed goat so we must bottle feed him several times a day, I made the mistake one morning of letting Billy come in the house to feed, and that was pretty much the beginning of the end. Now every chance he gets, Billy is running into the house and the kids seem to think he belongs inside with them,and so every chance they get they let him inside. Trying to convince my 5yr & 3 yr old that goats are not house pets is not going so well.So here I am every time I turn around putting the goat back out side,and when I do he sits at the door screaming to come inside. I don't know how many of you have ever heard a baby goat cry, but it's not pretty. It's bad enough that he follows me EVERY WHERE when I am outside, but to have him under my feet inside is driving me NUTZ! I have to admit it though seeing him curled up with the boys on the floor watching TV is pretty darn cute, but just not cute enough to clean up goat pee.