In Fallout New Vegas, why does Caesars Legion keep attacking me?

I got invited by that dude on the strip and I went to Cottonwood Cove with No Companions, my reputation is neutral, and i was wearing legion veteran armor and a veteran helmet. Now that ive entered the fort i walked to the area with Caesars tent in it, and EVERY time i get to the tent They turn hostile. Whats going on?