Lost,
When younger son was home and had girls over and at first I was "worried" about what they'd think about me when I tried to set down rules with my son who disrespected us at every turn. At one point there was a naked girl during the middle of the night trapesing from his bedroom to the bathroom! I realized I need not worry whether or not these girls will have hurt feelings as they had no respect for my home either. Often times they engaged in smoking pot with son in his room and at one point had my ex husband not gone into check on them for fear of invading his privacy they would have died. You see they had been smoking pot in the bedroom (unbenounced to me & carelessly) and smoldering embers started a fire on some of his cloths on the floor. When my ex opened up the door, the room was encompassed with smoke. The smoke alarms outside of his room weren't going off yet because it was still contained in his room with the door shut. (Unknown by us he had removed the smoke alarm from his room so he could smoke in there). Eventually the smoke would have escaped under the door, however, that may have been too late and died from smoke inhalation. They both had no clue what was happened because they were stoned and sleeping. That was a scary night for sure.
I say as soon as you can set and keep boundaries, do it. It is hard when they are rude and you fear because of their tempermant and lack of concern for your dignity and respect. It WILL NOT just get better. I promise you. You will have to strum up courage to stand against him and his ways. He will continue to PUSH and PUSH at those boundaries until he's walking all over you if you don't get a handle on it now.