How do I sign out of iMessage on my Mac only?

On the Mac, open Messages > Settings (Preferences on older macOS) > iMessage tab > Sign Out. That only signs out the Mac, your phone isn't touched. Also on your iPhone go to Settings > Messages > Text Message Forwarding and turn the iMac off, otherwise normal green texts can still show up there. If the family shares one login on that iMac, giving them their own user account is cleaner long term.