Firewalls protect a host by monitoring network packets and attempting to identify good vs. bad traffic. A complement program to firewalls is Host Intrusion Prevention Systems (HIPS). HIPS works to protect a host by monitoring applications that execute. HIPS tries to look at what the program does, either by intercepting system calls or watching packets or other system activity. These may be rule based or may assign scores for certain activity.
There is a lot of variety in the quality of these products, make sure you research them well and evaluate them extensively.