Chris's Mediocre Obfuscation Script (CMOScript)

The goods

If you just want to download the latest source or try the live CGI version of CMOScript, here you go:

Download link: http://sandgnat.com/cmos/obfu.pl. (latest version)

Examples (try it out -- live CGI rule generator) here: http://sandgnat.com/cmos/cmos.jsp

Otherwise, read on!

Introduction

Users don't want spam. Mail admins will put filters in place to attempt to reduce the spam so their users are happy. SpamAssassin is a filter that uses a cumulative score. Simple word rules such as adding a score when the word "viagra" appears in the subject line or body of a message has been quite effective in filtering viagra spam. However, spammers are sneaky fellers. They will do anything they can to get their spam into your inbox. One tactic they use to bypass word based rules/filters is obfuscation.

While there are many different kinds of obfuscation (HTML comment tags, small fonts, etc... see Jennifer's Rules and MaskedWordList for more), let's focus on the following three forms of word obfuscation:

• Replacing of characters with one or more characters that are visually similar: Viagra => Vi@gra (or) Viagra => \/iagra
• Gappy text; addition of "filler" characters: VIAGRA => V-I-A-G-R-A (or) Viagra => v.ia.gra
• Missing vowels: vicodin -> vicodn

All three of these methods rely on humans ability to recognize the original word regardless of extra, missing, or replaced characters.

SpamAssassin rules can be written that match these obfuscation techniques. For instance, one day you receive a spam with "v1codin" in the subject line. Easy, you think, and write a rule...

header MY_VICODIN Subject =~ /v[i1]codin/i

... and life is good. However, the next day your vicodin-peddling chum sends you another spam with a slightly different "v|codin" in the subject line. Doh. Let's get smarter...

header MY_VICODIN Subject =~ /v[i1l]c[o0]d[i1l]n/i

... that'll keep 'em at bay. However, the next day... DOH! Now they're coming in as "V_Ì+Ç+0-D_Ï-Ñ"!!! The low-life scum! GRRRR!!!! You hate playing whack-a-mole with your rules, but how can you feasibly write a rule that matches all the possible permutations? (Ok, so maybe this example is a little far fetched, but they really do use these tactics)

This page details my attempt at creating a rule generation script for obfuscated text that will catch many of the permutations of the obfuscation techniques above.

I will call it CMOScript (See-Moh-Script) because I'm getting sick of typing "The script".

Script background

ChrissMediocreObfuScript (CMOScript) is a script written in perl that will generate a rules file consisting of rules that match obfuscated text. The primary goal (duh!) is to match as much spam as possible with as few false positives as possible.

This thing sort of evolved as I was writing it. My spam corpus contains mostly smut spam. The default SpamAssassin ruleset wasn't catching many of the obfu "naughty words". Originally, the script was intended to be run on rules files from the default distribution, such as 20_porn.cf... however after running stats, I found I wasn't doing much better than the original ruleset.

Now, I create a simple rules file with very short rules that match the naughty words. I use CMOScript to generate a complex obfuscation detecting rules file. The hit rate on my obfuscation detecting smut rules are pretty decent for my corpus.

Instructions

• Manually create a "source" rules file with either body or Subject header rules. Keep these rules simple and do not attempt to detect obfuscation.
• Run CMOScript on that source file to generate an obfuscated rules file.
• Add the destination rules file to your configuration (Do not add the source rules file).
• Update and redeploy the destination rules file (re-run the script) whenever you modify the source rules file.

Here's how I actually use it in my environment:

user1@ns1:~$ ./bin/obfu.pl -o /etc/spamassassin/local_pron.cf.src > local_pron.cf
user1@ns1:~$ sudo chown root.root local_pron.cf
user1@ns1:~$ sudo mv local_pron.cf /etc/spamassassin/local_pron.cf

The first line will generate the destination rules file from the source rules file and place it in my home directory. The second line changes the ownership of the destination file to root. The third line places the destination rules file in the appropriate place for my configuration.

Usage:  obfu.pl [arguments] [source rules file] > [destination rules file]
Arguments:      -o Create rules that ONLY match if text is obfuscated
                -g Don't use gappy text obfuscation check. (faster matches)
                -s Use simple gappy text pattern (faster matches) (can't be used with -g)
                -m <n> Allow n characters for gappy text match (more lenient matches: catches v..iag.ra)
                   (probably causes more false positives)
                -w Don't use special gap for short words
                   (disabling this catches more spam, but probably causes more false positives)
                -u Don't check for (unicode) html entities/entity ranges (faster matches)
                -h Output high ASCII characters directly (greater than 127; don't use \xFF in rules)
                -v Make vowels matches optional
                   (Very experimental-NOT RECOMMENDED when using gappy text!)
                -D Print debug information (as comments in generated rules file)

How it works

CMOScript operates either on a pipe or takes a filename arg, like many standard unix tools (including SpamAssassin). You input your source rules file to the script and it spits out an obfuscation-matching rules file which you can then pipe to a file in your /etc/spamassassin directory, or whatever.

SpamAssassin body and header rules are extracted from the source rules file (As of this writing, Subject is the only header rule supported). Each rule is broken apart into tokens. For each token that is a letter of the alphabet such as 'a', a replacement pattern such as: [a4\*\@\xC0-\xC5\xAA\xE0-\xE5] is looked up and inserted into the target rule. As each source rule from your rules file is converted to an obfuscation detecting rule, it is renamed with a prefix of "LOCAL_OBFU_".

(Optional, Default) G*A*P filler is inserted between each simple character token.

(Optional) The rule is made to ignore non-obfuscated versions of the word. This can be useful, for instance, to match commonly obfuscated words that are not always spammy. Example: your Mother may tell you she just "refinanced" her "mortgage", but a spammer might tell you you need to "R.efin.an.ce" your "M0rtg@ge". You can flip a switch and generate rules that match the latter but not the former.

(Optional) All vowels are made "optional characters" via the "?" regexp modifier (This is still experimental. It seems to generate too many false positive hits when used in conjunction with gappy text detection).

Tips and Tricks

• Test your rules on a corpus for false positives. Add any false positives found to the source rules as 0-width negative assertions.
  • Example: body PRICES /(?!price \$)\bprices\b/ # This should find "GRE4T PR1CES" but not "price $" as in "The price $25.00 is for a limited time only".
• Arrange your source rules into separate files.
  • Each file should contain rules you want processed in the same manner.
  • Write a short shell script that executes obfu.pl on each file, using the options that make sense for the rules in the file.
    • For example, put your definite spam-sign words in one file and use the default command line options on that file.
    • Put your everyday english words that are frequently obfuscated (eg: CL1CK H3R3) in another file and use the -o (match obfu only) option on that file.
• If you have source rules that are very small (3 characters or less), gap checking for those rules will use a very restricted gap pattern. This is an attempt to reduce false positives that will happen on random binary data, such as image attachments.
• In your source rules file, if you put the rule declaration before the matching score/describe lines, the script will find and rename the score/describe lines.
• CMOScript automatically adds the /i (case insensitive) modifier, if you didn't include it in the source rule.
• Character classes already in the source rule are not obfuscated.
  • Do not attempt to perform your own obfuscation checks (don't do: "/c[a@]sh/i"; do: "/cash/i").
  • Instead of using character classes in your source rules file, you can use "(?:)". (don't do: "/m[ou]ney/i"; do: "/m(?:o|u)ney/i")
  • You can use this to your advantage: The source rule "/\btit\b/i" will match on the phrase "Isn't it?", but the source rule "/(?<![n]['])\btit\b/i" will not (If you don't grok perl regexp, it performs a negative lookbehind for "n'").
• The generated rules are probably very SLOW. It is likely prudent to use them somewhat sparingly. (Anyone have any data to back this up?) I have about 300 of these rules in my system, but it is a personal email server. Always test your rules, including performance!

Misc Info

• CMOScript is public domain. This means you can use it however, whenever, wherever you like. You are encouraged and expected to modify it. In the spirit of community, I ask (but obviously can't require) that you contribute your modifications back to the wiki page at ChrissMediocreObfuScript or contact me (ChrisThielen).
• I'm a Java programmer pretending to know how to write Perl (fix it if you can!).
• Works for me, YMMV! ;)

Wishes/Suggestions

Add yours to this list:

• Add wishes/suggestions here

Known Bugs

Feel free to add bugs to this list:

• Some replacement characters only look correct when they're either upper or lower case. You may see some false positive matches that aren't human readable as the non-obfuscated word.
• False positives occur on random binary data such as attached images. This mostly happens with very short source rules (3 characters or fewer)
• If your source rule isn't trivial, the script isn't particularly good about determining when and when not to add G*A*P filler patterns. eg: a rule generated from /sexy?/ will match "5.E.X#" instead of just "5.E.X".
• HTML Entities that are converted to 3-byte unicode characters do not currently work. I've commented them out in the source file. I am planning on making the unicode handling code more robust in the future.

Old Bugs

• Word boundaries at the beginning and end of the word are broken. A simple regexp example: "\b[a@]ss\b" does NOT match " @ss " as you would expect. The \b is looking for \w\W or \W\w before the "a" but the " @" makes that impossible.
  • Update 2003/12/01: This is fixed in version 1g by matching word boundaries only in the cases where the following character is a word. -- ChrisThielen
• The G*A*P filler pattern is very grabby (simply anything that's not a "word character" (\w)). In my corpus, this generates some matches on binary data, such as picture attachments. I also saw this match on typical headers: "X-Spam-Checker-Version: SpamAssassin 2.53<cr><lf>X-Spam-Level....". See the match? "53<\n>X"
  • Update 2003/12/01: This is changed in version 1g. The gap token is now a very Very VERY large character class. Let me know how it works (feedback is appreciated). Use the command line switch -s to revert to the (older) simple gap method. -- ChrisThielen
  • Update 2003/12/12: I've removed this bug and replaced it with the more general: Binary data false positive. It's not just the gap pattern that is culprit to matching random binary data. Short patterns in general seem to be.

TODO

• Add support for 3-byte (or more) unicode characters.

Version History: ChrissMediocreObfuScriptVersionHist

Courtesy of ChrisThielen

Feedback