($Id: CHANGELOG.sanitizer,v 1.94 2006/01/02 03:18:56 bre Exp $)

NOTE: Sanitizer development is for the most part sponsored by FRISK
      Software International, http://www.f-prot.com/.  Please consider
      buying their anti-virus products to show your appreciation.

Revision 1.75:     (January 02, 2006)

    Added code to recognize the most common/important file formats based
    on actual file contents, not just file name and MIME-type.  Added
    magic to detect WMF files, to allow reliable blacklisting of said
    files, see http://isc.sans.org/diary.php?storyid=994 for info.

    Added generic code to detect when people try to disguise 
    non-JPEG/GIF/PNG as such files and defang such attachments.

    Removed the references from the HTML Cleaner's output, the owners of
    the linked web sites were unhappy because their URLs were being
    associated with spam as a result of being in Anomy's verbose logs.

Revision 1.74:     (August 05, 2005)

    Fixed a bug where disinfection wouldn't result in the modification
    count of a message being incremented.  This didn't matter to users of 
    sanitizer.pl, however some 3rd party systems relied on the modification 
    count to determine whether to use the Sanitizer's output or not.  This
    is a critical fix for those systems.

Revision 1.73:     (August 05, 2005)

    Fixed a bug in MIME parser when encountering junk headers at the
    very beginning of a new MIME part.

    Cleaned up some of the test cases for more recent versions of F-Prot
    AntiVirus, added a version check to the F-Prot regression test.

Revision 1.72:     (July 10, 2005)

    Fixed bug in code which detects Date: header buffer overflows, it was
    false-alarming on Yahoo DomainKeys headers.

    Lengthened maximum word-length in headers from 196 to 256 bytes, again
    to decrease the odds that we'll break DomainKeys.

    Added sanity checks to configuration parser, to make sure that settings
    such as msg_defanged and msg_blacklisted, which get used within message
    headers contain only valid characters (0-9, A-Z, a-z and -).

    Test-cases sanitizer.uu-rfc822 and sanitizer.logging are updated.

Revision 1.71:     (May 25, 2005)

    Fixed minor bug in quoted-printable encoding, as reported by Michal 
    Weinfurtner <weinfurt (at) karneval.cz>.
    
    Fixed crashing behavior when multiple Content-Transfer-Encoding headers
    were present in the same message part.

    Added mailblogger.pl, to the distribution.  This program has nothing
    to do with security, but uses the MIMEStream parser to extract images
    from e-mail and can subsequently generate thumbnails and re-post both
    text and images to a web-site, to implement e-mail->www gateway 
    functionality.  I use it to blog from my cell phone. :-)

Revision 1.70:     (January 4, 2005)

    Raised limits on max header size from approximately 64k to 256k.

    Made error reporting 

    Added support for new F-Prot Daemon result codes to Sanitizer::FProt.

Revision 1.69:     (September 2, 2004)

    Added zip_policy.pl from Advosys (http://advosys.ca/) to the contrib/
    directory, after being invited to do so by Derrick Webber of Advosys.

    Added sanitizer.procmail ruleset to contrib/, illustrating how to 
    implement a quarantine and add custom headers to infected e-mails.

    Fixed priority bug in filename detection code, which would in some 
    cases give higher priority to Content-IDs than it gave to the MIME
    filename attributes.

    Made the file-name/MIME-type sanity checks configurable (default on)
    via. the feat_sane_names variable.  Set to 0 to disable.

    Wrote very simple HTTP client for communication with F-Prot daemon, 
    thus eliminating the dependancy on LWP::Simple.

    Fixed incorrect changelog entry below (in the entry for rev 1.57 the 
    word "ScanFile" was used where it should have said "FileScan"), and 
    added support for scripts which want to pass the name of a detected
    infection using the a line "Anomy-FileScan-VirusName: blah" like.  
    This makes the following new variables available to the file replacement
    tempalte:

       %VIRUSNAME    - Propogated from Anomy-FileScan-VirusName
       %SUMMARY      - Propogated from Anomy-FileScan-Summary
       %DESCRIPTION  - Propogated from Anomy-FileScan-Description

    This corrects problems, implements and expands on suggestions 
    (posted here http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=235352)
    by Derrick Hudson (dman at dman13.dyndns.org).

Revision 1.68:     (May 7, 2004)

    Added system_io_file variable to allow plugging in of custom
    replacements for the IO::File module, to facilitate internal FRISK 
    development.

    Fixed a problem with the mime-type auto-detection code which would
    corrupt certain messages when feat_log_after was enabled.  This 
    probably also have caused problems in other cases, but so far none have
    been reported.

    Include the TNEF hooks in Sanitizer in default distribution and made 
    inclusion of Anomy::TNEFStream "lazy" to save cycles in one-shot modes.
    Note that the Anomy::TNEFStream modules still isn't distributed by 
    default.

    Tuned the MIME parser to catch more of the exploits illustrated on 
    http://testvirus.org/.  Also fixed a bug in the position counting.  These
    two changes combined effect almost all of the test cases (lines containing
    pos= and MIME info almost all change).

    Added the following options to configure the HTML cleaner (all are off
    by default):

       feat_html_noexe     Disallow links to executables
       feat_html_unknown   Allow unknown HTML tags
       feat_html_paranoid  Paranoid HTML Cleaner mode, bans all src= links
                           and enables feat_html_noexe paranoia as well.

Revision 1.67:     (March 23, 2004)

    Added code to decrease the odds that attachments with content-IDs
    ending in ".com" get mistakenly treated as executables.

    Tweaked MIME parsing to catch a few more odd virus-generated messages.

    Obfuscated some of the testcase results using simple rot13 encoding,
    so the Anomy .tar.gz file wouldn't be flagged as "infected" by various
    virus scanners.

Revision 1.66:     (January 12, 2004)

    Fixed an endless loop caused by an error in the uuencode detection
    routine.  Thanks to Paul M. Hirsch for reporting this one.

Revision 1.65:     (December 17, 2003)

    Fixed a bug with attachment deletion, in some cases things wouldn't
    actually get removed, but would be appended after the "attachment
    removed" message.

Revision 1.64:     (December 09, 2003)

    Modified handling of quoted-printable encoded messages in order to
    improve security and avoid the often-reported "PDF corruption
    problem" caused by silly mail clients which QP-encode binary files.

    Handling of broken MIME created by a couple in-the-wild viruses has
    also been fixed.

    This is a highly experimental release, use with care!

Revision 1.63:	   (July 08, 2003)

    Fixed a bug in HTML cleaner to do with XML-style tag termination.

    Improved the MIME parser to handle better the obfuscated MIME created
    by the Ronoper worm.

Revision 1.62:     (June 19, 2003)

    Updated HTMLCleaner to avoid endless loop in s/// on Perl 5.00503.

    Updated HTMLClenaer and Sanitizer to allow users to specify in the
    configuration file certain replacement tags or replacement
    attribute names to use in place of "DEFANGED_".  Added the default
    replacement tag <p> for defanged <div> tags.  So now, defanged
    <div> tags will be replaced with <p><DEFANGED_div> instead of just
    <DEFANGED_div> The syntax for configuring the HTML cleaner is
    rather ugly, and may be changed in the future.

    Updated testall.sh script to warn the user if he attempts to use the
    Sanitizer in an UTF-8 enabled environment (LANG=*.UTF8) and added a
    instructions regarding unicode issues in the file UNICODE.TXT.

    Test cases have been updated to work properly on FreeBSD 4.6.2 with
    Perl 5.00503, and on Unicode enabled RedHat 8 or 9 machines.

Revision 1.61:     (unreleased)

    Added recognition of the Content-Description and Content-ID headers,
    made the file name policy code treat them as if they might contain
    filenames as well in some cases.

    Made the MIME parser tolerate spaces in attribute definitions (e.g.
    Filename = "Foo.exe" instead of Filename="Foo.exe").  Also updated
    the MIME parser so "type=..." attributes would be interpreted 
    separately from the Content-Type.

    Minor fixes to filename policy code.
    
    Minor fixes to the testcases.
    
Revision 1.60:     (May 28, 2003)

    Minor update to MIME type checking rules, to allow more legal MIME
    types.  
    
    Made the multipart detection code less aggressive, in small text
    messages it would mistake common ascii-graphic signatures for message
    boundaries and mess up the parsing quite badly.

    Made the filename policy code check ALL possible file names against
    each rule, instead of just checking the "default" one.  If
    feat_mime_files is set, then the default file-name for that mime
    type will be checked as well.  This is a major improvement to
    security, but requires that filename rules are ordered so that
    that all DROP/DEFANG/MANGLE rules precede any ACCEPT rules.

    Made the sanitizer read /etc/mime.types (if it exists) to generate a
    more complete list of default filenames for unnamed parts.

Revision 1.59:     (May 9, 2003)

    Fixed detection and handling of GPG/PGP messages.  This corrects the
    broken PGP handling introduced by revision 1.58, as reported by
    Rick Johnson (rjohnson at medata dot com).

Revision 1.58:     (May 8, 2003)

    Added the "try" prefix for use within configuration files which
    include other configuration files.  This makes loading of the file
    optional - if it doesn't exist or is unreadable the Sanitizer simply
    ignores it.

    Minor change to the sanitizer.appledouble check, as suggested by
    Dag Nummedal (nummedal at ed.ntnu.no).

    Made text/html parts exempt from feat_force_name behaviore, to solve
    unfortunate display problems with Outlook clients as reported by David
    Santinoli (david at santinoli . com).  Also fixed a minor bug in the
    feat_force_name behavior.

    Made minor improvements to header parsing code, to handle the broken
    MIME generated by a number of viruses out there.

    Changed the header rewriting code to quote attribute values only when
    they contain non-alphanumeric characters.  This is done because some
    versions of Eudora apparently don't like format="flowed", but do like
    format=flowed.

    Fixed a minor bug in the HTML cleaner, where it was missing some 
    invalid attribute definitions which Explorer will accept.  Thanks to
    Paul Wallingford (paul at cybergestalt dot net) for the report.

    Added the Advosys tnef2multipart.pl script to the contrib/ directory.

    Added caching to the QP encode/decode routines, to decrease the number
    of times that a different QP encoding strategy is chosen for a given
    string.  This won't solve QP-related encoding/decoding problems
    entirely, but it may decrease their frequency.

    More changes to MIME parser, to handle broken messages created by the
    Bridex worm and a few other odd massmailer worms/viruses.  This, again,
    modifies white-space and log offsets in the test-cases.

Revision 1.57:     (November 14, 2002)

    Fixed a MIME header bug introduced in revision 1.56 by the Bugbear
    fix. The bug was causing certain MIME headers to be corrupted,
    primarily effecting recipeints of signed or encripted mail.
    Thanks to Ton Vandepoel (tom.vandepoel at be.ubizen.com) for the
    bug report.

    Augumented the config file syntax to allow any directive to be
    prefixed with "before X" or "after X ", where X is a valid Unix
    timestamp.

Revision 1.56:     (October 22, 2002)

    Modified the MIME attribute parser slightly, so it will detect the
    entire filename as sent by Bugbear and other viruses, in spite of 
    those names not conforming to the MIME standard.
    
    Added more detection of potential security abuses based on invalid
    RFC822 comments.

    Added expiramental support for more detailed communication between
    scanners and the Sanitizer itself.  STDOUT from file scanners will
    be scanned for the following tokens:
    
      Anomy-FileScan-Result: CODE
      Anomy-FileScan-Summary: summary
      Anomy-FileScan-Description: Text description of result
      Anomy-FileScan-NewName: newname.bla
      Anomy-FileScan-NewFile: /path/to/new/attachment/data
      Anomy-FileScan-NewType: MIMETYPE
      Anomy-FileScan-NewEnc: ENCODING

    The FileScan-Result code overrides the exit code of the scan program,
    and the summary and description fields override the defaults built into
    the sanitizer.  As a side effect of providing a new name or new data
    file, the part encoding will be forced to Base64 - unless some other
    encoding is explicitly requested.
    
    A sample scanner script which uses some of these features is the
    zip_script in the contrib/ directory (a "scanner" which will encapsulate
    all "scanned" attachments in a ZIP file).

Revision 1.55:     (October 08, 2002)

    Fixed dependancies in sanitizer.pl, so it no longer requires LWP::Simple
    unless people are actually using the FProt daemon scanner.

    Modified the HTMLCleaner rules to avoid a rendering bug in Eudora's
    built-in HTML viewer, which was triggered primarily by defanging of
    Outlook's HTML messages.

Revision 1.54:     (September 18, 2002)

    Added the hcp:// protocol to the list of banned href= and src=
    destinations in HTMLCleaner.pm, for reasons discussed here:
    http://online.securityfocus.com/archive/1/287482/2002-08-15/2002-08-21/0
    
    Tightened security on href= attributes in general.

Revision 1.53:     (September 17, 2002)

    Fixed a minor bug to do with F-Prot daemon support in 1.52.

Revision 1.52:     (unreleased)

    Added built-in support for F-Prot Antivirus for Linux, both the
    small business (command line) and enterprise (daemon) versions.  The
    command line client is auto-detected and used if present, use of the
    daemon can be requested by invoking sanitizer.pl with "-fprotd" as
    the first argument.

    Enterprise customers will be able to enable automatic disinfection
    of incoming messages, by adding the "-disinf" parameter to the 
    fprotd command line in the file_list_2 scanner definition, and will
    see the name of the detected threat in the Sanitizer logs.

    The default policy for infected content is to mangle the filename and
    MIME type, but still pass the data on to the user.  Upgrading this to 
    a "drop" or "save" policy is recommended.

    Note: Use of F-Prot on machines where it is available be manually
    disabled in two ways: either invoke sanitizer.pl with "-nofprot" as the
    first argument or redefine file_list_2.

Revision 1.51:     (unreleased)

    Created feat_no_partial (enabled by default), which defangs any incoming
    message/partial messages, to address the problems described in
    http://www.securiteam.com/securitynews/5YP0A0K8CM.html

    Added support treating the first part of a message/partial message the
    same was as if it were message/rfc822.  This should catch any security
    risks present in the *first part*, but not any subsequent parts.

    Fixed a bunch of other minor things.

Revision 1.50:     (unreleased)

    Improved boundary guessing routine and header parser to deal with 
    still more non-RFC compliant messages.  This still needs work.

    Fixed the problem with $m being in the range 0-11 instead of 1-12, when
    used in filename templates. Thanks to Paulius Bulotas (paulius at
    kaktusas.org) for the report.

    Fixed minor logging bug to do with non-MIME messages and
    feat_log_inline=2.
    
    Added a few tags and attributes common to email to the HTML cleaner, 
    to lower the noise level.

    Added GNU GPL, GNU LGPL and Artistic Licenses to the distribution (in
    the file COPYING).  This bloats the distribution somewhat, but is
    necessary for strict compliance with the GNU licenses.

    Added module for interfacing Anomy with the daemon version of F-Prot
    Antivirus for Linux, which is significantly faster than the command-line
    version.  The daemon version will most likely be made available to
    purchasers of enterprise-class licenses for F-Prot Antivirus for Linux.

    Fixed a bug when truncating unusually long MIME fields, as reported by
    Will Day (wd at hpgx.net).
    
    Updated default configuration to recognize blacklisted filenames with
    trailing dots (which are ignored by Windows).

    Added protection against MIME recursion DoS attacks.  The Sanitizer 
    itself was vulnerable to such attacks.  The default maximum allowed 
    recursion level is 20, which should be more than enough.  This value
    can be tweaked by setting the max_mime_depth variable.

    Fixed a problem which occured when re-encoding Base64 encoded
    attachements with odd line lengths.  Thanks to Joerg Lenneis
    <lenneis at wu-wien.ac.at> for the bug report.

Revision 1.49:     (February 15, 2002)

    Fixed a minor white-space related bug in MIME header parsing.
    
    Made the configuration file parser tolerant of Windows/DOS formatted
    text.

    Created a seperate distribution containing only the stuff used by
    the HTML cleaner, for users of David F. Skoll's MIMEDefang program.

    Made minor tweaks to the messages logged by the HTML cleaner, added
    code to prevent re-defanging of HTML tag attributes.

    Implemented a much more elegant fix to the lack-of-trailing-newlines
    buglet fixed in 1.47.  This should make all those extra newlines go
    away again...

    Added "branch" functionality to the file policy mechanism.  By
    appending ^N (where N is a number) to an "unknown" or "warn" policy,
    the policy matcher can be made to branch to a given rule instead of
    evaluating the next rule sequentially.

    Improved UUencoded attachment detection to handle null modes and 
    protect users against the silly Outlook "begin  blah" annoyance, as
    described here: http://www.rodos.net/outlook/#begin

    Made the content header parser deal better with certain RFC-incompliant
    MIME types generated by broken PGP plugins.  Added a feat_fixmime check
    to rewrite the offending headers so they comply with standards.

    Fixed broken file_list_7 in default policy - thanks to Tuomas
    Lukinmaa, <tumu at iki.fi> for bringing this to my attention.

    Added code to defang bare CR characters in message headers, due to 
    the following Outlook bug (enabled by feat_fixmime): 
    http://www.openoffice.nl/special_interest/outlookbug.html 

Revision 1.48:     (January 04, 2002)

    Happy new year!  Updated copyright notices again.

    Improved HTMLCleaner to properly handle STYLE tags which have 
    attributes.  Thanks to Andrew (andrew at ledge.co.za) for pointing 
    this out.

    Explicitly set all temporary files opened to binary mode, to 
    improve portability.  Improved newline handling code a bit more - 
    it now properly handles differing newline conventions within
    embedded/encoded parts of the same message.

Revision 1.47:   (not released)

    Added feat_newlines, to allow people to specify what sort of newlines
    to use in the sanitizer's output.  Default (0) is to use a newline
    convention "autodetected" from the first chunk of data.  Attempted to
    address a number of platform dependant newline issues within the code
    itself - this still needs some work though.

    Added the "warn" policy, which acts just like "unknown" except for
    the fact that it also increments the modification counter.  Added a
    test for this to the filenames test.

    Fixed a buglet to do with a lack of trailing newlines in parts which
    are re-encoded as 8bit instead of Base64 when feat_log_after is in
    use.  This has the side effect of adding newlines to almost all of
    the test cases.

    Updated copyright notice in a few files to mention the year 2001.
    Seemed fitting to fix that before we enter 2002... Thanks to Dave
    Cridland for pointing that out. :-)

    Moved John Hardin's macro scanning code into the module
    Anomy::Sanitizer::MacroScanner, to facilitate sharing of that code
    between different implementations of the sanitizer engine.
    
Revision 1.46:   (skipped)

Revision 1.45:   (December 12, 2001)

    WARNING:  Scoring works again - but not like it used to!
    WARNING:  The default configuration has been updated quite a bit,
              and does some NEW THINGS.  You have been warned.

    Most test cases were modified for this release (I've gotta start
    releasing things more often...).

    Almost complete rewrite of HTML sanitization code, to switch from a 
    default-allow to default-deny strategy.  Primary benefits:

      - Old problem with <style> blocks becoming visible solved.
      - Rudimentary support for simple (safe) style markup.
      - Defangs non-standard HTML introduced by recent MS Office products.
      - CID defanging is more precise.
      - A few safe META and LINK tags are now recognized, fixing (for example)
        problems with charset-definitions.
      - Web-bugs can now be optionally defanged (feat_webbugs).
      - Initial measurements at FRISK indicate that the "false-positive" 
        rate is somewhat lower than before.  Plenty of room for improvement
        though...

    The html_evil_tags and html_javascript configuration variables are no
    longer used by the Sanitizer.

    MIMEStream now guesses boundary strings for multipart/ parts, if no
    boundary string is specified in the header.  The Sanitizer adds the
    missing boundary string to the Content-Type header, if found and
    feat_fixmime=1.

    This release re-introduces ! policies and the score_bad feature,
    although the semantics are slightly different - see the manual for
    details.  The score_panic, score_panic_code and score_bad_code
    variables (and features) have been removed.  For semi-safe backwards
    compatibility, the panic policy acts just like the drop policy.

    The default configuration was dramatically improved (it's now very
    close to my "recommended" configuration, although it obviously lacks
    a proper virus scanner).  It should now be easie to "tweak" the
    config to change policies for classes of files or add a virus
    scanner, without having to copy whole rules.

    Added feat_log_after, which formats messages in such a way as to
    facilitate after-the-fact insertion of logs into messages, without 
    forcing the log process to rewrite the entire message.

    Fixed a bug in the Sanitizer where certain "fixes" made to the message
    headers could get lost if "feat_log_inline = 2" was used.  Also fixed
    minor glitches within MIMEStream.

    Minor tweaks to header encoding routines.

    Added sanity-checks for certain MIME types, which defangs the attachment
    if the name doesn't match the MIME type.  This blocks (amongst other
    things) the audio/x-wav exploit commonly used Nimda, BadTrans and others.

    Fixed an ugly bug to do with handling of nested multiparts.  I'm not
    sure whether this was present in 1.44 or not.
    
    Added incomplete support for RFC2231 MIME Parameter Value and Encoded
    Word Extensions.  This needs more work.
    
    Modified header checks to allow longer individual words within headers
    (the limit was 128 characters, raised it to 196).  Relaxed the checks
    on subject lines quite a bit as well - they were downright irritating.

Revision 1.44:   (October 11, 2001)

    WARNING:  I renamed the def_charset variable to var_def_charset, this
              may break custom configurations.  Sorry!

    Note: this release lacks all scoring code, so if you are using the
          bad_score or panic_score features DON'T UPGRADE.  For this 
          reason the filename test-case will FAIL as released.

    Updated test cases:  

       appledouble - White space.
       base64      - Minor logging change, white space.  Made much
                     longer, to test handling of >8k input.
       force_hdr   - New test case.
       logging     - Partial line fix.
       plugin      - Plugin interface fix, tests new scanner arguments.

    Added defanging of cid: URLs, since it can be used to trick browsers
    into executing .XYZ attachments as .EXE programs.  Also about: URLs
    since they seem to have some pretty suspicious properties as well.

    Fixed a problem reported by Mark Salazar to do with inline HTML logs
    causing the following qmail error:
    "SMTP cannot transfer messages with partial final lines. (#5.6.2)"

    Moved encoded-character decoding to earlier in the HTML defanging
    process, so that an obscured tag like "<SCR&amp;#73;PT>" will be
    properly defanged (thanks to John Hardin for the tip).

    Fixed a brown-paper-bag-bug in the scanner plugin code (policy exit
    code definitions were being ignored).

    Made a few internal modifications to Log.pm to allow global hooks and
    made a couple of changes to the Sanitizer itself to make hooks more 
    useful.

    Fixed an ugly bug in Base64 encoder and the "feat_log_inline = 2" code.
    
    Added the force_headers and force_header_N variables, which allow 
    people to require certain headers to be present in all messages and
    provide default values for each.  This is disabled by default.

    Added %REPLY_TO, %ERRORS_TO and %HEADER(<name>) tags to the format of
    the scanner command line, to allow people to pass more information to
    the scanners than just the file name.  Examples are in the plugin 
    test-case and Sanitizer.pm configuration comments.

    Fixed a bug in the boundary-ambiguity checker, which could cause
    MIME boundaries containing comments to be ignored, depending on the
    case of the "boundary=" part of the attribute.

Revision 1.43:    (September 04, 2001)

    Note: this release lacks all scoring code, so if you are using the
          bad_score or panic_score features DON'T UPGRADE.  For this 
          reason the filename test-case will FAIL as released.

    Updated test cases:  

       rfc822    - search for X-Mac in the .diff file,
       msg-crlf  - removed dependancy on output from "rm"
       bad_html  - added LINK tag test
       logging   - new test case
       ALL       - may have minor diffs in white space or charsets

    Added the LINK tag to the defanged HTML list.  Explained here:
    http://www.whitehatsec.com/labs/advisories/WH-Security_Advisory-08232001.html

    Added FRAMESET to defanged HTML list, it didn't make sense to just
    defang the FRAME tag.

    Finished refactoring Sanitizer.pm so Anomy::Sanitizer is a proper
    OO module with few irritating package globals.  This makes
    embedding the Sanitizer engine into other programs much cleaner.

    Fixed minor bugs to do with filename character mangling and X-Mac-*
    header defanging.  This modifies the rfc822 changelog.
    
    Fixed a bug to do with feat_log_inline=2 and modifications in the top
    level part's headers.

    Fixed file_characters feature so eight-bit characters in the allowed
    character list would actually work.

    Made the charset of embedded messages configurable, via the def_charset
    variable.

Revision 1.42:    (August 21, 2001)

    Note: this release modifies ALL test cases.
    Note: this release lacks all scoring code, so if you are using the
          bad_score or panic_score features DON'T UPGRADE.  For this 
          reason the filename test-case will FAIL as released.

    Changed file_characters so if defined to 0 all characters would be
    allowed (test disabled).

    Added Xavier Roche's check_for_virus script to the contribution
    directory.  His instructions for using it are included as comments at
    the top of the file.

    Added code to MIMEStream.pm to handle RFC822 headers ending with CRLF
    instead of just LF.  Created a new test-case for this.

    Fixed minor bug in MIMEStream to do with default attachment disposition.

    Added the FORM tag to the list of blacklisted HTML tags (see
    discussion on Bugtraq 14-16. August 2001), added two new configuration
    variables, html_evil_tags and html_javascript to allow people to
    customize their HTML defanging a bit more.  See the defaults in
    Anomy/Sanitizer.pm for details.
    
    MAJOR logging code rewrite:

    Created Anomy::Log for creating more detailed, nested logs.

    Added variables feat_log_xml and feat_log_trace, for enabling XML
    formatted logs and detailed traces for debugging respectively. Attached
    and stderr logs are always more detailed than embedded text logs,
    embedded text logs are never XML formatted.

    Modified feat_log_inline=1 behavior, to make it more reliable - it used
    to add parts to multipart/alternative sections which would always be
    invisible in common MUAs.  We don't do this any more, but insert as much
    as is available directly into the text/plain and text/html parts.

    Added new feat_log_inline=2 behavior - in this mode, the Sanitizer will
    guarantee that a visible log can be inserted by forcing the top-level
    message type to multipart/mixed if it is neither multipart/mixed or
    text/*.

Revision 1.41:

    Unreleased, see 1.42 entry for a summary of changes.

Revision 1.40:    (August 7, 2001)

    NEWS:  Sanitizer development is now for the most part sponsored by
           FRISK Software International, http://www.f-prot.com/.  Please
    consider buying their anti-virus products to show your appreciation.

    Moved all functionality from sanitizer.pl to Anomy::Sanitizer.pm.

    Added the RegisterScanner() function to Anomy::Sanitizer, moved John's
    Macro scanner back out of Sanitizer.pm and into sanitizer.pl.

    Fixed a bug to do with case sensitivity in MIME types.  Thanks to 
    Bernd Rothert for the report.
 
    Added the following variables:

      file_characters    - Specify safe characters/ranges for file names.
      sanitizer_log_disp - Content-Disposition: header for inline logs.
      sanitizer_log_type - Content-Type: header for inline logs.

    Made messages more flexible - now arbitrary var_XYZ variables can be 
    defined in the configuration file itself and referenced in msg_ABC
    variables using %XYZ constructs.

    Fixed a bug to do with the "unknown" policy, updated filenames test
    case to check for it in the future.

    Fixed a bug in the MIME header parser, which couldn't handle attribute
    boundaries as generated by Prontomail:  "text/plain ;charset=..." 
    (note the " ;" instead of the usual "; ").

    Made testall script check prerequisites (Perl modules) before running 
    the test themselves, to save newbies a bit of confusion.

    Updated documentation to include Postfix instructions and improved
    sendmail configuration.  Added a contrib/ directory to the distribution
    for storing user-submitted tips and tricks.  Added the simplifier to 
    the distribution as well.

Revisions 1.36-1.39:

    Unreleased, see 1.40 entry for a summary of changes.

Revision 1.35:    (January 31, 2001)

    Testcase change summary:
    
        - Subject related changes are in the filename and appledouble tests
        - A new part, a default name and the force_names features were 
          added to the filename test
        - The bad_html test defangs a new tag
        - The following tests may output different amounts of white space:
          base64, boundary, exchange, forwarded.

    Made "defang" the fallback policy for "save" or "scan" policies when
    creating the temporary file fails for some reason.

    Added the "file_default_name" and "feat_force_name" variables.  The
    former is the file name used for comparing unnamed parts with the
    filename policies, the latter makes the sanitizer add a file name to
    unnamed parts, based on the part's MIME type.  Updated filename testcase
    to test these features.

    Added a rule to truncate overly long Subject lines which might be
    interpreted as file names.  This addresses at least part of the Outlook
    Express 5.5 HTML.dropper bug discussed on Bugtraq on the 17. Jan. 2001.

    Added the LAYER tag to the list of defanged HTML tags.  This is in
    response to the discussions on Bugtraq about how to exploit web-based
    email solutions by using layers to put new buttons/links on top of the
    email program's buttons.

    Added more default file names for different MIME-types.

    Modified multipart-MIME and Base64 code to properly preserve multipart
    pre- and postambles and whitespace near MIME boundaries.

    Fixed a very minor white-space related bug in MIMEStream's header
    parser.

Revision 1.34:    (December 29, 2000)

    WARNING: This release modifies the output of the test cases.  Your .diff
    files should contain lots of MIME headers (Content-Type, etc), but
    almost nothing else except in the uu-rfc822 case where the uuencoded
    output also changes.

    Added the "feat_fixmime" option, which will make the sanitizer try to
    output valid MIME messages even when it's input is invalid.  This will
    convert illegally encoded multipart/* or message/* parts so they are
    legal and legible (8bit encoding), and will fix ambiguous boundary
    strings (see below).  This feature is enabled by default.

    Modified header cleaning routine to leave clean headers on multipart
    parts completely unmodified, like in other parts.  This modifies the
    output of all test cases.

    Fixed bug in Base64 decoding routine.  This could corrupt messages 
    which only had a single Base64-encoded part.  Added a test case.

    Fixed boundary detection code to cope with how different mailers 
    treat RFC822 comments within boundary strings.  Ambiguous boundaries
    are replaced with unambiguous ones, unless feat_fixmime is disabled.
    Added a test case.

Revision 1.33:    (December 14)

    Fixed a bug in how forwarded/uuencoded attachments are handled 
    which could cause the next MIME boundary to get corrupted.  Simplified
    the forwarded message handler a bit, added a test case for it.

Revision 1.32:    (November 30)

    This is the second "finding lots of bugs with test-cases" release.

    Fixed a pretty ugly bug in the inline/uuencoded handler.  It was
    corrupting attachments - everyone should upgrade to fix this problem.

    Fixed a bug in the filename truncation code, made the filename scoring
    more consistant (all matches increment score by one unless ! policy
    modifier is used).

    Improved the testcase framework to allow people to test their own
    configurations against future versions of the sanitizer and added more
    tests.

    Added basic mime-type -> filename mappings, to catch unnamed MS-TNEF
    attachments (winmail.dat).  At the moment very few mime-types are
    recognized, more will be added in the future.

    Stopped using recent (perl 5.005) "substr" syntax.  This should allow
    the sanitizer to work unmodified with older (at least 5.004) versions of
    Perl.

    This revision *may* work-around out-of-memory problems in the
    MIME::Base64 module under Solaris.

Revision 1.31:    (November 24, 2000)

    Made the 'DEFANGED' text used by all sanitization routines user
    configurable.  Also made the strings 'BLACKLISTED' and 'PANIC'
    user-configurable.
    
    Updated the documentation quite a bit.

Revision 1.29, 1.30:    (not released)

    Added basic handling of RFC822 comments within message headers, negating
    possible problems with name="filename.v(comment)bs" attributes slipping
    through.

    Added code to make outputted Base64-encoded data use the same line
    lengths as the input data used, when possible.

    Made output-header encoding code less aggressive.

    Added handling of inline/message/rfc822 parts (uuencoded rfc822
    messages).

    Minor tweaks to HTML sanitizer.

    Added the testcases/ directory, created test cases for some of the
    sanitizer's features (with more to come).

Revision 1.28:    (November 7, 2000)

    Fixed the decoding of MIME-encoded (QP or Base64) encoded headers to be
    more in line with the relevant RFCs.  This fixed a bug where underscores
    ('_') mistakenly got converted to spaces.

    Fixed internal inconsitancies to do with trailing newlines on internal
    representations of individual headers.  (phew!)

    Made the MIME parser ignore artificial white space introduced when
    recombining and decoding long headers that had been encoded and then
    split between lines.  The assumption is that the encoding protects all
    "important" white spaces.  In other cases (where no encoding has been
    done) the "newline, white space" sequence is replaced by a single space.
    This behavior is expiramental, pending a more complete review of the
    relevant RFCs and user feedback.

    Ensured that new data will be properly MIME-encoded if necessary when
    rebuilding message headers after some sort of sanitization has been
    deemed necessary.

    Added protection against Exchange Server 5.5 DOS involving null MIME
    attributes, such as charset="".

    Converted the file README.sanitizer to HTML (sanitizer.html) and added a
    chapter on configuration with qmail, more detailed configuration hints
    and some information for people interested in hacking the code.

Revision 1.27:

    Never released.

Revision 1.26:    (September 15, 2000)

    Fixed a bug which caused HTML to be sanitized in message headers, no
    matter what the configuration said.

    Fixed a problem with inline uuencoded attachments that had spaces or
    other weird characters in their names.  This could lead to
    HTML-defanging of uuencoded stuff, which is bad.
    
    Added protection against empty boundary string attacks against 
    Exchange Server 5.5.  Also fixed a bug in the feat_boundaries 
    feature.

Revision 1.25:    (August 18, 2000)

    Fixed a problem scanning in-line uuencoded HTML attachments.

Revision 1.24:    (August 18, 2000)

    Fixed a bug in the configuration file parser, which would cause
    problems when comments were placed in the same lines as actual
    configuration commands.

Revision 1.23:    (August 17, 2000)

    Fixed a bug in my version of John's macro scanner, which he fixed
    himself in version 1.114 of his sanitizer.  Now PowerPoint files
    should be properly scanned.  Also emulated John's x-mac paranoia,
    so Eudora won't work around file-name mangling.

    Fine-tuned the Date header stuff a little more.

    Minor tweaks to log output.

    Fixed a bug in the configuration file parser, now escape sequences 
    in regular expressions will actually work.

    Added rudimentary instructions to the README about how to use the
    sanitizer to filter in-transit mail on a mail hub, with or without
    procmail.

Revision 1.22:    (July 28, 2000)

    Improved the generic header buffer overflow protection a bit - it now
    mangles long words a bit more unpredictably.

    Fixed a buglet to do with trailing white space on policy definitions,
    and another to do with filename mangling when both "filename" and "name"
    attributes are present in the MIME headers.  Thanks to Robert Forsman
    for the bug report.

Revision 1.21:    (July 22, 2000)

    Added generic protection against buffer overflows based on very long
    words in message headers: any single word over 128 characters in length
    will be split in two.  There is a more strict limit on the Date: header,
    which is known to be exploitable in some versions of Outlook.

    This is the "I drank too much in Eger" release.

Revision 1.20:    (July 13, 2000)

    Made sanitizer exit code policy-definable, with score_bad and
    score_panic configuration variables, the panic policy and the !
    policy suffix.  The README contains more details.

    This is the "Going back-and-forth in Polish trains" release.

Revision 1.19:    (June 12, 2000)

    Misc. minor bugfixes.
    
    Added expiramental support for scanning forwarded messages as if 
    they were message/RFC822 parts.

    Fixed the HTML sanitizer so it wouldn't break email addresses which
    look like HTML tags: <like@this>

    Fixed RFC compliance of Base64 decoder - it will now properly ignore
    any invalid input, including newlines.  Unfortunately, this needs yet 
    another buffer, and slows us down when decoding such attachments.

Revision 1.18:    (May 31, 2000)

    Fixed the built in macro scanner.

    Avoided unneeded disk I/O when the drop policy is mandated for a 
    given set of files.

    Added more useful date expansions to the filename template used by 
    the virus scanning code and the save/drop policies.

Revision 1.17:    (May 30, 2000)

    Fixed bugs in the attachment dropping/saving code.  It should work
    now!

    Implemented the "unknown" policy, which I accidentally documented before
    the 1.16 release (instead of after).  It's untested though.

Revisions 1.13-1.16:

    NOTE:  This is release includes a great many changes, it almost 
    certainly has bugs!!

    Misc. internal code cleanups.
    
    Implemented configuration system.
    Added support for external virus scanners and policies.

    Initial comparison of my implementation with the relevant RFCs.
    See the README for information about how I'm doing...

Revision 1.12:

    Re-merged the text/plain and text/html sanitizers, after a brief
    discussion with John Hardin.

    Added HTML sanitization of part headers.

    Fixed a bug in the uuencoded-part scanner (MIMEStream.pm).
    Made HTML-defanging log "safe".
    
    Implemented logging to stderr, activated it by default.

Revision 1.11:

    Attempted to address some rather exotic problems with the HTML sanitizer
    and inline uuencoded attachments in HTML parts.  Are such combinations
    even possible?

    Fixed logging bug in HTML sanitizer.

    Will no longer switch to HTML sanitization mode when HTML appears in the
    middle of a text/plain part.  Hopefully mailers are at least sane enough
    for this to be safe.  The sanitizer was getting confused by discussions
    of HTML exploit techniques on Bugtraq. ;-)  We still switch if that sort 
    of stuff appears at the very top of the part though.

Revision 1.10:

    Replaced $buggies with $bugscore, which the scans increment by
    various amounts.  Latter versions may discard or bounce messages if
    the score exceeds some arbitrary value, e.g. 100.

    Fixed a bug in the long tag splitting code in HTML sanitizer.
    Reorganized HTML sanitization code to make it more maintainable.

    Added code for parsing inline UU-encoded attachments to MIMEStream.pm,
    used the code to sanitize such attachments.

    Added code for guessing MIME types from an attachment's name or the
    first few bytes of it's content (primitive magic).  This is used when we
    don't have an explicit handler for a part's MIME type.

    The .htm and .html extensions are now on the filename white-list, since
    we should be able to detect and sanitize HTML anywhere the mail readers
    can.  Assuming no mail readers can auto-open the contents of an archive
    or gziped file, that is.  

    Added programming-related extensions to the white list (uncompiled code
    is safe, scripts aren't).

    Added defanging of shell scripts beginning with '#!'.

    Did some performance tests, results are in the README.

Revision 1.9:

    Improved PGP handling a little.

    Improved HTML sanitizer quite a bit and made it log all modifications.

    Redid filename mangler to follow a "default deny" policy:
        if file name matches black-list, it is defanged,
        elsif file name matches white-list, don't defang,
        else defang.

    When attachment names are defanged, their MIME type is also defanged.

Revision 1.8:

    I fixed a tiny bug in 1.7 and sneakily copied it over the original 
    1.7 tarball.  So the released 1.7 was actually CVS revision 1.8.

Revision 1.7:

    Moved filename extension regexp to a seperate variable, preparing
    to expell it from the source entirely, and put it in a configuration 
    file.
    
    Fixed a bug in PGP signature handling.  Unsetting $feat_verbose would
    make the $feat_trustpgp flag have no effect (forcing a scan).

    Finished porting John's HTML sanitizer.  Improved it a little.
    
    Closed the last (known) potential run-away-memory-hog in the MIME 
    streamer.  $/ is cool, everybody go read "man perlop"!  Redid the
    MIME streamer's line buffering so I could choose an arbitrary EOL.

Revision 1.6:

    First public release.