anomy-list

Re: The QP encoding issue - again!

From: Dustin Puryear (101507@xyz.molar.is)
Date: Fri 14 Nov 2003 - 02:09:15 GMT

  • Next message: Dustin Puryear: "Re: The QP encoding issue - again!"

    Then again, it might work. (My main problem right now is that I don't have
    an example of a PDF that breaks when going through Anomy. If someone has one
    then send me the original and the broken one.)

    If the MIME QP encoded PDF doesn't go through Anomy then it works.
    If the MIME QP encoded PDF does go through Anomy then it might fail.

    The question then is: What makes it fail?

    Am I right that the damage is occuring in MIMEStream::RawRead()?
    Specifically, here:

            if (!$reader->{"INforce"})
            {
                    # Slurp at most 8k of input at a time.
                    $line = $reader->{"parent"}->Read(8192);
            }
            else
            {
                    # Slurp 8k of input at a time.
                    $reader->{"IN"}->read($line, 8192);

                    if (defined $reader->{"newline_in"})
                    {
                        # Figure out what our newline convention is...
                        my $nl = $reader->{"newline_in"};
                            if ((!$nl) && ($line =~ /(\015?\012)/s))
                            {
                                my $writer = $reader->{"writer"} || $reader;
                            $reader->{"newline_in"} = $nl = $1;
                                    $writer->{"newline_out"} = $nl
                                      unless ($writer->{"newline_out"});
                            }

                            $line =~ s/(\015?\012)/\012/gs; # if ($nl);
                    }
            }

    If I detect that I have a QP encoded PDF then won't stopping the newline
    conversation fix this?

    A little feedback from the author would be great right now. :)

    What is INforce?

    ----- Original Message -----
    From: "Dustin Puryear" <101507@xyz.molar.is>
    To: <101544@xyz.molar.is>
    Sent: Thursday, November 13, 2003 7:15 PM
    Subject: Re: [anomy-list]: The QP encoding issue - again!

    > Hmm, on second though this won't fix the real problem. I was
    > misunderstanding the true cause of this issue. Has someone created a HOWTO
    > for solutions on this? It looks like we need to go with the wrapper script
    > solution, unless there is a fix coming through for Anomy?
    >
    > ----- Original Message -----
    > From: "Dustin Puryear" <101507@xyz.molar.is>
    > To: "Noel Clarkson" <101595@xyz.molar.is>;
    > <101544@xyz.molar.is>
    > Sent: Thursday, November 13, 2003 11:27 AM
    > Subject: Re: [anomy-list]: The QP encoding issue - again!
    >
    >
    > > Here is my initial idea. Let me know what you think.
    > >
    > > I configure Sanitizer.pm to look for the new configuration parameter
    > > "qpe_special_files", which works list a normal file_list. So a
    > configuration
    > > may be:
    > >
    > > bin/sanitizer.pl 'qpe_special_files = (?)\.pdf' < test.txt
    > >
    > > Next, in SanitizeFile I check for the filename:
    > >
    > > ---><----
    > > sub SanitizeFile
    > > ...
    > >
    > > # XX -
    > > # qpencoding_special_files
    > > print "OrigFilename = $ofn\n";
    > > if ($ofn =~ $conf->{"qpe_special_files"}) {
    > > $part->{"is_qpe_special_file"} = 1;
    > > print "Yes, it is a PDF\n";
    > > print "Setting is_qpe_special_file = 1\n";
    > > }
    > > else {
    > > $part->{"is_qpe_special_file"} = 0;
    > > print "No, it is not a PDF\n";
    > > print "Setting is_qpe_special_file = 0\n";
    > > }
    > >
    > > # Check policies...
    > > my @rules = (0..$conf->{"file_list_rules"}, "default");
    > > my $rec = 0;
    > > my $ip = 1;
    > > my $mysize = 0;
    > > ---><----
    > >
    > > This just sets a flag for later on in $reader that will tell
    > > MIMEStream.pm::RawRead() that we are reading a quoted-printable encoded
    > file
    > > that we know is actually binary. (Thanks Outlook.)
    > >
    > > In MIMEStream.pm I modified RawRead(). The big change is that I will
    only
    > do
    > > the newline code if $reader->{"is_qpe_special_file"} is false:
    > >
    > > ---><----
    > > sub RawRead
    > > ...
    > >
    > > my $line;
    > > if (!$reader->{"INforce"})
    > > {
    > > # Slurp at most 8k of input at a time.
    > > $line = $reader->{"parent"}->Read(8192);
    > > }
    > > else
    > > {
    > > # Slurp 8k of input at a time.
    > > $reader->{"IN"}->read($line, 8192);
    > >
    > > # XX
    > > if (defined $reader->{"newline_in"} &&
    > > !$reader->{"is_qpe_special_file"})
    > > {
    > > # Figure out what our newline convention is...
    > > my $nl = $reader->{"newline_in"};
    > > if ((!$nl) && ($line =~ /(\015?\012)/s))
    > > {
    > > my $writer = $reader->{"writer"} || $reader;
    > > $reader->{"newline_in"} = $nl = $1;
    > > $writer->{"newline_out"} = $nl
    > > unless ($writer->{"newline_out"});
    > > }
    > >
    > > $line =~ s/(\015?\012)/\012/gs; # if ($nl);
    > > }
    > > }
    > >
    > > ---><----
    > >
    > > This just ensures that MIMEStream.pm doesn't do any newline conversions
    > when
    > > it is reading a known-binary file that is quoted-printable encoded.
    > >
    > > What are your thoughts on this kind of solution?
    > >
    > >
    > > ----- Original Message -----
    > > From: "Noel Clarkson" <101595@xyz.molar.is>
    > > To: "Dustin Puryear" <101640@xyz.molar.is>
    > > Cc: <101544@xyz.molar.is>
    > > Sent: Tuesday, November 11, 2003 6:42 PM
    > > Subject: Re: [anomy-list]: The QP encoding issue - again!
    > >
    > >
    > > > Hi there,
    > > >
    > > > a while back the suggested way to go with this seemed to be to save
    the
    > > > pdf, scan it for viruses, if the scan comes up clean then leave the
    > > > orriginal pdf in place, if not then do something else with the pdf. I
    > > > was looking into doing this and writing a bit of a description on how
    to
    > > > do it but it kind of fell by the wayside and I've got too many things
    to
    > > > do to pick it up now. If you think this might be the way to solve
    your
    > > > problem, have a look back through the archives, below is a quote out
    of
    > > > one of the mail messages that sums up what the idea is.
    > > >
    > > > From an email around May 2003 on the Anomy List:
    > > > > > >
    > > > > > > Currently the only way to avoid this problem is to wrap the
    > > > > > > sanitizer in a procmail ruleset or helper script which will
    > > > > > > quarantine the original unmodified message and use it verbatim
    if
    > > > > > > the sanitizer doesn't find any security risks in it.
    > > > > >
    > > > > > I could do that, as I always write incomming message to disk
    before
    > > > > > sending to sanitizer, then amavis adn another script for loging
    > > > > > attachment in DB.
    > > > > >
    > > > > > But how could I know if sanitizer hit something or no, I desn't
    see
    > a
    > > > > > particular exitcode using $?
    > > > > >
    > > > > I didn't see the score_bad option and the "!" possible on the
    policy
    > to
    > > > > increment the score. I now use this to get an exitcode != 0 and got
    a
    > > > > new working script which send the original message if the exit code
    > is
    > > > > 0.
    > > > >
    > > > > It just needs a little bit more disk io ;-// but it works.
    > > > >
    > > >
    > > >
    > > >
    > > > If you do get something to work, then I'm sure others would appreciate
    > > > if you can post a description so that we all don't have to work it out
    > > > step by step.
    > > >
    > > > cheers,
    > > >
    > > > noel
    > > >
    > > >
    > > >
    > > > Dustin Puryear wrote:
    > > > > Precisely. Because binary files, i.e., a PDF, is being encoded using
    > QP
    > > > > encoding Anomy is garbling them. (Not that Anomy is buggy. It's the
    > mail
    > > > > clients that are buggy.) If you hit the Anomy archives you will
    > > understand
    > > > > the problem.
    > > > >
    > > > > The actual plan right now is to either undo the changes made by
    Anomy,
    > > or to
    > > > > simply stop Anomy from making the changes in the first place. I'm
    > simply
    > > > > hoping someone has already modified and tested these changes in
    Anomy.
    > > > >
    > > > > ----- Original Message -----
    > > > > From: "Marvin Herbold" <101682@xyz.molar.is>
    > > > > To: "Dustin Puryear" <101507@xyz.molar.is>
    > > > > Cc: <101544@xyz.molar.is>
    > > > > Sent: Monday, November 10, 2003 7:52 PM
    > > > > Subject: Re: [anomy-list]: The QP encoding issue - again!
    > > > >
    > > > >
    > > > >
    > > > >>Umm - aren't PDF files binary? Simply converting LF to CRLF would
    > > > >>royally screw up these files... like in places where there is
    supposed
    > > > >>to be a 0x0A byte without a preceeding 0x0D byte.
    > > > >>
    > > > >>Dustin Puryear wrote:
    > > > >>
    > > > >>
    > > > >>>I'm glad you mentioned that. That is actually the plan of attack
    that
    > > we
    > > > >>>decided on, but for some reason I didn't list it. If someone has
    > > already
    > > > >>>done this then let us know. I would hate to reproduce the effort.
    > > Thanks!
    > > > >>>
    > > > >>>Reports suggest that this happens mostly with some graphics files
    and
    > > > >>
    > > > > PDFs.
    > > > >
    > > > >>>Anyone have experience with this on other files?
    > > > >>>
    > > > >>>
    > > > >>
    > > > >>--
    > > > >>Marvin Herbold
    > > > >>101682@xyz.molar.is
    > > > >>http://www.herbold-family.com
    > > > >>
    > > > >>
    > > > >>
    > > > >>
    > > > >>
    > > > >
    > > > >
    > > > >
    > > >
    > > >
    > > >
    > > >
    > > >
    > >
    > >
    > >
    > >
    >
    >
    >
    >



    hosted by molar.is