2 ** m_getfld.c -- read/parse a message
4 ** This code is Copyright (c) 2002, by the authors of nmh. See the
5 ** COPYRIGHT file in the root directory of the nmh distribution for
6 ** complete copyright information.
13 ** This module has a long and checkered history. First, it didn't burst
14 ** maildrops correctly because it considered two CTRL-A:s in a row to be
15 ** an inter-message delimiter. It really is four CTRL-A:s followed by a
16 ** newline. Unfortunately, MMDF will convert this delimiter *inside* a
17 ** message to a CTRL-B followed by three CTRL-A:s and a newline. This
18 ** caused the old version of m_getfld() to declare eom prematurely. The
19 ** fix was a lot slower than
21 ** c == '\001' && peekc (iob) == '\001'
23 ** but it worked, and to increase generality, MBOX style maildrops could
24 ** be parsed as well. Unfortunately the speed issue finally caught up with
25 ** us since this routine is at the very heart of MH.
27 ** To speed things up considerably, the routine Eom() was made an auxilary
28 ** function called by the macro eom(). Unless we are bursting a maildrop,
29 ** the eom() macro returns FALSE saying we aren't at the end of the
32 ** After mhl was made a built-in in msh, m_getfld() worked just fine
33 ** (using m_unknown() at startup). Until one day: a message which was
34 ** the result of a bursting was shown. Then, since the burst boundaries
35 ** aren't CTRL-A:s, m_getfld() would blinding plunge on past the boundary.
36 ** Very sad. The solution: introduce m_eomsbr(). This hook gets called
37 ** after the end of each line (since testing for eom involves an fseek()).
38 ** This worked fine, until one day: a message with no body portion arrived.
41 ** while (eom(c = getc(iob), iob))
44 ** loop caused m_getfld() to return FMTERR. So, that logic was changed to
45 ** check for (*eom_action) and act accordingly.
47 ** [ Note by meillo 2011-10:
48 ** as msh was removed from mmh, m_eomsbr() became irrelevant. ]
50 ** This worked fine, until one day: someone didn't use four CTRL:A's as
51 ** their delimiters. So, the bullet got bit and we read mts.h and
52 ** continue to struggle on. It's not that bad though, since the only time
53 ** the code gets executed is when inc (or msh) calls it, and both of these
54 ** have already called mts_init().
56 ** [ Note by meillo 2012-02:
57 ** MMDF-style maildrops (4x ^A) and mts_init() were removed. ]
59 ** ------------------------
60 ** (Written by Van Jacobson for the mh6 m_getfld, January, 1986):
62 ** This routine was accounting for 60% of the cpu time used by most mh
63 ** programs. I spent a bit of time tuning and it now accounts for <10%
64 ** of the time used. Like any heavily tuned routine, it's a bit
65 ** complex and you want to be sure you understand everything that it's
66 ** doing before you start hacking on it. Let me try to emphasize
67 ** that: every line in this atrocity depends on every other line,
68 ** sometimes in subtle ways. You should understand it all, in detail,
69 ** before trying to change any part. If you do change it, test the
70 ** result thoroughly (I use a hand-constructed test file that exercises
71 ** all the ways a header name, header body, header continuation,
72 ** header-body separator, body line and body eom can align themselves
73 ** with respect to a buffer boundary). "Minor" bugs in this routine
74 ** result in garbaged or lost mail.
76 ** If you hack on this and slow it down, I, my children and my
77 ** children's children will curse you.
79 ** This routine gets used on three different types of files: normal,
80 ** single msg files, "packed" unix or mmdf mailboxs (when used by inc)
81 ** and packed, directoried bulletin board files (when used by msh).
82 ** The biggest impact of different file types is in "eom" testing. The
83 ** code has been carefully organized to test for eom at appropriate
84 ** times and at no other times (since the check is quite expensive).
85 ** I have tried to arrange things so that the eom check need only be
86 ** done on entry to this routine. Since an eom can only occur after a
87 ** newline, this is easy to manage for header fields. For the msg
88 ** body, we try to efficiently search the input buffer to see if
89 ** contains the eom delimiter. If it does, we take up to the
90 ** delimiter, otherwise we take everything in the buffer. (The change
91 ** to the body eom/copy processing produced the most noticeable
92 ** performance difference, particularly for "inc" and "show".)
94 ** There are three qualitatively different things this routine busts
95 ** out of a message: field names, field text and msg bodies. Field
96 ** names are typically short (~8 char) and the loop that extracts them
97 ** might terminate on a colon, newline or max width. I considered
98 ** using a Vax "scanc" to locate the end of the field followed by a
99 ** "bcopy" but the routine call overhead on a Vax is too large for this
100 ** to work on short names. If Berkeley ever makes "inline" part of the
101 ** C optimiser (so things like "scanc" turn into inline instructions) a
102 ** change here would be worthwhile.
104 ** Field text is typically 60 - 100 characters so there's (barely)
105 ** a win in doing a routine call to something that does a "locc"
106 ** followed by a "bmove". About 30% of the fields have continuations
107 ** (usually the 822 "received:" lines) and each continuation generates
108 ** another routine call. "Inline" would be a big win here, as well.
110 ** Messages, as of this writing, seem to come in two flavors: small
111 ** (~1K) and long (>2K). Most messages have 400 - 600 bytes of headers
112 ** so message bodies average at least a few hundred characters.
113 ** Assuming your system uses reasonably sized stdio buffers (1K or
114 ** more), this routine should be able to remove the body in large
115 ** (>500 byte) chunks. The makes the cost of a call to "bcopy"
116 ** small but there is a premium on checking for the eom in packed
117 ** maildrops. The eom pattern is always a simple string so we can
118 ** construct an efficient pattern matcher for it (e.g., a Vax "matchc"
119 ** instruction). Some thought went into recognizing the start of
120 ** an eom that has been split across two buffers.
122 ** This routine wants to deal with large chunks of data so, rather
123 ** than "getc" into a local buffer, it uses stdio's buffer. If
124 ** you try to use it on a non-buffered file, you'll get what you
125 ** deserve. This routine "knows" that struct FILEs have a _ptr
126 ** and a _cnt to describe the current state of the buffer and
127 ** it knows that _filbuf ignores the _ptr & _cnt and simply fills
128 ** the buffer. If stdio on your system doesn't work this way, you
129 ** may have to make small changes in this routine.
131 ** This routine also "knows" that an EOF indication on a stream is
132 ** "sticky" (i.e., you will keep getting EOF until you reposition the
133 ** stream). If your system doesn't work this way it is broken and you
134 ** should complain to the vendor. As a consequence of the sticky
135 ** EOF, this routine will never return any kind of EOF status when
136 ** there is data in "name" or "buf").
143 static int m_Eom(int, FILE *);
144 static unsigned char *matchc(int, char *, int, char *);
145 static unsigned char *locc(int, unsigned char *, unsigned char);
147 #define eom(c,iob) (msg_style == MS_MBOX && \
148 (((c) == *msg_delim && m_Eom(c,iob)) ||\
149 (eom_action && (*eom_action)(c))))
151 static unsigned char **pat_map;
154 ** This is a disgusting hack for "inc" so it can know how many
155 ** characters were stuffed in the buffer on the last call
156 ** (see comments in uip/scansbr.c).
160 int msg_style = MS_DEFAULT;
163 ** The "full" delimiter string for a packed maildrop consists
164 ** of a newline followed by the actual delimiter. E.g., the
165 ** full string for a Unix maildrop would be: "\n\nFrom ".
166 ** "Fdelim" points to the start of the full string and is used
167 ** in the BODY case of the main routine to search the buffer for
168 ** a possible eom. Msg_delim points to the first character of
169 ** the actual delim. string (i.e., fdelim+1). Edelim
170 ** points to the 2nd character of actual delimiter string. It
171 ** is used in m_Eom because the first character of the string
172 ** has been read and matched before m_Eom is called.
174 char *msg_delim = "";
176 static unsigned char *fdelim;
177 static unsigned char *delimend;
178 static int fdelimlen;
179 static unsigned char *edelim;
180 static int edelimlen;
182 static int (*eom_action)(int) = NULL;
185 # define _ptr _p /* Gag */
186 # define _cnt _r /* Retch */
187 # define _filbuf __srget /* Puke */
188 # define DEFINED__FILBUF_TO_SOMETHING_SPECIFIC
194 # define _base __base
195 # define _filbuf(fp) ((fp)->__cnt = 0, __filbuf(fp))
196 # define DEFINED__FILBUF_TO_SOMETHING_SPECIFIC
199 #ifndef DEFINED__FILBUF_TO_SOMETHING_SPECIFIC
200 extern int _filbuf(FILE*);
205 m_getfld(int state, unsigned char *name, unsigned char *buf,
206 int bufsz, FILE *iob)
208 register unsigned char *bp, *cp, *ep, *sp;
209 register int cnt, c, i, j;
211 if ((c = getc(iob)) < 0) {
218 /* flush null messages */
219 while ((c = getc(iob)) >= 0 && eom(c, iob))
233 if (c == '\n' || c == '-') {
234 /* we hit the header/body separator */
235 while (c != '\n' && (c = getc(iob)) >= 0)
238 if (c < 0 || (c = getc(iob)) < 0 || eom(c, iob)) {
240 /* flush null messages */
241 while ((c = getc(iob)) >= 0 && eom(c, iob))
254 ** get the name of this component. take characters up
255 ** to a ':', a newline or NAMESZ-1 characters,
256 ** whichever comes first.
262 bp = sp = (unsigned char *) iob->_IO_read_ptr - 1;
263 j = (cnt = ((long) iob->_IO_read_end -
264 (long) iob->_IO_read_ptr) + 1) < i ? cnt : i;
265 #elif defined(__DragonFly__)
266 bp = sp = (unsigned char *) ((struct __FILE_public *)iob)->_p - 1;
267 j = (cnt = ((struct __FILE_public *)iob)->_r+1) < i ? cnt : i;
269 bp = sp = (unsigned char *) iob->_ptr - 1;
270 j = (cnt = iob->_cnt+1) < i ? cnt : i;
272 while (--j >= 0 && (c = *bp++) != ':' && c != '\n')
276 if ((cnt -= j) <= 0) {
278 iob->_IO_read_ptr = iob->_IO_read_end;
279 if (__underflow(iob) == EOF) {
280 #elif defined(__DragonFly__)
281 if (__srget(iob) == EOF) {
283 if (_filbuf(iob) == EOF) {
286 advise(NULL, "eof encountered in field \"%s\"", name);
290 iob->_IO_read_ptr++; /* NOT automatic in __underflow()! */
294 iob->_IO_read_ptr = bp + 1;
295 #elif defined(__DragonFly__)
296 ((struct __FILE_public *)iob)->_p = bp + 1;
297 ((struct __FILE_public *)iob)->_r = cnt - 1;
307 ** something went wrong. possibilities are:
308 ** . hit a newline (error)
309 ** . got more than namesz chars. (error)
310 ** . hit the end of the buffer. (loop)
314 ** We hit the end of the line without
315 ** seeing ':' to terminate the field name.
316 ** This is usually (always?) spam. But,
317 ** blowing up is lame, especially when
318 ** scan(1)ing a folder with such messages.
319 ** Pretend such lines are the first of
320 ** the body (at least mutt also handles
325 ** See if buf can hold this line, since we
326 ** were assuming we had a buffer of NAMESZ,
329 /* + 1 for the newline */
332 ** No, it can't. Oh well,
333 ** guess we'll blow up.
336 advise(NULL, "eol encountered in field \"%s\"", name);
340 memcpy(buf, name, j - 1);
344 ** mhparse.c:get_content wants to find
345 ** the position of the body start, but
346 ** it thinks there's a blank line between
347 ** the header and the body (naturally!),
348 ** so seek back so that things line up
349 ** even though we don't have that blank
350 ** line in this case. Simpler parsers
351 ** (e.g. mhl) get extra newlines, but
352 ** that should be harmless enough, right?
353 ** This is a corrupt message anyway.
355 fseek(iob, ftell(iob) - 2, SEEK_SET);
360 advise(NULL, "field name \"%s\" exceeds %d bytes", name, NAMESZ - 2);
366 while (isspace(*--cp) && cp >= name)
373 ** get (more of) the text of a field. take
374 ** characters up to the end of this field (newline
375 ** followed by non-blank) or bufsz-1 characters.
377 cp = buf; i = bufsz-1;
380 cnt = (long) iob->_IO_read_end - (long) iob->_IO_read_ptr;
381 bp = (unsigned char *) --iob->_IO_read_ptr;
382 #elif defined(__DragonFly__)
383 cnt = ((struct __FILE_public *)iob)->_r++;
384 bp = (unsigned char *) --((struct __FILE_public *)iob)->_p;
387 bp = (unsigned char *) --iob->_ptr;
389 c = cnt < i ? cnt : i;
390 while ((ep = locc( c, bp, '\n' ))) {
392 ** if we hit the end of this field,
395 if ((j = *++ep) != ' ' && j != '\t') {
397 j = ep - (unsigned char *) iob->_IO_read_ptr;
398 memcpy(cp, iob->_IO_read_ptr, j);
399 iob->_IO_read_ptr = ep;
400 #elif defined(__DragonFly__)
401 j = ep - (unsigned char *) ((struct __FILE_public *)iob)->_p;
402 memcpy(cp, ((struct __FILE_public *)iob)->_p, j);
403 ((struct __FILE_public *)iob)->_p = ep;
404 ((struct __FILE_public *)iob)->_r -= j;
406 j = ep - (unsigned char *) iob->_ptr;
407 memcpy(cp, iob->_ptr, j);
419 ** end of input or dest buffer - copy what
423 c += bp - (unsigned char *) iob->_IO_read_ptr;
424 memcpy(cp, iob->_IO_read_ptr, c);
425 #elif defined(__DragonFly__)
426 c += bp - (unsigned char *) ((struct __FILE_public *)iob)->_p;
427 memcpy(cp, ((struct __FILE_public *)iob)->_p, c);
429 c += bp - (unsigned char *) iob->_ptr;
430 memcpy(cp, iob->_ptr, c);
435 /* the dest buffer is full */
437 iob->_IO_read_ptr += c;
438 #elif defined(__DragonFly__)
439 ((struct __FILE_public *)iob)->_r -= c;
440 ((struct __FILE_public *)iob)->_p += c;
449 ** There's one character left in the input
450 ** buffer. Copy it & fill the buffer.
451 ** If the last char was a newline and the
452 ** next char is not whitespace, this is
453 ** the end of the field. Otherwise loop.
457 *cp++ = j = *(iob->_IO_read_ptr + c);
458 iob->_IO_read_ptr = iob->_IO_read_end;
459 c = __underflow(iob);
460 iob->_IO_read_ptr++; /* NOT automatic! */
461 #elif defined(__DragonFly__)
462 *cp++ =j = *(((struct __FILE_public *)iob)->_p + c);
465 *cp++ = j = *(iob->_ptr + c);
469 ((j == '\0' || j == '\n') && c != ' ' && c != '\t')) {
473 #elif defined(__DragonFly__)
474 --((struct __FILE_public *)iob)->_p;
475 ++((struct __FILE_public *)iob)->_r;
490 ** get the message body up to bufsz characters or
491 ** the end of the message. Sleazy hack: if bufsz
492 ** is negative we assume that we were called to
493 ** copy directly into the output buffer and we
496 i = (bufsz < 0) ? -bufsz : bufsz-1;
498 bp = (unsigned char *) --iob->_IO_read_ptr;
499 cnt = (long) iob->_IO_read_end - (long) iob->_IO_read_ptr;
500 #elif defined(__DragonFly__)
501 bp = (unsigned char *) --((struct __FILE_public *)iob)->_p;
502 cnt = ++((struct __FILE_public *)iob)->_r;
504 bp = (unsigned char *) --iob->_ptr;
507 c = (cnt < i ? cnt : i);
508 if (msg_style == MS_MBOX && c > 1) {
510 ** packed maildrop - only take up to the (possible)
511 ** start of the next message. This "matchc" should
512 ** probably be a Boyer-Moore matcher for non-vaxen,
513 ** particularly since we have the alignment table
514 ** all built for the end-of-buffer test (next).
515 ** But our vax timings indicate that the "matchc"
516 ** instruction is 50% faster than a carefully coded
517 ** B.M. matcher for most strings. (So much for
518 ** elegant algorithms vs. brute force.) Since I
519 ** (currently) run MH on a vax, we use the matchc
522 if ((ep = matchc( fdelimlen, fdelim, c, bp )))
526 ** There's no delim in the buffer but
527 ** there may be a partial one at the end.
528 ** If so, we want to leave it so the "eom"
529 ** check on the next call picks it up. Use a
530 ** modified Boyer-Moore matcher to make this
531 ** check relatively cheap. The first "if"
532 ** figures out what position in the pattern
533 ** matches the last character in the buffer.
534 ** The inner "while" matches the pattern
535 ** against the buffer, backwards starting
536 ** at that position. Note that unless the
537 ** buffer ends with one of the characters
538 ** in the pattern (excluding the first
539 ** and last), we do only one test.
542 if ((sp = pat_map[*ep])) {
545 ** This if() is true unless
546 ** (a) the buffer is too
547 ** small to contain this
549 ** or (b) it contains
550 ** exactly enough chars for
551 ** the delimiter prefix.
552 ** For case (a) obviously we
553 ** aren't going to match.
554 ** For case (b), if the
555 ** buffer really contained
556 ** exactly a delim prefix,
557 ** then the m_eom call
558 ** at entry should have
559 ** found it. Thus it's
560 ** not a delim and we know
561 ** we won't get a match.
563 if (((sp - fdelim) + 2) <= c) {
566 ** Unfortunately although fdelim has a preceding NUL
567 ** we can't use this as a sentinel in case the buffer
568 ** contains a NUL in exactly the wrong place (this
569 ** would cause us to run off the front of fdelim).
571 while (*--ep == *--cp)
575 /* we matched the entire delim prefix,
576 ** so only take the buffer up to there.
577 ** we know ep >= bp -- check above prevents underrun
583 /* try matching one less char of delim string */
585 } while (--sp > fdelim);
589 memcpy( buf, bp, c );
591 iob->_IO_read_ptr += c;
592 #elif defined(__DragonFly__)
593 ((struct __FILE_public *)iob)->_r -= c;
594 ((struct __FILE_public *)iob)->_p += c;
607 adios(NULL, "m_getfld() called with bogus state of %d", state);
611 msg_count = cp - buf;
615 static char unixbuf[BUFSIZ] = "";
624 register char *delimstr;
627 ** Figure out what the message delimitter string is for this
628 ** maildrop. (This used to be part of m_Eom but I didn't like
629 ** the idea of an "if" statement that could only succeed on the
630 ** first call to m_Eom getting executed on each call, i.e., at
631 ** every newline in the message).
633 ** If the first line of the maildrop is a Unix "From " line, we
634 ** say the style is MBOX and eat the rest of the line. Otherwise
639 if (fread(text, sizeof(*text), 5, iob) != 5) {
640 adios(NULL, "Read error");
642 if (strncmp(text, "From ", 5)!=0) {
643 adios(NULL, "No Unix style (mbox) maildrop.");
646 delimstr = "\nFrom ";
648 while ((c = getc(iob)) != '\n' && cp - unixbuf < BUFSIZ - 1)
652 c = strlen(delimstr);
653 fdelim = (unsigned char *) mh_xmalloc((size_t) (c + 3));
656 msg_delim = (char *)fdelim+1;
657 edelim = (unsigned char *)msg_delim+1;
660 strcpy(msg_delim, delimstr);
661 delimend = (unsigned char *)msg_delim + edelimlen;
663 adios(NULL, "maildrop delimiter must be at least 2 bytes");
665 ** build a Boyer-Moore end-position map for the matcher in m_getfld.
666 ** N.B. - we don't match just the first char (since it's the newline
667 ** separator) or the last char (since the matchc would have found it
668 ** if it was a real delim).
670 pat_map = (unsigned char **) calloc(256, sizeof(unsigned char *));
672 for (cp = (char *) fdelim + 1; cp < (char *) delimend; cp++ )
673 pat_map[(unsigned char)*cp] = (unsigned char *) cp;
678 ** test for msg delimiter string
682 m_Eom(int c, FILE *iob)
684 register long pos = 0L;
690 if ((i = fread(text, sizeof *text, edelimlen, iob)) != edelimlen
691 || (strncmp(text, (char *)edelim, edelimlen)!=0)) {
692 if (i == 0 && msg_style == MS_MBOX)
694 ** the final newline in the (brain damaged) unix-format
695 ** maildrop is part of the delimitter - delete it.
699 fseek(iob, (long)(pos-1), SEEK_SET);
700 getc(iob); /* should be OK */
704 if (msg_style == MS_MBOX) {
706 while ((c = getc(iob)) != '\n' && c >= 0 && cp - unixbuf < BUFSIZ - 1)
716 ** Return the Return-Path and Delivery-Date
717 ** header information.
719 ** Currently, I'm assuming that the "From " line takes the following form:
720 ** "From" sender@host date (sendmail delivery)
723 get_returnpath(char *rp, int rplen, char *dd, int ddlen)
728 if (!(bp = strchr(ap, ' ')))
731 /* Get the Return-Path information from the "From " envelope. */
732 snprintf(rp, rplen, "%.*s\n", (int)(bp - ap), ap);
735 ** advance over the spaces to get to
736 ** delivery date on envelope
741 /* Now get delivery date from envelope */
742 snprintf(dd, ddlen, "%.*s\n", 24, bp);
749 static unsigned char *
750 matchc(int patln, char *pat, int strln, char *str)
752 register char *es = str + strln - patln;
755 register char *ep = pat + patln;
756 register char pc = *pat++;
765 while (pp < ep && *sp++ == *pp)
768 return ((unsigned char *)--str);
774 ** Locate character "term" in the next "cnt" characters of "src".
775 ** If found, return its address, otherwise return 0.
778 static unsigned char *
779 locc(int cnt, unsigned char *src, unsigned char term)
781 while (*src++ != term && --cnt > 0)
784 return (cnt > 0 ? --src : (unsigned char *)0);