2 ** m_getfld.c -- read/parse a message
4 ** This code is Copyright (c) 2002, by the authors of nmh. See the
5 ** COPYRIGHT file in the root directory of the nmh distribution for
6 ** complete copyright information.
14 ** This module has a long and checkered history. First, it didn't burst
15 ** maildrops correctly because it considered two CTRL-A:s in a row to be
16 ** an inter-message delimiter. It really is four CTRL-A:s followed by a
17 ** newline. Unfortunately, MMDF will convert this delimiter *inside* a
18 ** message to a CTRL-B followed by three CTRL-A:s and a newline. This
19 ** caused the old version of m_getfld() to declare eom prematurely. The
20 ** fix was a lot slower than
22 ** c == '\001' && peekc (iob) == '\001'
24 ** but it worked, and to increase generality, MBOX style maildrops could
25 ** be parsed as well. Unfortunately the speed issue finally caught up with
26 ** us since this routine is at the very heart of MH.
28 ** To speed things up considerably, the routine Eom() was made an auxilary
29 ** function called by the macro eom(). Unless we are bursting a maildrop,
30 ** the eom() macro returns FALSE saying we aren't at the end of the
33 ** The next thing to do is to read the mts.conf file and initialize
34 ** delimiter[] and delimlen accordingly...
36 ** After mhl was made a built-in in msh, m_getfld() worked just fine
37 ** (using m_unknown() at startup). Until one day: a message which was
38 ** the result of a bursting was shown. Then, since the burst boundaries
39 ** aren't CTRL-A:s, m_getfld() would blinding plunge on past the boundary.
40 ** Very sad. The solution: introduce m_eomsbr(). This hook gets called
41 ** after the end of each line (since testing for eom involves an fseek()).
42 ** This worked fine, until one day: a message with no body portion arrived.
45 ** while (eom(c = Getc(iob), iob))
48 ** loop caused m_getfld() to return FMTERR. So, that logic was changed to
49 ** check for (*eom_action) and act accordingly.
51 ** [ Note by meillo 2011-10:
52 ** as msh was removed from mmh, m_eomsbr() became irrelevant. ]
54 ** This worked fine, until one day: someone didn't use four CTRL:A's as
55 ** their delimiters. So, the bullet got bit and we read mts.h and
56 ** continue to struggle on. It's not that bad though, since the only time
57 ** the code gets executed is when inc (or msh) calls it, and both of these
58 ** have already called mts_init().
60 ** ------------------------
61 ** (Written by Van Jacobson for the mh6 m_getfld, January, 1986):
63 ** This routine was accounting for 60% of the cpu time used by most mh
64 ** programs. I spent a bit of time tuning and it now accounts for <10%
65 ** of the time used. Like any heavily tuned routine, it's a bit
66 ** complex and you want to be sure you understand everything that it's
67 ** doing before you start hacking on it. Let me try to emphasize
68 ** that: every line in this atrocity depends on every other line,
69 ** sometimes in subtle ways. You should understand it all, in detail,
70 ** before trying to change any part. If you do change it, test the
71 ** result thoroughly (I use a hand-constructed test file that exercises
72 ** all the ways a header name, header body, header continuation,
73 ** header-body separator, body line and body eom can align themselves
74 ** with respect to a buffer boundary). "Minor" bugs in this routine
75 ** result in garbaged or lost mail.
77 ** If you hack on this and slow it down, I, my children and my
78 ** children's children will curse you.
80 ** This routine gets used on three different types of files: normal,
81 ** single msg files, "packed" unix or mmdf mailboxs (when used by inc)
82 ** and packed, directoried bulletin board files (when used by msh).
83 ** The biggest impact of different file types is in "eom" testing. The
84 ** code has been carefully organized to test for eom at appropriate
85 ** times and at no other times (since the check is quite expensive).
86 ** I have tried to arrange things so that the eom check need only be
87 ** done on entry to this routine. Since an eom can only occur after a
88 ** newline, this is easy to manage for header fields. For the msg
89 ** body, we try to efficiently search the input buffer to see if
90 ** contains the eom delimiter. If it does, we take up to the
91 ** delimiter, otherwise we take everything in the buffer. (The change
92 ** to the body eom/copy processing produced the most noticeable
93 ** performance difference, particularly for "inc" and "show".)
95 ** There are three qualitatively different things this routine busts
96 ** out of a message: field names, field text and msg bodies. Field
97 ** names are typically short (~8 char) and the loop that extracts them
98 ** might terminate on a colon, newline or max width. I considered
99 ** using a Vax "scanc" to locate the end of the field followed by a
100 ** "bcopy" but the routine call overhead on a Vax is too large for this
101 ** to work on short names. If Berkeley ever makes "inline" part of the
102 ** C optimiser (so things like "scanc" turn into inline instructions) a
103 ** change here would be worthwhile.
105 ** Field text is typically 60 - 100 characters so there's (barely)
106 ** a win in doing a routine call to something that does a "locc"
107 ** followed by a "bmove". About 30% of the fields have continuations
108 ** (usually the 822 "received:" lines) and each continuation generates
109 ** another routine call. "Inline" would be a big win here, as well.
111 ** Messages, as of this writing, seem to come in two flavors: small
112 ** (~1K) and long (>2K). Most messages have 400 - 600 bytes of headers
113 ** so message bodies average at least a few hundred characters.
114 ** Assuming your system uses reasonably sized stdio buffers (1K or
115 ** more), this routine should be able to remove the body in large
116 ** (>500 byte) chunks. The makes the cost of a call to "bcopy"
117 ** small but there is a premium on checking for the eom in packed
118 ** maildrops. The eom pattern is always a simple string so we can
119 ** construct an efficient pattern matcher for it (e.g., a Vax "matchc"
120 ** instruction). Some thought went into recognizing the start of
121 ** an eom that has been split across two buffers.
123 ** This routine wants to deal with large chunks of data so, rather
124 ** than "getc" into a local buffer, it uses stdio's buffer. If
125 ** you try to use it on a non-buffered file, you'll get what you
126 ** deserve. This routine "knows" that struct FILEs have a _ptr
127 ** and a _cnt to describe the current state of the buffer and
128 ** it knows that _filbuf ignores the _ptr & _cnt and simply fills
129 ** the buffer. If stdio on your system doesn't work this way, you
130 ** may have to make small changes in this routine.
132 ** This routine also "knows" that an EOF indication on a stream is
133 ** "sticky" (i.e., you will keep getting EOF until you reposition the
134 ** stream). If your system doesn't work this way it is broken and you
135 ** should complain to the vendor. As a consequence of the sticky
136 ** EOF, this routine will never return any kind of EOF status when
137 ** there is data in "name" or "buf").
144 static int m_Eom(int, FILE *);
145 static unsigned char *matchc(int, char *, int, char *);
146 static unsigned char *locc(int, unsigned char *, unsigned char);
148 #define Getc(iob) getc(iob)
149 #define eom(c,iob) (msg_style != MS_DEFAULT && \
150 (((c) == *msg_delim && m_Eom(c,iob)) ||\
151 (eom_action && (*eom_action)(c))))
153 static unsigned char **pat_map;
156 ** defined in sbr/m_msgdef.c = 0
157 ** This is a disgusting hack for "inc" so it can know how many
158 ** characters were stuffed in the buffer on the last call
159 ** (see comments in uip/scansbr.c).
161 extern int msg_count;
164 ** defined in sbr/m_msgdef.c = MS_DEFAULT
166 extern int msg_style;
169 ** The "full" delimiter string for a packed maildrop consists
170 ** of a newline followed by the actual delimiter. E.g., the
171 ** full string for a Unix maildrop would be: "\n\nFrom ".
172 ** "Fdelim" points to the start of the full string and is used
173 ** in the BODY case of the main routine to search the buffer for
174 ** a possible eom. Msg_delim points to the first character of
175 ** the actual delim. string (i.e., fdelim+1). Edelim
176 ** points to the 2nd character of actual delimiter string. It
177 ** is used in m_Eom because the first character of the string
178 ** has been read and matched before m_Eom is called.
180 extern char *msg_delim; /* defined in sbr/m_msgdef.c = "" */
181 static unsigned char *fdelim;
182 static unsigned char *delimend;
183 static int fdelimlen;
184 static unsigned char *edelim;
185 static int edelimlen;
187 static int (*eom_action)(int) = NULL;
190 # define _ptr _p /* Gag */
191 # define _cnt _r /* Retch */
192 # define _filbuf __srget /* Puke */
193 # define DEFINED__FILBUF_TO_SOMETHING_SPECIFIC
199 # define _base __base
200 # define _filbuf(fp) ((fp)->__cnt = 0, __filbuf(fp))
201 # define DEFINED__FILBUF_TO_SOMETHING_SPECIFIC
204 #ifndef DEFINED__FILBUF_TO_SOMETHING_SPECIFIC
205 extern int _filbuf(FILE*);
210 m_getfld(int state, unsigned char *name, unsigned char *buf,
211 int bufsz, FILE *iob)
213 register unsigned char *bp, *cp, *ep, *sp;
214 register int cnt, c, i, j;
216 if ((c = Getc(iob)) < 0) {
223 /* flush null messages */
224 while ((c = Getc(iob)) >= 0 && eom(c, iob))
238 if (c == '\n' || c == '-') {
239 /* we hit the header/body separator */
240 while (c != '\n' && (c = Getc(iob)) >= 0)
243 if (c < 0 || (c = Getc(iob)) < 0 || eom(c, iob)) {
245 /* flush null messages */
246 while ((c = Getc(iob)) >= 0 && eom(c, iob))
259 ** get the name of this component. take characters up
260 ** to a ':', a newline or NAMESZ-1 characters,
261 ** whichever comes first.
267 bp = sp = (unsigned char *) iob->_IO_read_ptr - 1;
268 j = (cnt = ((long) iob->_IO_read_end -
269 (long) iob->_IO_read_ptr) + 1) < i ? cnt : i;
270 #elif defined(__DragonFly__)
271 bp = sp = (unsigned char *) ((struct __FILE_public *)iob)->_p - 1;
272 j = (cnt = ((struct __FILE_public *)iob)->_r+1) < i ? cnt : i;
274 bp = sp = (unsigned char *) iob->_ptr - 1;
275 j = (cnt = iob->_cnt+1) < i ? cnt : i;
277 while (--j >= 0 && (c = *bp++) != ':' && c != '\n')
281 if ((cnt -= j) <= 0) {
283 iob->_IO_read_ptr = iob->_IO_read_end;
284 if (__underflow(iob) == EOF) {
285 #elif defined(__DragonFly__)
286 if (__srget(iob) == EOF) {
288 if (_filbuf(iob) == EOF) {
291 advise(NULL, "eof encountered in field \"%s\"", name);
295 iob->_IO_read_ptr++; /* NOT automatic in __underflow()! */
299 iob->_IO_read_ptr = bp + 1;
300 #elif defined(__DragonFly__)
301 ((struct __FILE_public *)iob)->_p = bp + 1;
302 ((struct __FILE_public *)iob)->_r = cnt - 1;
312 ** something went wrong. possibilities are:
313 ** . hit a newline (error)
314 ** . got more than namesz chars. (error)
315 ** . hit the end of the buffer. (loop)
319 ** We hit the end of the line
320 ** without seeing ':' to terminate
321 ** the field name. This is usually
322 ** (always?) spam. But, blowing
323 ** up is lame, especially when
324 ** scan(1)ing a folder with such
325 ** messages. Pretend such lines are
326 ** the first of the body (at least
327 ** mutt also handles it this way).
331 ** See if buf can hold this line,
332 ** since we were assuming we had
333 ** a buffer of NAMESZ, not bufsz.
335 /* + 1 for the newline */
338 ** No, it can't. Oh well,
339 ** guess we'll blow up.
342 advise(NULL, "eol encountered in field \"%s\"", name);
346 memcpy(buf, name, j - 1);
350 ** mhparse.c:get_content wants to
351 ** find the position of the body
352 ** start, but it thinks there's a
353 ** blank line between the header
354 ** and the body (naturally!), so
355 ** seek back so that things line
356 ** up even though we don't have
357 ** that blank line in this case.
358 ** Simpler parsers (e.g. mhl)
359 ** get extra newlines, but that
360 ** should be harmless enough, right?
361 ** This is a corrupt message anyway.
363 fseek(iob, ftell(iob) - 2, SEEK_SET);
368 advise(NULL, "field name \"%s\" exceeds %d bytes", name, NAMESZ - 2);
374 while (isspace(*--cp) && cp >= name)
381 ** get (more of) the text of a field. take
382 ** characters up to the end of this field (newline
383 ** followed by non-blank) or bufsz-1 characters.
385 cp = buf; i = bufsz-1;
388 cnt = (long) iob->_IO_read_end - (long) iob->_IO_read_ptr;
389 bp = (unsigned char *) --iob->_IO_read_ptr;
390 #elif defined(__DragonFly__)
391 cnt = ((struct __FILE_public *)iob)->_r++;
392 bp = (unsigned char *) --((struct __FILE_public *)iob)->_p;
395 bp = (unsigned char *) --iob->_ptr;
397 c = cnt < i ? cnt : i;
398 while ((ep = locc( c, bp, '\n' ))) {
400 ** if we hit the end of this field,
403 if ((j = *++ep) != ' ' && j != '\t') {
405 j = ep - (unsigned char *) iob->_IO_read_ptr;
406 memcpy(cp, iob->_IO_read_ptr, j);
407 iob->_IO_read_ptr = ep;
408 #elif defined(__DragonFly__)
409 j = ep - (unsigned char *) ((struct __FILE_public *)iob)->_p;
410 memcpy(cp, ((struct __FILE_public *)iob)->_p, j);
411 ((struct __FILE_public *)iob)->_p = ep;
412 ((struct __FILE_public *)iob)->_r -= j;
414 j = ep - (unsigned char *) iob->_ptr;
415 memcpy(cp, iob->_ptr, j);
427 ** end of input or dest buffer - copy what
431 c += bp - (unsigned char *) iob->_IO_read_ptr;
432 memcpy(cp, iob->_IO_read_ptr, c);
433 #elif defined(__DragonFly__)
434 c += bp - (unsigned char *) ((struct __FILE_public *)iob)->_p;
435 memcpy(cp, ((struct __FILE_public *)iob)->_p, c);
437 c += bp - (unsigned char *) iob->_ptr;
438 memcpy(cp, iob->_ptr, c);
443 /* the dest buffer is full */
445 iob->_IO_read_ptr += c;
446 #elif defined(__DragonFly__)
447 ((struct __FILE_public *)iob)->_r -= c;
448 ((struct __FILE_public *)iob)->_p += c;
457 ** There's one character left in the input
458 ** buffer. Copy it & fill the buffer.
459 ** If the last char was a newline and the
460 ** next char is not whitespace, this is
461 ** the end of the field. Otherwise loop.
465 *cp++ = j = *(iob->_IO_read_ptr + c);
466 iob->_IO_read_ptr = iob->_IO_read_end;
467 c = __underflow(iob);
468 iob->_IO_read_ptr++; /* NOT automatic! */
469 #elif defined(__DragonFly__)
470 *cp++ =j = *(((struct __FILE_public *)iob)->_p + c);
473 *cp++ = j = *(iob->_ptr + c);
477 ((j == '\0' || j == '\n') && c != ' ' && c != '\t')) {
481 #elif defined(__DragonFly__)
482 --((struct __FILE_public *)iob)->_p;
483 ++((struct __FILE_public *)iob)->_r;
498 ** get the message body up to bufsz characters or
499 ** the end of the message. Sleazy hack: if bufsz
500 ** is negative we assume that we were called to
501 ** copy directly into the output buffer and we
504 i = (bufsz < 0) ? -bufsz : bufsz-1;
506 bp = (unsigned char *) --iob->_IO_read_ptr;
507 cnt = (long) iob->_IO_read_end - (long) iob->_IO_read_ptr;
508 #elif defined(__DragonFly__)
509 bp = (unsigned char *) --((struct __FILE_public *)iob)->_p;
510 cnt = ++((struct __FILE_public *)iob)->_r;
512 bp = (unsigned char *) --iob->_ptr;
515 c = (cnt < i ? cnt : i);
516 if (msg_style != MS_DEFAULT && c > 1) {
518 ** packed maildrop - only take up to the
519 ** (possible) start of the next message.
520 ** This "matchc" should probably be a
521 ** Boyer-Moore matcher for non-vaxen,
522 ** particularly since we have the alignment
523 ** table all built for the end-of-buffer
524 ** test (next). But our vax timings
525 ** indicate that the "matchc" instruction
526 ** is 50% faster than a carefully coded
527 ** B.M. matcher for most strings. (So much
528 ** for elegant algorithms vs. brute force.)
529 ** Since I (currently) run MH on a vax,
530 ** we use the matchc instruction. --vj
532 if ((ep = matchc( fdelimlen, fdelim, c, bp )))
536 ** There's no delim in the buffer
537 ** but there may be a partial one
538 ** at the end. If so, we want
539 ** to leave it so the "eom" check
540 ** on the next call picks it up.
541 ** Use a modified Boyer-Moore
542 ** matcher to make this check
543 ** relatively cheap. The first
544 ** "if" figures out what position
545 ** in the pattern matches the
546 ** last character in the buffer.
547 ** The inner "while" matches the
548 ** pattern against the buffer,
549 ** backwards starting at that
550 ** position. Note that unless
551 ** the buffer ends with one of
552 ** the characters in the pattern
553 ** (excluding the first and last),
554 ** we do only one test.
557 if ((sp = pat_map[*ep])) {
587 if (((sp - fdelim) + 2) <= c) {
590 ** Unfortunately although fdelim has a preceding NUL
591 ** we can't use this as a sentinel in case the buffer
592 ** contains a NUL in exactly the wrong place (this
593 ** would cause us to run off the front of fdelim).
595 while (*--ep == *--cp)
599 /* we matched the entire delim prefix,
600 ** so only take the buffer up to there.
601 ** we know ep >= bp -- check above prevents underrun
607 /* try matching one less char of delim string */
609 } while (--sp > fdelim);
613 memcpy( buf, bp, c );
615 iob->_IO_read_ptr += c;
616 #elif defined(__DragonFly__)
617 ((struct __FILE_public *)iob)->_r -= c;
618 ((struct __FILE_public *)iob)->_p += c;
631 adios(NULL, "m_getfld() called with bogus state of %d", state);
635 msg_count = cp - buf;
641 static char unixbuf[BUFSIZ] = "";
651 register char *delimstr;
654 ** Figure out what the message delimitter string is for this
655 ** maildrop. (This used to be part of m_Eom but I didn't like
656 ** the idea of an "if" statement that could only succeed on the
657 ** first call to m_Eom getting executed on each call, i.e., at
658 ** every newline in the message).
660 ** If the first line of the maildrop is a Unix "From " line, we
661 ** say the style is MBOX and eat the rest of the line. Otherwise
662 ** we say the style is MMDF and look for the delimiter string
663 ** specified when nmh was built (or from the mts.conf file).
666 msg_style = MS_UNKNOWN;
669 if (fread(text, sizeof(*text), 5, iob) == 5
670 && strncmp(text, "From ", 5) == 0) {
672 delimstr = "\nFrom ";
674 while ((c = getc(iob)) != '\n' && c >= 0)
678 while ((c = getc(iob)) != '\n' && cp - unixbuf < BUFSIZ - 1)
683 /* not a Unix style maildrop */
684 fseek(iob, pos, SEEK_SET);
685 if (mmdlm2 == NULL || *mmdlm2 == 0)
686 mmdlm2 = "\001\001\001\001\n";
690 c = strlen(delimstr);
691 fdelim = (unsigned char *) mh_xmalloc((size_t) (c + 3));
694 msg_delim = (char *)fdelim+1;
695 edelim = (unsigned char *)msg_delim+1;
698 strcpy(msg_delim, delimstr);
699 delimend = (unsigned char *)msg_delim + edelimlen;
701 adios(NULL, "maildrop delimiter must be at least 2 bytes");
703 ** build a Boyer-Moore end-position map for the matcher in m_getfld.
704 ** N.B. - we don't match just the first char (since it's the newline
705 ** separator) or the last char (since the matchc would have found it
706 ** if it was a real delim).
708 pat_map = (unsigned char **) calloc(256, sizeof(unsigned char *));
710 for (cp = (char *) fdelim + 1; cp < (char *) delimend; cp++ )
711 pat_map[(unsigned char)*cp] = (unsigned char *) cp;
713 if (msg_style == MS_MMDF) {
714 /* flush extra msg hdrs */
715 while ((c = Getc(iob)) >= 0 && eom(c, iob))
724 ** test for msg delimiter string
728 m_Eom(int c, FILE *iob)
730 register long pos = 0L;
738 if ((i = fread(text, sizeof *text, edelimlen, iob)) != edelimlen
739 || strncmp(text, (char *)edelim, edelimlen)) {
740 if (i == 0 && msg_style == MS_MBOX)
742 ** the final newline in the (brain damaged) unix-format
743 ** maildrop is part of the delimitter - delete it.
748 fseek(iob, pos, SEEK_SET);
751 fseek(iob, (long)(pos-1), SEEK_SET);
752 getc(iob); /* should be OK */
756 if (msg_style == MS_MBOX) {
758 while ((c = getc(iob)) != '\n')
763 while ((c = getc(iob)) != '\n' && c >= 0 && cp - unixbuf < BUFSIZ - 1)
775 ** Return the Return-Path and Delivery-Date
776 ** header information.
778 ** Currently, I'm assuming that the "From " line
779 ** takes one of the following forms.
781 ** From sender date remote from host (for UUCP delivery)
782 ** From sender@host date (for sendmail delivery)
786 get_returnpath(char *rp, int rplen, char *dd, int ddlen)
788 char *ap, *bp, *cp, *dp;
791 if (!(bp = cp = strchr(ap, ' ')))
795 ** Check for "remote from" in envelope to see
796 ** if this message uses UUCP style addressing
798 while ((cp = strchr(++cp, 'r'))) {
799 if (strncmp(cp, "remote from", 11) == 0) {
800 cp = strrchr(cp, ' ');
806 ** Get the Return-Path information from
807 ** the "From " envelope.
810 /* return path for UUCP style addressing */
811 dp = strchr(++cp, '\n');
812 snprintf(rp, rplen, "%.*s!%.*s\n", (int)(dp - cp), cp, (int)(bp - ap), ap);
814 /* return path for standard domain addressing */
815 snprintf(rp, rplen, "%.*s\n", (int)(bp - ap), ap);
819 ** advance over the spaces to get to
820 ** delivery date on envelope
825 /* Now get delivery date from envelope */
826 snprintf(dd, ddlen, "%.*s\n", 24, bp);
834 static unsigned char *
835 matchc(int patln, char *pat, int strln, char *str)
837 register char *es = str + strln - patln;
840 register char *ep = pat + patln;
841 register char pc = *pat++;
850 while (pp < ep && *sp++ == *pp)
853 return ((unsigned char *)--str);
859 ** Locate character "term" in the next "cnt" characters of "src".
860 ** If found, return its address, otherwise return 0.
863 static unsigned char *
864 locc(int cnt, unsigned char *src, unsigned char term)
866 while (*src++ != term && --cnt > 0)
869 return (cnt > 0 ? --src : (unsigned char *)0);