MM edit history - Frank da Cruz <fdc@columbia.edu>

Beginning with MM-0.90.4 (Notes are chronological, newest stuff at the bottom).

Warning: Includes learning experiences and false starts.

------------------------------
Wed Jan  9 19:42:31 2002

Short story: I added some code to MM to address the truncating problem
that occurs when you edit an outgoing message, save the result, and
exit from the editor.

Long version:

I spoke with Melissa about MM and found out where the latest (patched) 
sources were:

  /src/acis/ccmd/src
  /src/acis/mm/obj/solaris25

I made an ~fdc/mm/ directory on prost with mm and ccmd subdirectories.
Copied all the files from the /src tree.  Checked that they built OK; in
~/mm/ccmd:

  make solaris

In ~/mm/mm (it looks to me like the current Makefile unconditionally defines
SOLARIS?):

  make

This makes an MM 0.90.4(xx) binary:

  Columbia MM, version 0.90.4(24)
  Copyright (c) 1986, 1990
	  The Trustees of Columbia University in the City of New York
  Compiled by fdc@watsol on 9 Jan 2002 6:16pm
  Report bugs to bug-mm
  
Now the truncating problem: The editor is run by mm_execute() in exit.c,
which is just a fork() and execvp().  Then the upper fork calls
wait_for_process(pid), which calls do_wait(pid, true), where "true"
means "blocking".  do_wait() might need looking at but first assume it
works as advertised.  Where is Howie's sleep()?  Nowhere.

I added an msleep() (millisecond sleep) routine to datime.c, for Solaris
only.  Uses select().  Builds without complaint.

Then I added a chk_edit(filename) routine to edit.c, which sleeps for
half a second and then in a loop, gets the size of the filename,
compares it with the previous size, and if they are not equal, sleeps
for 1/10 second and tries again, up to 25 times.  If, after 25
iterations, the size still doesn't settle down, or some other bad thing
happens, chk_edit() returns < 1.

Then I found all the places in edit.c where mm_execute() is called with
editor[0], and after each of these, I called chk_edit() on the temp file.
If chk_edit() returns < 1, we set a trouble flag, which is supposed to
inhibit deletion of the temp file.

The very first time I tried it, I caught it in the act:

  (Exit from EMACS here)
  /p/kd/fdc/.mm-outgoing.25233: 0 6263 (BAD)
  /p/kd/fdc/.mm-outgoing.25233: 6263 6263 (OK)
  S>

This seems to exonerate wait_for_process() (e.g. of killing EMACS
prematurely).  The messages won't appear unless there is at least one
size mismatch.  If the size doesn't settle down after 3 seconds (500
msec + 25 x 100 msec), it prints "Possible problem with edit, temp file
saved: " and the filename.

I didn't notice any kind of edit history file, but a copy of this message
is in ~fdc/mm/NOTES.fdc.

I'll be the guinea pig, feel free to join in; time will tell.

- Frank

P.S. The fact that the BUG command puts "SunOS" in the Bug-Report:
header doesn't really mean SunOS -- the Solaris versions really are
compiled for Solaris, but the real name of Solaris is SunOS.

P.P.S. This message itself was truncated when I exited from EMACS, but
the new code fixed it automatically.

P.P.P.S. Melissa or Postmaster, I guess you can add me to the bug-mm list :-)

------------------------------

Date: Thu, 10 Jan 2002 9:40:01 EST
From: Melissa Metz <melissa@columbia.edu>
To: Frank da Cruz <fdc@columbia.edu>
Cc: bug-mm@columbia.edu, kermites@columbia.edu, unixsys@columbia.edu,
        postmaster@columbia.edu, Vace Kundakci <vace@columbia.edu>,
        Walter Bourne <walter@columbia.edu>
Subject: Re: MM truncating problem maybe fixed?
In-Reply-To: Your message of Wed, 9 Jan 2002 20:20:27 EST
Message-ID: <CMM.0.90.4.1010673601.melissa@broccoli>

Cool.  

> I didn't notice any kind of edit history file, but a copy of this
> message is in ~/mm/NOTES.fdc.

We kept track of the edit histories in RCS.  e.g. cd /src/acis/mm/src,
co -l edit.c, <edit>, ci -u datime.c, <type in comment>.  The comment
can be as long as you like; you can use "ci -u edit.c <filename" to
include a file.

I'll put your version up in a couple of days.  But let's get your
changes into the /src tree first :-).

> P.S. The fact that the BUG command puts "SunOS" in the Bug-Report:
> header doesn't really mean SunOS -- the Solaris versions really are
> compiled for Solaris, but the real name of Solaris is SunOS.

Right, uname also reflects the name SunOS.  Sun claims that Solaris is
the whole operating system and all applications, while SunOS is just
the kernel.
                                        --Melissa
------------------------------
Thu Jan 10 09:24:55 2002

Various refinements and tuning, #ifdef adjustment, message improvement, etc.

The main impact of these changes is (a) there is always at least a 500-msec
delay after returning from EMACS, and (b) if you return from EMACS with an
empty file, there's a longer delay because MM has no way of knowing if the
file is empty because it hasn't been written yet (this has happened to me
many times), or because you really saved an empty file.

read_from_temp(), which is called after every editor invocation, executes an
fsync() on the file to forestall truncation, but doesn't.  I suspect fsync()
has no effect on a file open with O_RDONLY.  The Solaris 2.5.1 man page
suggests as much:

  ERRORS
     EBADF: fildes is not a valid file descriptor open for writing.

although in Solaris 2.5.1 fsync() on a read-only file returns 0 and does not
set errno.  The Solaris 8 man page, on the other hand, says "EBADF: The fildes
argument is not a valid file descriptor".  I wonder if EMACS does an fsync(),
and if it does, how can MM be reading partial files?

------------------------------
Sat Jan 12 15:25:07 2002

At some point in the last few years, MM started adding a blank line to the end
of every message after returning from the editor.  Off-by-one bug at the end
of read_from_temp(), fixed in edit.c.

Changed copyright date from "1986, 1990" to "1986, 2002".  This makes it a bit
more obvious that it is being maintained, handles Y2K, etc.  I made this
change only in misc.c to keep the differences small.

------------------------------
Sun Jan 13 11:34:11 2002

Changed #ifdefs around new code from #ifdef SOLARIS to
#if defined(BSD) || defined(_AIX) || defined(SVR4), since (a) this is what
is used for dttzone(), which depends on the same struct definitions (timeval
and timezone) and functions (gettimeofday()), and (b) to include SunOS, which
also has/had the truncating problem.  datime.c, edit.c.

But surprisingly, this did not pick up Solaris -- I assumed that defining
SOLARIS would imply SVR4 but it doesn't, so I changed the #ifdefs for the
new code to:

  #if defined(BSD)  || defined(SOLARIS) || defined(SVR4) || defined(_AIX)

But I did not change the #ifdefs around dttzone().  datime.c, edit.c.

Tried building for SunOS:
  ccmd: "make clean; make sun" failed at link time, but then worked if I
    manually deleted ccmd.a, test.o, and skel.o and did "make sun" again.
  mm: "make clean; make" fails because of the hardwired SOLARIS definition
    in Makefile.  Copied the Makefile out of /src/acis/mm/obj/sunos41 and
    used that instead and mm build ok.

Tried building for Solaris 8 on newcunix in /p/kd/fdc/mm/solaris8/:
  ccmd: "make clean; rm -f ccmd.a test.o skel.o; make solaris" (ok)
  mm:   "make clean; make" (ok)

Tried building for HP-UX 11.20 but it bombed right away with undefined
symbols.  Didn't try fixing or building on AIX or anywhere else.

In heavy testing and real use of MM over the past several days, I have not
experienced a single truncation (normally I have 3-4 per day).  However, after
rearranging the code on Thursday, I also have not seen a single instance of
the file size changing.  The 500-msec pause appears to be sufficient, but the
subsequent size-checking loop is still there as a precaution (e.g. for high
load, memory full, swap-space low).

I'd suggest the MM version number be bumped up and the new source replace
the old source on the public FTP server.

------------------------------
Other things on my list:

Maybe add a SET command to enable/disable chk_edit() since most people
don't need it.

Check and enable or fill in msleep() for other platforms.

Character sets, when sending a message [done, see below]:

 . Don't strip 8th bit from text (what is 8th bit used for now?)
 . SET CHARACTER-SET <keyword> (default US-ASCII), SHOW CHARACTER-SET
 . Add header: Content-Type: text/plain; charset="iso-8859-1" or whatever.
 . Add header: Content-Transfer-Encoding: 7bit (or 8bit) according to charset.
 . Add header: MIME-version <-- get right syntax.

MM shouldn't "do" anything with character sets, just let the user declare
which one is used in the outbound message and not strip the 8th bit from text,
Subject, or other header data characters.  The SET CHARACTER-SET command would
take a keyword list of MIME charset names for only those MIME charsets that
are also international standard sets (i.e. ISO registered), since it's impolite
and antisocial to send email in proprietary character sets; an internal table
tells whether each one is 7bit or 8bit.  The commands and headers are easy,
but figuring how to undo whatever is currently done with the 8th bit is hard.

Slightly more ambitious tasks:

 . Add switches (cmswi()) to some of the commands; for example:
   COPY /APPEND /REPLACE /BODY-ONLY

 . COPY and MOVE to pipelines.

 . Hooks to Kermit, e.g. COPY /DOWNLOAD [done, see below]

 . Allow definition of macros

 . Allow macros to accept parameters

 . Case-sensitive text searches.

 . Pattern or regexp text searches.

 . Devise a way of accomplishing OR in message selection; currently
   conditions are all connected by implied ANDs.  This would also
   require syntax for grouping.

 . Add an option that tells MM, when it reads a message and finds a
   charset header, to send the corresponding ISO 2022 escape sequence
   to switch the terminal into the character set used by the message or,
   if there is no charset header, to switch it back into its prevailing
   charset (such as Latin-1).  Of course this can work only with standard
   ISO-registered character sets, not with (e.g.) Windows code pages or
   random MIME charsets, and only with ISO-2022 / ANSI X3.64 compliant
   terminals (e.g. VT220 or 320).

The latter would require MM to have to several distinct charset settings:

 . The default (global) charset for sending 

 . The default charset for reading (when message doesn't specify one)

 . A way to send a message with a given charset without affecting the
   global one (e.g. SEND /CHARACTER-SET:blah).

 . A way to tell MM, when replying, whether to automatically choose
   the same charset as the original message (again, without affecting
   the global outbound charset), e.g. REPLY /SAME-CHARSET or something,
   plus a global setting for whether this should be the default.

Also:

 . Consolidate the many makefiles and make a master makefile that builds
   both ccmd and mm [done, see below].

 . Clean up compiler warnings [mostly done].

 . Check/update/fix ports to HP-UX, AIX, FreeBSD, etc.

Definitely NOT on my list:

 . Enclosures (if I want this I'll use another mail client!)
 . Desktop metaphors (folders, in-boxes, address books, "My New Messages",
   "My Favorite Friends", etc -- fooey, ditto!)

------------------------------
Wed Jan 16 13:12:29 2002

A message was lost -- first time in nearly week.  I was replying to a message
and editing the reply.  I was typing really fast, including the ^X^S^X^C to
save and exit EMACS.  In this case the MM S> prompt appeared immediately, with
no 0.5sec pause.  Therefore there must be a path thru the code in which
chk_edit() is not called.  (Later the same day several more messages were
truncated; same deal: no pause.)

------------------------------
Wed Jan 23 14:02:56 2002

A reply was totally lost -- 0 bytes -- but this time MM paused for the
full 3 seconds.

------------------------------
Sun Feb  3 11:20:12 2002

IDEA: add SET TMPDIR to allow temp files to be written on a local disk.
e.g. "set tmpdir /tmp/fdc".  Create it if it doesn't exist.  Give it 600
perms.  But what if it does exist and is not owned by the MM user?  Give
an error message.  Give all temp files mode 600 on open.

------------------------------
Mon Feb  4 16:53:13 2002

Many composed messages & replies lost.  A big problem is that MM always
deletes the .mm-xxx tmp file after reading it in (unless it *knows* there's
a problem).  Not deleting it would at least let us get it back but then:
(a) successive messages use the same filename (.mm-outgoing-<pid>), and (b)
when *would* we delete them?  <-- When MM EXITs.

------------------------------
Mon Apr 15 16:20:53 2002

Built Ken Harrenstein's updated version (see KLH.NOTES) and started running
it in debug mode ("make fdc-solaris").  It built fine on Solaris, works fine
so far.  When exiting from EMACS, we always get the debug message: "wait3
errno 2" (no such file or directory).  Looked through sources again for any
invocations of the editor that do not call chk_edit() afterwards; didn't find
any.  Added more debugging statements.

------------------------------
Tue May 14 11:54:37 2002

Melissa was getting more truncations.  Replaced the version of
read_from_temp() that does calloc() and then (potentially) a bunch of
rmallocs() with the original simple version that does just one malloc(), and
added debugging after every operation.  Suspicion: read() and malloc() take
an int, but the old code was calling it with sbuf.st_size, which is long.
Changed the code to (a) convert the long to an int, (b) check that (long) of
the int is equal to the original long, and then (c) to use the int in all
the malloc() and read() calls.

------------------------------
Thu May 16 19:20:52 2002

Melissa was still getting truncations.  The messages indicated that
everything in read_from_temp() was working right.  But upon return the
message is empty.  That means that the pointer to the message is not being
returned right.  Aha.  The last statement in read_from_temp() is
return(newtext).  But newtext is on the stack; made it static (it shouldn't
matter but you never know).  Added debugging checks and messages to
read_from_temp()'s callers.

------------------------------
Tue May 21 17:44:21 2002

Replying to message, entered EMACS, edited the reply, exited EMACS, debugging
messages showed everything normal, yet the message was truncated to 0 bytes.
There should be NO path through the code where that can happen without a
message.

At this point we still have two problem cases:

 1. Debugging messages all come out normal, but file is stil truncated.
 2. There are no debugging messages at all, and file is truncated.

------------------------
Sun Jul 21 15:15:47 2002

Fixed stupid debugging message for empty temp file:
  Text length mismatch [gnuemacs_edit_outgoing]: 0 != 0.

Added more debugging messages, including a message when a missing final
terminator is supplied in case this correlates with truncation.  edit.c.

Added "set user-temp-files-keep" (keyword chosen so it goes at end of list,
since it wasn't clear to me how to insert a new keyword -- do I have to change
all the #defines in set.h?).  Made the default for this "yes" if FDC_EDITFIX
is defined, since otherwise what good is it?  set.h, set.c, extern.h.

Filled in action for "set user-temp-files-keep yes" -- just check the boolean
variable for this setting in purge() and change all open()s for temporary
files to include O_TRUNC (since we keep recycling the same name).  Then in
cmd_quit(), delete the temp files (i.e. when user says BYE or QUIT).  Thus
after editing a message, the temp files stay around until the next time you
enter the editor from within MM, or you (really) exit from MM, so now if any
message is truncated after editing, you can get it back from the temp file --
and if the temp file itself is truncated, we know it's the editor's fault and
not MM's.  edit.c.

------------------------
Fri Sep 13 16:06:03 2002

Enclosed all verbose debugging messages in edit.c in #ifdef VERBOSE_MESSAGES
except "Temp file kept: <name>" in gnuemacs_edit_outgoing().

------------------------
Sat Sep 14 13:31:27 2002

The 8th bit of data characters seems to be used by CCMD to mark whether each
character was (or is to be) echoed or quoted or something; see CC_CHR,
CC_QUO, etc, in ccmd.h and mm/parse.c.  But this stuff should apply to
commands, not data text.

 1. Send, ^E, create 8-bit text in EMACS, exit from EMACS, send: Stripped.
 2. Send, ^B, insert 8-bit file, send immediately: Stripped.
 3. Send, type 8-bit chars directly into text buffer: Stripped.

In case 3, it's obviously because the text is being read by CCMD.  But in
cases 1 and 2, there must be some code that zips through the new text and
strips off the bits.

When EMACS returns (and also when "Insert file:" inserts data from a file),
the data is still 8-bit, so the bits are being stripped later.

In cmd_send() in sendcmds.c, upon return from get_msg(&outgoing), the 8th bits
are gone.  OK, so get_msg() sets up Ctrl-E handler (editaction()) and then
calls parse(), which calls the editor if I type Ctrl-E, but then strips the
bits.  BUT... if you "set use-editor-always", parse is not called, and the
bits are not stripped.  This works even when replying to messages.

------------------------
Sun Sep 15 14:19:48 2002

Adding new header items...

The idea is that normally these would be omitted, but if the user gives a
CONTENT-TYPE command, the other two would be filled in too, using default
values.  Use "reply-to" as a model.  Commands for the others would be
available to override the default values.

message.h:
[ok]  Add symbols for new headers:
      #define MIME_VERSION        21
      #define CONTENT_TYPE        22
      #define TRANSFER_ENCODING   23
[ok]  Add new header members to struct mail_msg definition.

parsemsg.c:
[ok]  Add header keywords to header_keys[] -> header_tab -> header_fdb...
[xx]  Add cases to parse_msg()... (not necessary).

cmds.H:
[ok]  Add CMD_ symbols for CONTENT-TYPE, MIME-VERSION, and TRANSFER-ENCODING.

cmds.c:
[ok]  Add CONTENT-TYPE, MIME-VERSION, and TRANSFER-ENCODING to send_keys2[].
[ok]  Add them to hdr_keys[] too.
[ok]  Add prototypes.
[ok]  Add to dispatch table mm_cmds[].

display.c:
[ok]  Add cases to cmd_display().

send.c:
[ok]  Add action routines: cmd_content_type(), etc.

NOTE: "help header-fields" is missing.

------------------------
Mon Sep 16 10:14:55 2002

Updated copyright date in VERSION text, misc.c.

------------------------
Tue Sep 17 10:11:38 2002

Fixed Linux build with help from <ishikawa@yk.rim.or.jp> and <azz@gnu.org>...

ccmd/Makefile
  test and skel targets should not have hardwired -ltermlib.
  Added TERMLIB symbol to ccmd Makefile to handle this.

ccmd/machdep.h
  #define HAVE_BSTRING and HAVE_TZ_SYSV for Linux.

mm/Makefile
  In Linux, link with ncurses, not termlib; handled in Linux target.

mm/sysh/s-linux.h
  define NEED_TIME and HAVE_BSTRING

mm/config.h
  Contains hardwired #include "sysh/s-sun58.h".  Noticed that there are a
  bunch of config.h-blah files, where blah == platform-name, and each one
  #includes a different "sys/s-blah.h".  Made one for Linux (and while I was
  at it, for HP-UX, AIX, etc, but these are probably not right).  Then I
  changed each target in the MM makefile to link config.h to the appropriate
  config.h-blah.

We still get warnings about mktemp() but so what; unfortunately mkstemp()
is not a compatible replacement, so if somebody really cares about this,
they'll have to write an mm_mktemp() routine that handles both cases.
Anyway, once I installed movemail in /usr/local/lib/mm/ Linux MM works fine.

------------------------
Tue Sep 17 20:04:29 2002

Adding a new command...

DOWNLOAD command for Read Mode, plus SET/SHOW DOWNLOAD-FILTER.
  Works just like PRINT and PRINT-FILTER but downloads rather than prints.

cmds.H: 
 [ok] Add CMD_blah symbol (in alphabetical order).

cmds.c:
 [ok] Add DOWNLOAD keyword to mm_top_keys_2[].
 [ok] Add DOWNLOAD keyword to mm_send_keys_inv[].
 [ok] Add DOWNLOAD keyword to mm_read_keys_2[].
 [ok] Add prototype for cmd_download().
 [ok] Add cmd_download function pointer to mm_cmds[].

print.c:
 [ok] Copy cmd_print() to cmd_download() with minor edits.
 [ok] Modify cmd_download() to have a /FILENAME: switch.
 [ok] Modify cmd_literal() to allow LITERAL DOWNLOAD.

set.H:
 [ok] Define SET_DOWNLOAD_FILTER

set.c:
 [ok] Define default download_filter = "kermit -s - -a %s".
 [ok] Add "download-filter" to set_cmds.
 [ok] Add to download_filter to set_variables[].

extern.h:
 [ok] Add download_filter to list of extern strings.

Combined notes on new features into release-0.91.txt.

Moved old makefiles into mm/old/ and ccmd/old/.

Updated help files.

------------------------
Wed Sep 18 12:39:40 2002

Changed default download filter to "kermit -Ts -s - -a %s" (added the 'T')
to ensure that Kermit downloads in text mode.  set.c.

Changed DOWNLOAD command to allow up to four %s's in the download-filter
(and to fill them in without overflowing any buffers) for the benefit of
setups like "set download-filter cat > %s; sz %s; rm %s".  print.c.

Figured out how to set the breakmask for the _CMFLD call in cmd_download()
to allow all printable characters (except ?), so now you can type
DOWNLOAD /FILE:c:\foo$bar_baz\xxx.eml or /FILE:~/mail/xxx.tml.  print.c.

I noticed an apparent problem with the HEADERS command -- dates were coming
out wrong.  I thought I had broken something, even though I never went near
any code that deals with dates.  It turns out to be a longstanding problem.
Quoting Melissa:

  "This is the creeping-DST bug.  It hasn't always been there.  We've fixed
  it many times.  But DST is pretty slippery.  MM misinterprets the DST
  status of the message, so each time it writes out the file, you quit and
  restart MM, then read the file again, the time creeps +/- 1 hour.  Maybe a
  difference in the DST or time() library or timezone setting or something?

  "If you are restarting MM a lot (e.g. debugging), the message creeps 1
  hour each time, and after 31 iterations 9/17 16:00 (GMT) becomes 9/16
  09:00.  You can [see the effect] by copying a file, reading it in, writing
  it out again, and running diff to see if the time changed.

  "You could 'normalize' the envelope times by moving all the messages into
  a no-envelope format like RMAIL, then reading it back in with MM (which
  will initialize using the Date: header) and writing it back out in your
  desired format."

Envelope format refers to mbox-style message files in which each message
starts with a "From " line.  MM uses this line, rather than the Date:
header, to determine the message's timestamp.  The bug is the MM
recalculates the timestamp incorrectly and rewrites the "From " line every
time it writes out the mail file.

------------------------
Thu Sep 19 10:56:36 2002

When a CONTENT-TYPE command forces creation of a MIME-Version header, put
the MIME-Version header first.  It shouldn't make a difference, but all the
other mail clients put the MIME-Version header before any other MIME-related
headers so why take chances.  send.c.

Back to DST creep...  Copying my mail file and then writing out a new copy
from MM does indeed show all the "From " lines have changed by 5 hours!
This doesn't look like a DST difference -- it looks like UCT (GMT) versus
local time.  Turns out the problem was in ccmd/machdep.h.  The definition of
HAVE_TZ_SYSV wasn't taking because CCMD_OS_SOLARIS wasn't included among
the platforms that select it.

------------------------
Fri Sep 20 11:40:01 2002

Builds:

Built on Solaris 2.5.1, Solaris 8, Solaris 9, Red Hat Linux 7.1, FreeBSD 4.4.
Some of these required minor tweaks to mm/Makefile and creation of additional
config.h-blah files.

Failed to build on OpenBSD 3.0:

  cc -g -DCCMD_OS_BSD44  -c cmgrp.c
  cmgrp.c: In function rphelp':
  cmgrp.c:115: structure has no member named 'gr_name'
  etc etc...

The HPUX build bombed right away because CCMD uses ANSI C features but the
default cc on HPUX is the bundled non-ANSI one.  Changed the makefile targets
to require the ANSI compiler and it gets a little farther but still bombs.

------------------------
Tue Sep 24 13:43:54 2002

Added new mm/Makefile targets for NetBSD and OpenBSD that are like BSD44 and
FreeBSD but add -Dunix; this fixes the OpenBSD and NetBSD builds because their
cc does not define "unix".

I have been using MM as heavily as ever since Sept 14th, when I switched to
"use-editor-always" and have not lost a single byte.  I wonder if I lose that
option, will I start losing text again?  If so, the culprit is parse()...

------------------------
Sat Oct  5 13:03:06 2002

Linda Wallinger noticed that when you remail a message, the Content-Type
header disappears.  Further testing shows the same thing happens to the
MIME-Version header.  Diagnosis: the switch() statement in parse_msg()
didn't have cases for the three new headers.  Fixed in parsemsg.c.

------------------------
Sat Nov  2 11:20:37 2002



------------------------
Wed Nov  6 09:46:24 2002

Suddenly about 9:30 this morning, MM started adding From lines that were
20 hours off:

  From fdc@columbia.edu Tue Nov  5 13:37:57 2002
  Date: Wed, 6 Nov 2002 9:37:52 EST
  From: Frank da Cruz <fdc@columbia.edu>
  To: ...

Spamassasin rejects messages that have non GMT-offset timezones.
Fix MM to put dates in RFC2822 format.

------------------------
From Jeff on watsun:

while sitting at the command prompt on watsun.  mm core dumped
There is a core file in ~jaltman/core

Here is the back trace:

#0  0xef77636c in read ()
(gdb) bt
#0  0xef77636c in read ()
#1  0xef77ab90 in _filbuf ()
#2  0x443e8 in cmgetc (c=0xeffff958, fd=0xef7e7248) at ccmdmd.c:226
#3  0x41574 in fill (btab=0x748b4, brk=0xeffff9db "\n") at ccmd.c:854
#4  0x407a0 in parse (fdblist=0x7316c, value=0x7a260, usedfdb=0x7a258)
    at ccmd.c:437
#5  0x1fc84 in top_level_parser ()
#6  0x1f4a0 in main ()

------------------------
A way for users to fix stale locks?
Melissa: "No, actually, MM uses kernel locking over NFS.  Contrary to the
rumors, it works.  Except when the locks get left around."

------------------------
Mon Jan 27 09:45:21 2003

Rebuilt to use /opt/ACISmm/ paths.

------------------------

Date: Tue, 4 Mar 2003 15:25:03 EST
From: Eric Weaver <eric@columbia.edu>
To: bug-mm@columbia.edu
Cc: eric@columbia.edu
Subject: multiple Bcc's not processed
Bug-Report: Bug in Columbia MM, version 0.91.0(140) (SunOS)

Hi,
  If one puts multiple "Bcc" headers in their outgoing mail (like when editing
the body and headers in an editor), then only the last "Bcc" header gets
processed.  For example:

        From: eric
        To: joe
        Bcc: bob
        Bcc: tom

will only go to joe and tom.  Bob will not get a copy.  After composing and
getting back to the send prompt, a "display" will show the multiple "Bcc"
headers, but only the last one gets processed.
  Oh, to clarify a related item,... if one specifies multiple "bcc user[, user
[, ...]]" commands on MM's command line (like at the send prompt), then that
method works ok because MM puts all the users into one Bcc header.  /Eric

-----
Miscellaneous to do or check:

 . Subject: mm set dont-type-headers
   Expand buffer size or max count for this, we need a lot more these days.

 . Add X-Mailer: header?  and/or User-Agent:

 . Idea: "type [ | filter ]"

 . Stale locks.  Why?  MM catches SIGHUP and does the right thing.  The 
   problem is that when somebody's dialup connection drops, SIGHUP never comes.

 . Add an "mm" command that does nothing at top level, and goes to top level
   if at Send or Read level.

 . Make X-Spam-Score a known header so we don't have to search the whole
  text for it.

 . HEADERS ENTIRE-SUBJECT (as opposed to "subject contains")
   Ditto for FROM, etc.  Maybe make this a switch...  /FULL-MATCH ?

 . HEADERS RECEIVED

 . "keyword junk keyword vacation" gets a parse error if no messages have a
   "vacation" keyword.  This should just be text, not a keyword.  Especially
   since if it happens in a command file, it prevents the rest of the
   command file from working (the real error here is that command retry is
   active when the command source is not the terminal).

 . Add a command to enable translation of all correctly-tagged messages to
   UTF-8.  Perhaps by automatically choosing a pager like iconv | more.

 . Where is the mbox From line format specified?  RFC 976 (UUCP mail
   interchange format), section 2.4, "Envelope".  Not really.  RFC 822
   mentions the need for an envelope but does not describe it.  MRC says
   that at least the following formats have been seen, and nothing is
   written anywhere about what the default timezone is:

    *              From user Wed Dec  2 05:53 1992
    * BSD          From user Wed Dec  2 05:53:22 1992
    * SysV         From user Wed Dec  2 05:53 PST 1992
    * rn           From user Wed Dec  2 05:53:22 PST 1992
    *              From user Wed Dec  2 05:53 -0700 1992
    * emacs        From user Wed Dec  2 05:53:22 -0700 1992
    *              From user Wed Dec  2 05:53 1992 PST
    *              From user Wed Dec  2 05:53:22 1992 PST
    *              From user Wed Dec  2 05:53 1992 -0700
    * Solaris      From user Wed Dec  2 05:53:22 1992 -0700

   Therefore MM, to be general and portable, should support all these
   formats and be configurable per site (and possibly also per user) about
   defaulting the timezone.

From MRC:  "MS Source is in:

  ftp://ftp.cac.washington.edu/mail/ms.tar.Z

It's a baby MM that uses the c-client library.  All the basic stuff is
there, but it's missing a lot of more advanced facilities of both MM and
of the c-client library (MIME, sorting, threading...)."

------------------------------
Thu Oct  9 16:27:15 2003

Date creep, cont'd...  This time because of the timestamp in the From line.
Columbia's mail agent puts:

   From fdc@columbia.edu  Thu Oct  9 13:35:31 2003

where the time is local.  The problem is that when MM rewrites the "From "
line, it changes it to GMT.  I traced this to mkdate() in mm/dt.c.  The
#ifdefs did not pick up any of the timezone code in Solaris.  Rather than
mess with the #ifdefs and risk breaking something else, I simply tacked
"|| defined(MM_OS_SOLARIS)" onto the selection clauses that import and check
the external timezone variable.  This seems to fix the problem:

 . Wait for a message to arrive in incoming spool directory.
 . Check its "From " line.
 . Read the message with today's mm.
 . Write out the mail file.
 . Check the same message's From line

Now it's the same, rather than 4 hours different.  It works the same
no matter whether built in Solaris 2.5.1 or Solaris 9.

Still Needed:
 . There is no code to parse GMT +/-hhmm format; I think this could be
   handled by adding "+0100", "-0300", etc, and the corresponding values
   to the timezone keyword table.

Other Changes:
 . Added UTC as a synonym for GMT in the timezone table.
 . Merged solaris-cu and (bogus) cu-solaris targets.

------------------------------
Fri Oct 17 14:55:30 2003

In new_mail() [newmail.c] added #ifdef COLUMBIA..#endif code to default mail
spool directory to SPOOL_DIRECTORY/a/b/abc123 if /var/spool/.splitmail exists,
where abc123 is any user ID (one-char IDs are handled as .../a/a/a).

------------------------------
Mon Oct 27 13:09:36 2003

Checked envelope date after DST->EST switchover; still OK.

Changed default In-Reply-To: value to be the replied-to message's message-id
(instead of "Your message of <date>"), which is needed for Remedy and also
recommended ("SHOULD") by RFC 2822.

------------------------------

Date: Thu, 22 Apr 2004 7:48:02 EDT
From: Eric Weaver <eric@columbia.edu>
To: bug-mm@columbia.edu
Cc: eric@columbia.edu
Subject: [sys@rg03.ais.columbia.edu: New system errors for rg03 as of
        04-21-2004 09:30PM]

I think MM's sense of which day is it was messed up by the change to
Daylight Saving Time... The message below should be "yesterday", but it is
listed when I get a list of messages for "today".  Am I missing something?
/Eric

----------
Misc ideas not acted upon:

Add HEADERS ADDRESS (matching all messages whose to, from, or cc lists
contain the address -- similar to HEADERS TO).

Print a hint when mail file can't be opened because of lock?  What should
the hint say?  To run some variation of the "killme" script...

See: ~/mm/mail.txt message from ron@isri.unlv.edu

Idea for MIME charsets...  make iconv or whatever part of the crt-filter
pipeline.  Still have to set terminal charset so iconv can convert *to* it.

But then what about replying, etc?

Make MOVE and similar command default to "previous sequence" if the previous
command was "headers" or other command that created a sequence; otherwise to
current message.

Change DOWNLOAD to allow a filename as argument, or default to x.eml or
something (not mm_msg).

Add SET RECENT to define what "recent" means, e.g. last 5 days or "since
yesterday".

Make "previous-sequence" the default message sequence for COPY, etc.

Need "set include-mime-headers always" so I don't have to remember each
time.

Fix "content-type" to allow specification of HTML, etc.

Add HEADERS ADDRESS to list all messages sent from or to the given address
or person.

Implement the FOLLOW command. Given a message number, it constructs a
message sequence showing the whole thread.  Depends on In-Reply-To: showing
the Message-ID.  It can go backwards, forwards, or both.  To go forward:
find all messages that are In-Reply-To the Message-ID of the given message,
then find all the messages that are In-Reply-To the Message-IDs of each of
those messages, etc.  To go backward, find the message that this message is
In-Reply-To.  Find all the other messages that are In-Reply-To that same
message.  For each of these, find the messages that THEY are In-Reply-To, etc.

---------------------------------
Fri Aug 26 10:49:57 2005

Downloaded Nelson Beebe's mm 0.94 from Utah.
He did a very good job on the makefiles, adding a new top-level one.
You can find it in:

  ftp://kermit.columbia.edu/kermit/mm/utah/mm-0.94.tar.gz

I built it on Solaris 9 with "make OS=solaris2.x all".
It built OK (but it doesn't with the default OS=solaris10).
Started it and it worked OK, but, contrary to what Nelson
had said, it didn't have any of my changes.  Sent him email repeatedly
about this and he never answered.

------------------------
Thu Dec  1 18:09:27 2005

Made crude and minimal modifications to allow MM to use a replacement
for movemail that acts as a pop client:

.mmint
  set movemail-path /p/kd/fdc/scripts/pop.ksc

init.c:
  user_init() looks for .mm-use-pop or .cyrustt file in home directory.
  If found, sets global use_pop variable.
  The difference is:
  .mm-use-pop will go to the old pop server.
  .cyrus will go to the new one.

mm.c:
  After calling user_init(), if use_pop is set, prompts for POP password
  immediately upon startup.  The maximum length is 255.  Added a prompt_pw()
  routine to do this.  The password is kept in MM's memory and used every
  time the "movemail" program is called.

newmail.c:
  if use_pop set, make an argument vector for the POP script rather
  than regular movemail (first arg is dummy, second is ~/.mm-newmail,
  as always, and new third argument is password).  Skip over code
  that looks at the mail spool, and always call the movemail program.

------------------------
Fri Dec  2 20:05:12 2005

Changed the default filename for download from mm_msg to mm.eml, for the MS
Windows filetype association: print.c.  I suppose I should make a SET
command for this so we don't discriminate against file associations on other
platforms (Mac OS X?)  Also updated the copyright year: misc.c.

------------------------
Sat Dec  3 21:53:22 2005

When running MM using "Method 1", in which the POP script replaces movemail,
it quickly became obvious that MM would be contacting the POP server way too
often, sometimes 2-3 times in succession without a pause, which is not such
a big deal if it's just looking at files in the spool directory but we don't
want it pummeling the poor POP server.  I fixed this by having move_mail()
keep track of the time, and never call the POP script more often than 30
seconds apart (only when use_pop is set).  When move_mail() is called within
less than 30 seconds since the previous POP access it just says "no new
mail".  Then I realized that MM would try to get new mail only if it saw it
sitting in the spool (which won't happen with Cyrus, there won't be any
spool).  Fixing this required major surgery to new_mail().

------------------------
Sun Dec  4 15:13:28 2005

Let CHECK-INTERVAL be the minimum interval for contacting the POP server.
This way the user can control it.  However, MM still won't contact the POP
server more than once per 30 seconds.  move_mail(): newmail.c.

Moved the password prompt to the place where we call the POP script.  If
the password hasn't been set yet, prompt for it.  This way we don't ask for
the passowrd if MM is being used (e.g.) to examine some random file.
main(): mm.c; move_mail(): newmail.c.

But when I did this and then did "mm examine foo.txt", why did it still want
to check for new mail?  Changed cmd_get() to set a nocheck flag if the file
is read-only (EXAMINE) or it is not the user's mail file, and changed
new_mail() to check this flag.  file.c, newmail.c.

But it still checked for new mail.  Turns out that in the main program, the
call to new_mail() was executed unconditionally the first time through the
loop before we parsed any commands and therefore before any mail file was
loaded.  I changed it to call new_mail() only if there was a current mail
file.  top_level_parser(): mm.c.

I received spam from some place that put junk in the Date: field.  This
caused the POP script to write a From line with missing date, which MM did
not like at all.  Fixed the POP script to supply the current date and time
in this case.

------------------------
Mon Dec  5 10:27:27 2005

Fixed CONTENT-TYPE command to allow allow the user to specify the entire
content-type, not just charset.

Discovered that INPUT /CLEAR /NOWRAP has a fatal flaw, namely that if the
search target straddles an INPUT buffer boundary, it will never be seen.
Worked around this one temporarily by adding a trick to the script.  Need
to fix it in Kermit.

Announced Beta.01 to emailtech-wg@columbia.edu.

------------------------
Tue Dec  6 12:00:38 2005

Don't check spool file if movemail_path is empty, which means that we have
some other process delivering mail asynchronously to ~/.mm-newmail (a.k.a.
Method 2).

------------------------
Wed Dec  7 10:22:56 2005

Tried to build MM on Linux RH AS4.2, bombs out immediately with ANSI
strictness ravings.  Looked through my notes and saw that I had not actually
made any changes to ccmd except to get it to compile on non-Sun platforms.
So it should be safe to just take Nelson Beebe's version.  I did that, and
of course it built fine on Linux ("make gnu-linux").  When I bring this
version back to solaris and build it, it's OK there too, but clashes with
my copy of MM because MM refers to some ccmd header files which have been
changed to disagree with how some ccmd items are declared in the MM source.

Then I copied all of Nelsons's files into the mm tree, and then copied all
the files I have modifed on top of those.  Then I tried compiling.  Each of
my files got horrible errors, and I fixed them one by one, importing
Nelson's ansifications and modernizations and occasional fixes while taking
care to preserve (and sometimes modernize) my changes.  This involved about
30 *.[ch] files.  Finally MM built OK on Solaris, starts OK, but dumps core
if I give it any command.

It turns out that the array of function pointers declared towards the bottom
of cmds.c is NOT generated automatically; it has to be VERY CAREFULLY edited
by hand to match the CMD_blah values in cmd.h, which ARE generated
automatically from cmd.H.

------------------------
File pairs like cmds.H and cmds.h, that differ only in case, don't work on
case-insensitive file systems like Mac OS X.

HEADERS RECENT used to apply to the the messages that came in since MM was
started.  Now it applies to the ones since we last checked mail.

Re-enable locking code via #ifdefs.

#ifdef COLUMBIA lock mail file using file locks.

Add SET DEFAULT-DOWNLOAD-FILENAME (this name is OK).

Ctrl-C... 
Do you really want to exit MM? [y/n] y  <-- doesn't exit.
Do you really want to exit MM? [y/n] y

Add build instructions.
Add list of site flags.
Add list of feature flags.

Make the POP script remember whether it is logged in so it knows
what to send on Ctrl-C.

Add this to mm "make clean":
rm -f mm argsort mkdefines mkversion mm-trans mmversion sys-prof movemail -
prepatch

add linux target to ccmd makefile.

Message 14 has bad date string (continuing)
Message 15 has bad date string (continuing)
Message 16 has bad date string (continuing)
Message 17 has bad date string (continuing)

Here are headers from some of the offending messages:

From: Odetta Rogers  <newsletter@diversitybusiness.info>
Date: 16 Dec 2005 16:28:50 -0500

From: Best Postcard  <service@postcard.com>
Date: Sat, 24 Dec 2005 04:50:16 -0500 (EST)

This has happened twice in about 3 weeks.

There doesn't seem to be anything wrong with them.  Added code to mbox.c
to print the bad line, and to substitute the current date/time rather than
Wed Dec 31 19:00:00 1969.  mbox.c, 24 Dec 2005.

Changed Copyright year to 2006, misc.c, 8 Jan 2006.

Added QD as an invisible synonym for DELETE, because of 'more' -- first you
have to type 'q' to get out of more, then 'd' to delete the message.  But if
the message was less than a screenful, you often wind up typing 'qd'.  cmds.H,
cmds.c, etc, 8 Jan 2006.

MM writes a bogus in the Resent-Date: header:
     Resent-Date: Sun, 8 Jan 106 12:42:18 EST

The problem with date is in rfctime() in dates.c.  Old (which works):
    sprintf(str,"%3.3s, %d %3.3s %d %d:%02d:%02d %s",
	    day_names[t->tm_wday],
	    t->tm_mday, month_names[t->tm_mon], t->tm_year + 1900, ...

New (which doesn't):
    sprintf(str,"%3.3s, %d %3.3s %d %d:%02d:%02d %s",
	    day_names[t->tm_wday],
	    t->tm_mday, month_names[t->tm_mon], t->tm_year /*+ 1900*/, ...

What will I break if I uncomment the "+ 1900"?  rfctime() is called in only
two places: in writing the Resent-Date (which needs to be fixed), and in
sendmail() in sendmail.c, which as far as I can tell, adds the Date: header
to every outgoing message.

In sendmail(): 

  x->string = safe_strcpy (rfctime ((time_t)0));

In do_remail_one():

  msg->resent_date->string = safe_strcpy(rfctime(time(0)));

What if I make the second one like the first?  Makes no difference.  It
seems rfctime() is never called when I send a new message because in that
case, the date header has already been added elsewhere.  I tried to see
where, and how, and have the Resent-Date set the same way, but I couldn't
home in on it, so I just uncommented the "+ 1900" and everything seems to be
fine.  rfctime(): dates.c, 8 Jan 2006.

The new version of MM seems to rewrite the From: and Date: headers when I
remail.  Turns out that this happens because the following was added to
config.h in the Utah version:

/*
 *  Let's let sendmail do it's thing with From:, To:, Sender: headers.
 */

#define DONT_EMIT_FROM_HEADERS

and is easily fixed in config.h by not defining, or by undefining it once
defined.  config.h, 9 Jan 2006.

The other big change since reconciliation with the Utah code is that the
mail file is no longer locked, which is bad because it's too easy to start a
second copy of MM and then mess up your mail file.  Kernel locks are a big
pain when using NFS and multihomed hosts (as we do here), but first let's
see if we can put it back the way it was...  I did in config.h by defining
HAVE_F_LOCK within #ifdef COLUMBIA..#endif.  The resulting mm binary
interoperates with the old one in guaranteeing exclusive access to the mail
file and to ~/.mm-newmail.  9 Jan 2006.

When I start MM on Solaris, it says "Cannot restore tty attributes: error 9".
This comes from ccmdmd.c, the machine-dependent module of ccmd.  Error 9 is
"bad file number".  I see this has been noticed before by other people.
It looks like the ccmd routine unraw() is sometimes called on a file
descriptor that is not open.  I put in a check for this, as well as a check
that the tty modes have indeed been saved before we try to restore them.
ccmd/ccmdmd.c, 9 Jan 2006.

------------------

How about making a lockfile instead?  Then users can fix problems
themselves.  If MM GETs a mail file in R/W mode, it should create a file
in the same directory with ".lock" appended to its name.  The lock file
should contain:

  username
  hostname
  timestamp
  pid

But then it will only interoperate with itself.  Maybe it should do a kernel
lock AND a file lock.  That way, if kernel locking fails because the file is
in use, MM can give the user the info from the file lock.